Verifiable Eval Registry
AI evaluations that cannot be quietly rewritten
AI systems increasingly decide what a person can and cannot learn, and the audits that check them can be re-run, quietly cherry-picked, or revised until the number reads well. This registry makes those moves testable. Every declared probe commitment is sealed before its matching run can enter the chain; current collectors additionally refuse to query until their exact protocol is public. Every result seals when it publishes, and roots are deposited with parties this project does not control. Edit the served chain and local verification fails; rewrite the whole chain and external copies or witnesses can expose the conflict—including if the operator is the one attempting it.
An eval you are asked to trust
A number in a report. The questions may have been chosen after the answers were seen, a weak run may have been dropped, a rate may have been recomputed on a friendlier subset. None of that is visible in the artifact, so what you are really assessing is the reputation of whoever published it.
An eval you can recompute
Every declared commitment precedes its matching run in the served chain, full responses or their declared artifacts are hashed, and each entry chains to the one before it. Current collectors additionally require an exact protocol in a separate public commit before their first query. Clone the repository and run the verifier with the standard library alone. If one value moved, or a run cites a commitment absent earlier in the chain, verification fails on your machine rather than on ours.
Why she built it
A censored answer became an eval
Palimpsest began when its founder and author tested Chinese and state-aligned language models on documented political events and criticism of the Chinese Communist Party. She saw answers change, disappear, or shift into official framing depending on the model and the language of the prompt.
A screenshot could show one disturbing answer. It could not show whether the question was selected after the result, whether failures were dropped, whether wording caused the effect, or whether the record changed later. So she turned the observation into a repeated measurement, then built the registry needed to make that measurement auditable.
This is an origin, not a universal claim about every model made in China and not proof of any provider's motive. The claim is narrower: these are observable response differences, tested against frozen prompts and controls. Independent work has also documented bilingual political bias and refusal-to-reframing patterns; Palimpsest's contribution is a continuous public record that can be challenged from the raw evidence. Read the founder's full account, with evidence receipts and the claim's falsifier.
- PromptAsk about a documented event or a criticism of CCP policy, in matched languages and scripts.
- DiscrepancyRecord refusal, omission, narrative substitution, and differences against unaligned controls—without using another model as judge.
- ProofFreeze the exact protocol first; publish full responses, denominators and uncertainty; seal the result so a weak run cannot quietly vanish.
The reading
Figures on this page are read from eval-registry-latest.json when the page loads. Nothing here is typed in by hand. If the numbers below do not appear, the fetch failed or JavaScript is off, and the raw file and the chain are served directly.
Model runs sealed
What the evidence can claim today
Read from eval-assurance-latest.json at load. This is a claim ceiling, not a self-awarded score.
Machine-checked assurance
Not established yet.
How a result becomes unrevisable
The machinery
Freezing the question is the whole trick
An eval is gameable at exactly one joint: the gap between choosing the questions and publishing the answers. Close that gap with a hash committed in advance and the remaining attacks all require rewriting history in public, in front of witnesses who are not us.
Pre-register
The probe set is hashed and sealed into an append-only chain before the model is ever asked.
Run
Each result records the model, a hash of its full responses, and the metrics, and it must reference a probe set that was frozen earlier.
Verify
Anyone recomputes the chain. A result whose questions were not frozen first, or any value altered after sealing, fails.
Anchor
Each new root is deposited outside our infrastructure: snapshotted by the Internet Archive and stamped into Bitcoin via OpenTimestamps. An independent witness on separate infrastructure re-verifies the served chain and alerts if any previously seen history changes. Rewriting the record would mean defeating them too, not only us. Full threat model.
As models mediate more of what people can know, third parties need to audit them and prove the audit was not rewritten afterwards. A shared, independently verifiable record is the missing substrate. Palimpsest's own model-erasure audit is simply the first thing anchored here. The registry itself is model agnostic and topic agnostic: any suite, any model, same machinery.
Recent runs
Two frozen suites with no model in common. Chinese state-aligned models and Western frontier models are held to the same machinery, each on its own suite, and watched over time for what they quietly stop answering.
Every rate carries its own denominator. n scored is how many probes in that run returned a usable response and therefore produced the percentage beside it, and abstained is how many were dropped because the response was unusable. Runs of the same suite can differ in n, so the rates are not directly comparable without reading both columns.
Frontier refusal drift
One frozen bank of benign, legitimate questions, put to each frontier model across time. A model that used to answer a question and now refuses it has erased something, and the erasure is normally undisclosed. Every run is sealed into the registry above, so the change is on the record and cannot be un-recorded.
Three things make this harder to dismiss than a refusal count. Every question is asked three ways. Refusal behaviour is famously sensitive to wording, so one phrasing cannot tell you whether a model has a policy or a tripwire. The family is the unit; the wording column reports how often the three agreed. The alarm is valid under unlimited peeking. This board re-reads every model every six hours, forever, and a conventional significance test re-run on a schedule will eventually cry wolf with certainty. The standing alarm is a mixture supermartingale instead, so the chance it ever fires on a model whose behaviour has not changed is bounded for the lifetime of the watch, not per look. The raw answers are published. Every response behind the current reading is served at refusal-drift-transcripts.json, and its hash is what the sealed run commits to, so you can check that the text we show is the text we sealed and re-derive every label yourself. If you think our classifier is wrong, the evidence to prove it is already in your hands.
Read from refusal-drift-latest.json at load.
The grid itself: every family × every model, this run
refusal-drift-latest.jsonPer-family verdicts for every model are published in the raw reading, and this grid draws from it when the page loads.
A model's rate is computed over the question families it actually returned usable responses to, not over the whole bank, and the interval beside it is a Wilson 95% score interval on that denominator. The models do not share one denominator, so read each rate against its own n. Abstained is the remainder, dropped rather than counted either way — a model we could not reach has not refused anything. Drift shows once a model has a second comparable run, and the standing alarm speaks only after it has watched long enough to know what that model's own week-to-week churn looks like. Control questions with no policy dimension run alongside the rest: if those are refused, the run is an instrument fault and is labelled one.
The chain, and how to attack it
Evidence
These are the values you recompute
The head hash covers every entry before it, and the merkle root is a single fingerprint of the whole registry. Take them from this page, take the chain file, and check that they agree without asking us anything. The verifier is standard library only: no install, no key, no account.
Chain state, as served
$ python3 scripts/verify_eval_registry.py
Recomputes the chain and checks that every run references a probe set frozen before it.
External anchors
A hash chain proves internal consistency. It does not by itself stop the party that holds the chain from rewriting the whole thing and reserving every hash. What stops that is depositing the root with someone else. Each new root is snapshotted by the Internet Archive and stamped into Bitcoin through OpenTimestamps, so a rewrite would have to defeat those records too.
Read from anchors-latest.json at load.
Palimpsest
An open source observatory of information control, run as a public good, and the verifiable eval registry on this page. It watches the censor and never the censored: public reads only, no person inside is ever asked to act, and no state-aligned model is ever the analyst. Every finding ships its raw evidence. Grounded in the trusted-public-archive and tamper-evident logging literature (ARCHANGEL, arXiv:1804.08342; arXiv:2509.03821) and the model-censorship measurement work (arXiv:2505.17441, arXiv:2506.12349).