A sealed public registry for AI eval results that lets anyone prove, offline, that no result was rewritten or quietly deleted after publication.
A sealed public registry for AI eval results that lets anyone prove, offline, that no result was rewritten or quietly deleted after publication.
Project Details
Updated 07/11/26 · Provided via application · VerifiedPalimpsest is an AI evaluation registry, and it already runs. Every result seals the moment it goes up, into a public record that only appends. Nothing gets revised after the fact. Not by a lab, not by a government, not by me.
The first wing is live: a daily evaluation of Chinese state-aligned models, measuring how they refuse or reword sensitive answers. Each prompt gets sampled five times per run and scored with 95% confidence bands. Around 360 model responses enter the sealed record every day, on infrastructure I run around the clock. The datasets, schema, and methodology are published for researchers, with a citation format. A human validation study of the classifier is underway, and a working paper is drafted for arXiv.
This grant funds the second wing: Western labs. Their safety evaluation results go under the same seal, so a lab can't go back and touch up a grade it already published.
I build and operate Palimpsest alone. I'm a lawyer turned developer, on this full time, and the solo setup is deliberate: a registry that Chinese and Western audiences both trust can't belong to a lab or a government, or to any funder with a stake in the grades. The methodology has been through feedback from academics who study Chinese censorship. The next hires are two coders who read Mandarin, for the validation study. Budgeted, recruiting now.
Theory of Impact
Updated 07/19/26 · By grantmaking.aiEvery serious safety claim about a frontier model now routes through evals. Labs decide whether to ship on eval results, responsible scaling policies trigger on them, and regulators are starting to cite them. All of that evidence lives in ordinary web pages, PDFs and git repos that the publishing organisation can edit after the fact.
That's a weak foundation for the one record we most need to trust. If a capability result later becomes inconvenient, the cheapest response is a quiet revision. Nobody has to lie. The page just changes, and no outsider can prove it was ever different.
Palimpsest already solves the general version of this. It's a sealed ledger, live at palimpsest.info, that records what states and AI models erase over time, and anyone can verify the record offline without trusting me or my server. This grant funds the extension I've already started: sealing eval results at publication time, so any later edit, deletion or cherry pick is detectable by anyone.
Eval integrity is a precondition for every governance mechanism the field is betting on, from RSPs to third party audits to if-then commitments. Making tampering detectable raises the cost of safety washing and keeps the evidentiary record intact for the people who will eventually need it in a dispute. It requires nobody's permission or cooperation, which is the point.
People
Updated 07/19/26 · By grantmaking.aiTeam Member
Discussion
No comments yet. Be the first to share your thoughts.