PRML — tamper-evident pre-registration of AI evaluation criteria

An open specification that binds AI evaluation criteria to a SHA-256 digest before the run, so a third party can check that those exact criteria existed no later than a given time and have not changed since.

Project Details

PRML is an open specification for recording the criteria of an AI evaluation before it runs. The metric, the comparison rule, the threshold, the dataset and its hash, the seed and the system version are serialised to canonical bytes and hashed with SHA-256. The digest can be anchored at a public registry, which returns a signed receipt, an RFC 3161 timestamp from a public time-stamping authority, and an entry in a public transparency log. A sealed commitment lets the criteria stay private while the digest and the time are public.

What a record proves is narrow: that exact criteria object existed no later than that time and has not changed since. It does not prove the criteria were good, that the evaluation ran after that time, or that the result is correct.

I am one person. I wrote the specification and all four reference implementations — Python, JavaScript, Go and Rust. They produce the same bytes and run the same 21 conformance vectors. There is also a negative-conformance suite of 20 cases; Python and JavaScript run all 20, Go and Rust run the 16 that are not YAML-specific.

Where it is up to: the specification, the four implementations and the public registry are live and openly licensed. There is a GitHub Action and integrations with MLflow, Giskard and UK AISI's Inspect. The media type application/vnd.prml+yaml was registered with IANA on 3 September 2026 — that is a registration, not an endorsement. Two free tools are live. Nobody outside me has yet run any of this on their own evaluation. That is the honest state.

Theory of Impact

The chain, as short as I can make it:

Safety decisions increasingly rest on evaluation results — deployment gates, frontier safety commitments, statements to regulators.
The acceptance criteria behind those results live in ordinary documents. A document carries no evidence of when the criteria in it were set.
So a threshold loosened after the results were seen is indistinguishable, to a reader, from one fixed beforehand.
The reader who matters — an auditor, a regulator, a counterparty, another lab — therefore cannot separate a real safety margin from a re-scoped one. They have to trust the claimant's account of the claimant's own timeline.
PRML removes that one dependency. The criteria are hashed and time-evidenced, and the check can be done by someone who does not trust the claimant.

What I am not claiming. This does not make evaluations better, and it does not prove the criteria were the right ones. It also does not prove the evaluation ran after the record was made — binding execution to the record is the main open gap, and it is written into the specification as such. This is a small upstream contribution to whether safety claims can be audited at all, not an alignment result.

People

Cuneyt Ozturk

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

Funding ask

Open

Minimum

$4,500

Ideal

$6,000

How the money will be spent

$2,500 — canonical serialisation. Can two independent implementations be driven to produce different bytes for the same criteria object, and does the negative-conformance suite actually close the parser-differential surface?

$2,000 — the registry worker. It is the only privileged component in the system and the one that signs receipts.

$1,500 — the sealed-commitment scheme. Can one commitment be opened to a different criteria object, and is the reveal protocol sound?

That is roughly four to five days of an experienced reviewer at European consultancy rates. I have not signed a reviewer yet, so it is an estimate and not a quote. At the $4,500 minimum I do the first two and defer the third. If the review costs less, I return the balance or agree a wider scope first.

The reviewer is asked to try to break these, not to read the code and say it looks fine. The report is published in full whatever it finds.

This is not a second pot. The same ask is live on Manifund and that is where the money goes; this listing exists so funders can find it. I will not take two grants for one line of work.

Discussion

No comments yet. Be the first to share your thoughts.