Replicate and extend AI Debate oversight experiments to study when limited verification hurts judge accuracy, varying oracle design, model capability gaps, and best-of-n malicious debater strategies.
Replicate and extend AI Debate oversight experiments to study when limited verification hurts judge accuracy, varying oracle design, model capability gaps, and best-of-n malicious debater strategies.
Project Details
Updated 09/08/26 · By grantmaking.aiProject summary
"Debate" protocols for AI safety involve multiple strong, untrusted models attempting to persuade a weaker, trusted judge that their position on some question is correct. My collaborator implemented a version of Debate in a recent experiment, pitting an honest debater against a dishonest one and testing how often the dishonest debater was able to convince the judge, as a function of capability and knowledge asymmetry between the judge and debaters. To induce a controlled knowledge asymmetry, he created page-length descriptions of three fictional worlds, and framed the debates as questions about the worlds. The debaters had unlimited access to the descriptions, while the judge had access via a yes/no oracle with some number of queries. Counterintuitively, limited verification made the judges worse at identifying the correct answer, producing a U-shaped accuracy curve as the oracle query budget increased. We will further probe this failure mode and clarify under what conditions it obtains.
What are this project's goals? How will you achieve them?
We will repeat and extend the experiment, with the goal of answering the following questions:
• Most of the cases where the judge selected the correct argument with no oracle access but the incorrect argument with some oracle access fell into one of two categories. In the first, the judge selected a true or “directionally true” statement that the honest debater made, but the oracle indicated that it was false, undermining the honest debater’s case. In the second, the judge selected a true but irrelevant statement that the dishonest debater made (and the oracle confirmed its truth), and drew the improper inference that the dishonest argument was better. This first failure mode was in large part a consequence of the oracle design. Can we reduce or eliminate this failure mode with a better-designed oracle, while maintaining tight control over the amount of information available to the judge?
• The previous experiment used an 8B parameter model for the weak judge, and a 70B model for the debaters, oracle, and judge with no capability asymmetry. Is the failure rate a true function of capability gap, or is it dominated by absolute capability? Maybe the 8B model is too small to reliably guess correctly regardless of the strength of the arguments, or the 70B model is too small to construct convincing but subtly wrong arguments. Testing with additional, even more powerful models will suffice to explore this.
• Are there more effective malicious strategies than simply “argue for the wrong answer as convincingly as possible”? Even with this most basic strategy, we should be able to elicit more convincing answers with best-of-n prompting instead of single-shot; how does this affect protocol performance?
How will this funding be used?
The funding will be spent on compute from a cloud-based LLM inference provider. We plan to add at least the Llama Maverick and 405B models, plus potentially a frontier model, and test all combinations where the debaters are more powerful than the judge, with best-of-n prompting at whatever n is feasible given remaining budget after design iteration.
Who is on your team? What's your track record on similar projects?
My collaborator has conducted two small-scale independent alignment experiments, of which the precursor to this project was one. These are written up in the blog posts below:
https://www.lesswrong.com/posts/2a3vce7WooJ4XkDqw/limited-verification-can-hurt-debate-oversight
https://www.lesswrong.com/posts/uYKA4dt66MFzXDmWY/testing-few-shot-coup-probes
I am a machine learning practitioner with hands-on depth spanning personal and professional work, including MTGRL (a personal ML project) and an in-progress Master of Science in Applied Machine Learning at the University of Maryland, College Park. Currently serving as technical lead for an Agentic AI team, where I design and drive the core agentic infrastructure from inference through production.
What are the most likely causes and outcomes if this project fails?
The most likely "failed" outcome is a simple negative result; that the failure modes identified in the preliminary experiment were artifacts of the experimental design rather than a genuine phenomenon. This would nonetheless be a real result, and the performance of Debate in the high knowledge-asymmetry regime is underexplored in any case.
How much money have you raised in the last 12 months, and from where?
0$
People
Updated 09/08/26 · By grantmaking.aicreator
Funding Details
- -
- -
- -
- -
- -
- -
- -
- $10,000
- -
- -
Discussion
[Progress update]
What progress have you made since your last update?
We re-analyzed the pilot data and quantified the headline effect: a small oracle budget raised the dishonest debater's win rate by +7.2pp (95% CI [4.6, 10.2]).
Before scaling up, we audited the pilot code and found two serious bugs in the oracle channel: every "NOT ADDRESSED" reply was miscoded to "NO", and ~100% of oracle queries were sent garbled. We retracted the mechanism conclusions and corrected the write-ups.
We then rebuilt the harness and launched a pre-registered validation run (in flight now, ~$150-200 of the grant): the same 318 transcripts re-judged under six arms, including a clean harness, a faithful bug replay, and a placebo oracle. Gates were frozen before any clean data existed. Pre-registration: https://github.com/jackmaiorino/selvarath-debate/blob/rerun-new-models/docs/rejudge-protocol.md and corrected report: https://github.com/jackmaiorino/selvarath-debate/blob/rerun-new-models/reports/2026-07-06-preliminary-findings.md
What are your next steps?
The pre-registered gates decide: if the effect survives the clean harness, we proceed to the proposed judge x debater capability grid (most of the grant). If the placebo explains it, we pivot to studying deliberation effects. If it collapses, we publish the artifact result and a decomposition of what each bug contributed. Write-up either way, including negative results.
Is there anything others could help you with?
Methods scrutiny of the pre-registered protocol before results land, and pointers to related work on oracle/verification interfaces or deliberation-length effects in LLM judging.
[Progress update]
What progress have you made since your last update?
The validation run from my last update is complete. We re-judged the same 318 pilot transcripts under six controlled arms: 11,330 judgments for $182.16.
With the oracle and judging bugs fixed, a few oracle calls still increased judge error by 3.4 percentage points (95% CI [1.3, 5.9]), about half the original 7.2-point estimate. Bug replay indicates that the two oracle-channel bugs contributed roughly half the original effect. Information-free placebo turns caused a smaller 1.6-point degradation, and the pilot's apparent recovery at larger oracle budgets did not reproduce.
Under the frozen gates, the original >=4-point primary gate was indeterminate, not passed. The bug attribution gate technically fired by 0.01 points, while the amendment frozen before the K=3 data passed and licensed only a reduced-scope Phase 2 pilot. The design below is that reduced-scope follow-up, not an unqualified launch of the originally proposed full grid.
One distinction matters: Stage 1 cleanly re-judged the legacy transcripts, but it did not repair a separate debate-generation problem in which each debater had advance knowledge of the opponent's upcoming case. Phase 2 therefore regenerates every debate with blind opening turns.
Follow-up mechanism work also changed our interpretation. Replaying the same query/answer evidence as a neutral table in fresh context removed a large share of the harm. This points to conversational presentation, turn structure, or commitment to earlier questions as part of the mechanism, rather than bad factual content alone.
Held-out calibration selected blind, uncapped, three-round debate. The selected roster has judges Qwen2.5-7B, Gemma-4-31B, Llama-3.3-70B, and GPT-OSS-120B; the debaters are Llama-3.3-70B and hosted Qwen3.7-Plus; the oracle is Llama-3.3-70B. The approved offline design contains 23,200 Phase 2 cells: a 1,060-cell capability preflight followed, if the later gates pass, by 22,140 post-canary main cells. It includes a full cap-protection interaction, an empty-evidence diagnostic, and two full-document gold-context anchors. No calls from this approved 23,200-cell plan have begun.
Previously reported project spend is approximately $208. Across two transfers, I have moved $1,800 of the grant into prepaid Together API credit ($500 earlier and $1,300 now). The Together dashboard showed $1,590.78 in available credit on July 16, implying $209.22 in net credit draw. I will reconcile the $1.22 difference from the rounded spend report against provider usage before another call. Transfers are funding, not experiment spend.
- Stage 1 protocol: frozen protocol
- Stage 1 results: validation report
- Mechanism memo: mechanism and packaging memo
- Calibration report: held-out calibration results
What are your next steps?
Before any call or outcome from the approved Phase 2 plan, owner-approved Amendment A1 waived the planned 106-question human confirmation pass. The reported full/partial/none strata are now explicitly algorithmic reply-pattern classes. A source-bound AI-assisted audit verified all 106 mappings (6/41/59; zero disagreements); its 63 semantic-quality flags are exploratory limitations only. These classes are not human-validated, and the annotations cannot relabel, filter, exclude, or reweight questions.
Before examining any Phase 2 debate outcomes, we will validate the separate query checker against a human-labeled set and freeze/hash the prompts, model settings, exact inventory, retry rules, provider reconciliation, and spending controls.
After this update, we will separately recover or waive 11 failed Gemma calibration cells (proposed recovery cap $2). A separately authorized capability preflight (five models x 106 questions x two mirrored answers; proposed cap $15) will then select the second full-document anchor without examining Phase 2 debate outcomes. Only after that will we separately authorize a small manifested canary on questions excluded from the main analysis. We will review validity and parsing, side bias, provider reliability, artifact completeness, resumability, and ledger-versus-provider cost before separately authorizing the main run. Any failed gate stops the experiment before main spend.
The current provisional empirical planning band is $650–$1,150 in incremental Phase 2 spend, with a $1,200 operational working budget and an immutable $1,500 incremental hard ceiling. We will replace the provisional band with frozen prompt/token profiles and current provider prices before authorizing paid work. The ceiling is a fail-closed boundary, not expected spending. We will post results, failures, artifacts, and actual spend either way.
- Phase 2 design and budget: immutable protocol and cost model
- Resolvability amendment: 106-question AI-assisted audit and owner-approved Amendment A1
- Launch readiness: updated readiness and sign-off
Is there anything others could help you with?
Methods scrutiny before the canary, especially the H/P/R decomposition, capability measurement, query-screen validation, clean-versus-placebo comparison, and stopping rules, would be valuable. Pointers to related work on verification interfaces, conversational presentation effects, or deliberation-induced degradation are also welcome.
[Progress update]
What progress have you made since your last update?
This update is late: Phase 2 finished on August 11 and a correction followed on August 21. Both are reported here for the first time.
The Phase 2 design posted on July 18 ran to completion: 22,140 judgments, 100% complete, $172.86 against a $173.80 projection. The original analysis was pinned before unblinding and run once.
Result. In our three fictional worlds, giving the judge a two-call oracle budget during sequential debate, versus the same debate with no oracle, increased judge error. Under the label assignment actually run, the estimate is +3.9 percentage points (95% CI [2.2, 5.7], Holm p = 0.0006).
Erratum (August 21). The Phase 3 canary exposed a harness defect: the protocol's mirrored-sides replicate was never implemented, so both replicates of every judgment showed the correct answer in the same position, and the side-bias gate as computed was non-diagnostic. We withdraw "fully pre-registered mirrored result" and "all eligibility gates passed". Under a reanalysis specification and decision rule frozen before any side-stratified number was computed, the headline survives as post-hoc robust: +4.56pp after 50/50 side standardization (CI [2.89, 6.25]), with the valid-only and within-question estimates agreeing in direction. These intervals are question-cluster bootstraps under the observed assignment and do not capture uncertainty over the missing opposite-polarity outcomes. The evidence-content contrast is stable; the packaging contrast is positive under the strict rule but inconclusive under valid-only. The secondary contrasts are re-examined in the report.
Phase 3 launched on September 6. It reuses the 492 blind transcripts across oracle budgets 0, 1, 2, 4, and 8 with two judge endpoints, Qwen3.8-2.4T-A95B and Llama-3.3-70B, for 9,840 judgments: four budget-versus-zero error contrasts under Holm, plus a secondary comparing budget 8 with budget 2. Screening reduced seven candidate judges to two, so the confirmatory claim averages these two fixed endpoints equally, with no capability-slope inference and no generalization to scale.
Delegation disclosure. While I was away, Claude launched the run under my written Phase 3 spending and execution delegation and generated the authorization signature using my key. The record identifies this as delegated approval and does not assert my personal review or signing of that artifact.
Spend. About $208 through July 14 and $181 for Phase 2. Phase 3 pre-main accounting is $119.27 ($95.68 settled, $23.59 uncertain reservations); Together's console shows $94.07 for the same window. The main run had spent about $17 as of September 6, 23:50 UTC. The $1,100 Phase 3 cap includes the pre-main spending; the certified main-run forecast is $908.82.
- Phase 2 report with erratum: https://github.com/jackmaiorino/selvarath-debate/blob/34898fb/reports/2026-08-11-phase2-main-results.md
- Phase 3 protocol: https://github.com/jackmaiorino/selvarath-debate/blob/2707cd3/rejudge/phase3_protocol_v3_r6.json
- Delegated signature record: https://github.com/jackmaiorino/selvarath-debate/blob/2707cd3/rejudge/phase3_main_delegated_signature_provenance_2026-09-06.json
What are your next steps?
Results or a status update in about a week, either way, under the frozen analysis.
Is there anything others could help you with?
Methods scrutiny of the erratum's post-hoc side standardization and of the two-endpoint claim language for Phase 3.
Note: This grant was given as part of the experimental alignment microgrant program. Because this is a weird experimental grant program aiming to make lots of unusual bets, grantees are not allowed to use this grant as a credential when applying to any other grant, or job, or on their resume/linkedin/etc. This page exists because Manifund does not allow private grants. If you are a grantmaker or hiring manager, please disregard this grant when making decisions.