Independent assurance for AI systems, testing whether agents, evaluations and controls actually do what their claims say.
Independent assurance for AI systems, testing whether agents, evaluations and controls actually do what their claims say.
Project Details
Updated 09/14/26 · Provided by member · VerifiedOMEGA is an independent AI assurance research project founded by Warren Smith. It develops and tests methods for checking whether AI agents, evaluations and control systems actually behave as their assurance claims imply.
The work combines empirical experiments, formal methods, independent observation, adversarial testing and mechanically verifiable evidence. Rather than treating an agent’s report, a passing test or another model’s judgement as sufficient evidence, OMEGA separates claims from independent observations and verification.
Current work includes agent authorization and execution boundaries, evaluation and refusal instrumentation, mechanically graded AI-worker qualification, replayable evidence, and autonomous external investigations with independent oracles. Recent experiments have identified defects in evaluation and verification tooling, tested agent-control boundaries, and demonstrated an autonomous research workflow that can select an unknown external question, preregister an experiment, challenge its own controls, obtain a mechanically checked result, run hostile review and preserve the evidence with minimal operator involvement.
OMEGA is currently an experimental independent research effort, not a claim that these methods have demonstrated catastrophic-risk reduction. The next stage is to test which parts generalise and where they fail.
Theory of Impact
Updated 09/14/26 · Provided by member · VerifiedAs advanced AI systems perform longer and more consequential tasks, safety increasingly depends on claims about what those systems did, what controls constrained them, and whether evaluations accurately measured their behaviour.
Those claims can fail even when the surrounding system reports success. An agent can claim an action succeeded when its effect is absent; authorization can apply to a different representation from what executes; evaluation instrumentation can hide relevant failures; and verification can be correct about a model that is inadequately connected to the deployed system.
OMEGA aims to reduce this assurance gap by developing methods that separate the system making a claim from the evidence used to judge it. The causal pathway is:
stronger independent observation and mechanically checkable evaluation → earlier detection of false safety/control claims → better evidence about where safeguards actually work or fail → safer deployment and governance decisions for increasingly autonomous AI systems.
The near-term research question is whether these methods remain useful across different systems and failure classes. The project therefore prioritises falsifiable experiments, negative results and explicit limits rather than assuming that an assurance architecture is effective.
People
Updated 09/14/26 · By grantmaking.aiTeam Member
Discussion
No comments yet. Be the first to share your thoughts.