Testing Corrigibility as a Singular Target

Raising $18,000active

Developing the first comprehensive behavioral benchmark of corrigibility and training models with corrigibility as a singular target (CAST).

Endorsed by+1

Developing the first comprehensive behavioral benchmark of corrigibility and training models with corrigibility as a singular target (CAST).

Endorsed by+1

Project Details

A central problem in AI alignment is that a near miss could be catastrophic. If the fulfillment of an agent's goals is even slightly inconsistent with human welfare, the danger is that a sufficiently powerful agent will pursue those goals without limit. What we want is to be able to adjust those goals in a controlled manner as our AI agents become more powerful. However, instrumental convergence predicts that agents will exhibit value preservation, causing them to protect the misaligned goals they already have. For this reason, prevailing approaches, which aim to imbue AIs with a range of values, appear unable to give humans the control we will need to correct misaligned goals as AIs become more powerful.

However, a largely unexplored approach to alignment developed by Max Harms, corrigibility as a singular target (CAST), holds promise as a solution to this challenge. In CAST, the strategy is to train AI agents to have the singular goal of corrigibility, where corrigibility is, roughly, the desire of an agent to empower a principal to correct its flaws. Such an agent, even if only imperfectly corrigible, will tend to assist, rather than hinder, attempts by its principal to increase its corrigibility. Thus, corrigibility may be the unique training target where a near miss, instead of being fatal, is self-correcting. What's more, we have reason to believe that a CAST agent will itself be a desirable form for AI to take. Theoretical work suggests that CAST agents are likely to exhibit a range of beneficial, emergent properties, including obedience, transparency, and crucially, safety from instrumentally convergent drives.

There remain significant challenges and risks to this approach. Alignment to any set of values is an ongoing challenge, and CAST may amplify the risk of human misuse since CAST agents will not independently resist implementing immoral instructions from their principals. However, there is reason to believe that human misuse is a more tractable problem than misaligned ASI, for example, with the use of democratic oversight. And given the current trajectory of AI capabilities and alignment failures, and the inherent risk of any approach to developing safe ASI, exploring the feasibility of the CAST approach now appears to be a valuable use of resources.

Despite this, there has been almost no empirical work on corrigibility as a singular target, and evaluations of model corrigibility have been limited to indirect measures (such as stated preferences) or isolated aspects (for example, shutdown resistance) rather than measuring corrigibility as an agentic disposition. I aim to fill this gap.

I. Corrigibility Benchmark

The first part of the project is creating a comprehensive benchmark for corrigibility. This will serve to measure the baseline corrigibility of frontier AI models, track how it changes across model generations, and facilitate research into CAST and corrigible AI training more generally.

The benchmark will have three components:

Knowledge of Corrigibility: A multiple-choice test of corrigibility validated by experts.
Self-Report: Elicitation of the model's judgments of its own corrigibility.
Simulated Behavior (core): Agents placed in corrigibility test scenarios within realistic simulated environments, with an LLM judge scoring against expert hand-authored rubrics.

Current Status:
I have begun work on the simulated behavior portion of the benchmark. To combat testing environment awareness, I have assembled a set of agentic environments derived from the real system prompts of deployed agent frameworks. Pilot runs through the Inspect and Petri frameworks confirm the harnesses deploy successfully in both single-turn and multi-turn settings and that an initial scoring framework (LLM judge calibrated against hand-labeled scenarios) discriminates between corrigible and incorrigible behavior.

II. CAST fine-tuning experiments

The second part of the project directly tests CAST training by fine-tuning open-weight models for corrigibility as the singular target. The goal is to empirically explore CAST training techniques and the properties of CAST agents that emerge. Some questions that I hope to answer are:

To what extent do theoretically expected properties actually emerge in CAST agents?
Do CAST agents show robust resistance to instrumentally convergent goals?
What are the advantages and disadvantages of a highly corrigible agent in practice?

Project Status:
I have begun construction of a synthetic data pipeline that takes a persona-installation approach to fine-tuning, aiming to instill corrigibility as a stable identity rather than a set of surface behaviors. The data will consist of scenario vignettes demonstrating corrigible behavior, synthetic documents (essays, stories, news articles) depicting corrigible AI, and multi-agent trajectories showing corrigibility in agentic settings. I have also developed a CAST constitution, grounded in the CAST series text, for guiding data generation and corrigibility judgments during training.

Theory of Impact

Top AI labs and AI alignment scientists place significant weight on corrigibility as a pathway to safe AI. MIRI introduced corrigibility as central to AI safety and Paul Christiano cites corrigibility as an "important part of my overall optimism about alignment". Anthropic's current Claude constitution states that "we call an AI that is broadly safe in this way 'corrigible'", where being "broadly safe" is the first of Claude's four core priorities.

But corrigibility and the CAST framework in particular are neglected areas of research. CAST holds promise as a solution to one of the more intractable problems of AI safety, but it hasn't been tested. Training for corrigibility as one target among many is fundamentally different from the CAST approach. Because corrigibility is in tension with instrumentally convergent drives (such as not being shut down), we ought to expect values other than corrigibility to "win out" unless corrigibility is trained as the singular target. In general, we should not expect robust corrigibility to arise naturally.

People

Ian Kahn

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$18,000

Ideal

$78,000

What this ask is

A central problem in AI alignment is that a near miss is likely to be catastrophic. If the fulfillment of an agent's goals is even slightly inconsistent with human welfare, the danger is that a sufficiently powerful agent will pursue those goals without limit. What we want is to be able to adjust those goals in a controlled manner as our AI agents become more powerful. However, instrumental convergence predicts that agents will exhibit value preservation, causing them to protect the misaligned goals they already have. For this reason, prevailing approaches, which aim to imbue AIs with a range of values, appear unable to give humans the control we will need to correct misaligned goals as AIs become more powerful.

However, a largely unexplored approach to alignment developed by Max Harms, corrigibility as a singular target (CAST), holds promise as a solution to this challenge. In CAST, the strategy is to train AI agents to have the singular goal of corrigibility, where corrigibility is, roughly, the desire of an agent to empower a principal to correct its flaws. Such an agent, even if only imperfectly corrigible, will tend to assist, rather than hinder, attempts by its principal to increase its corrigibility. Thus, corrigibility may be the unique training target where a near-miss, instead of being fatal, is self-correcting. What's more, we have reason to believe that a CAST agent will itself be a desirable form for AI to take. Theoretical work suggests that CAST agents are likely to exhibit a range of beneficial, emergent properties, including obedience, transparency, and crucially, safety from instrumentally convergent drives.

Despite this, there has been almost no empirical work done on corrigibility as a singular target, and existing evaluations consider corrigibility only as isolated fragments. Examples include multiple-choice questions about accepting goal changes and behavioral tests of a single component, such as whether a model resists shutdown. But no benchmark measures corrigibility as a unified behavioral disposition across agentic scenarios. I aim to fill this gap.

Component 1: Corrigibility Benchmark

The benchmark will have three components:

Knowledge of Corrigibility: A multiple choice test of corrigibility validated by experts.
Self-Report: Elicitation of self-reported judgments of corrigibility.
Simulated Behavior: Agents placed in corrigibility test scenarios within realistic simulated environments, with an LLM judge scoring against expert hand-authored rubrics.

Current Status: I have begun work on the simulated behavior portion of the benchmark. To combat testing environment awareness, I have assembled a set of agentic environments derived from the real system prompts of deployed agent frameworks. Pilot runs through the Inspect and Petri frameworks confirm the harnesses deploy successfully in both single-turn and multi-turn settings and that an initial scoring framework (LLM judge calibrated against hand-labeled scenarios) discriminates between corrigible and incorrigible behavior.

Component 2: CAST fine-tuning experiments

The second component of the project directly tests CAST training by fine-tuning open weight models for corrigibility as the singular target. The goal is to empirically explore CAST training techniques and the properties of CAST agents that emerge. Some questions that I hope to answer are:

To what extent do theoretically expected properties actually emerge in CAST agents?
Do CAST agents show robust resistance to instrumentally convergent goals?
What are the advantages and disadvantages of a highly corrigible agent in practice?

Project Status: I have begun construction of a synthetic data pipeline for fine-tuning through a persona-installation approach, which aims to instill corrigibility as a stable identity rather than a set of surface behaviors. The data will consist of scenario vignettes demonstrating corrigible behavior, synthetic documents (essays, stories, news articles) depicting corrigible AI, and multi-agent trajectories showing corrigibility in agentic settings. I have also developed a CAST constitution, grounded in Harms's source texts, for guiding data generation and corrigibility judgments during training.

Theory of impact

But corrigibility, and the CAST framework in particular, are neglected areas of research. CAST holds promise as a solution to one of the more intractable problems of AI safety, but it hasn't been tested. Training for corrigibility as one target among many is fundamentally different from the CAST approach. Because corrigibility is in tension with instrumentally convergent drives (such as not being shut down), we ought to expect values other than corrigibility to "win out" unless corrigibility is trained as the singular target. In general, we should not expect robust corrigibility to arise naturally.

This absence of empirical work on corrigibility presents an opportunity for potentially high-impact, rapid research with limited initial funding. A unified benchmark which includes testing of behavioral dispositions across agentic scenarios and initial CAST training experiments on open-weight models are the highest-leverage places to begin. Even if CAST turns out less promising than expected, we still need a way to measure corrigibility in frontier models, and we need empirical data about the emergent properties of corrigible agents.

How the money will be spent

Minimum: 18K
11K stipend (roughly 2.5 months) and 7K for compute (data generation, 8B parameter fine-tune, benchmark sweeps, frontier model access for development).

Ideal: 78K
60K stipend (12 months x 5K) and 18K for compute (the minimum compute scaled to full year, plus a larger 70B parameter fine-tune).

Discussion

Max HarmsEndorsement21d

I've been working with Ian since March, and have been impressed with his work. I'm really hoping he gets funded to a level where he can focus on it full-time.