Evaluating a tool to deeply understand misaligned agent behavior post-hoc.
Evaluating a tool to deeply understand misaligned agent behavior post-hoc.
Project Details
Updated 09/21/26 · Edited by orgThis project is investigating the ability of Synthetic-Document fine-tuning to develop "micro model organisms" via belief implantation as a tool for model forensics work. The degree of monitorability we hold over advanced AI systems is rapidly decreasing deeply motivates development of techniques for a science of model intentions/motivations. Current forensics methods use contrastive belief implantation via prompting to causally test hypotheses of reasons for model behavior. However sufficiently aware models (w.r.t eval/situational awareness) can distinguish test environments from real ones forcing us into a cycle of developing more realistic RL environments (which is extremely hard). We propose the use of SDF to implant deeper seated beliefs than via prompting that can be used to causally test hypotheses of what intentions/motivations led to a specific misaligned behavior. Validation of SDF use in forensics not only adds a tool to the forensics kit but also addresses an asymmetry in current forensics workflows.
Singh et al. (Model Forensics paper) discusses the need for "positive controls": Model forensics abductively reasons about misalignment scenarios by generating hypotheses of model intentions or beliefs/interpretations of user instructions the model may hold that explains the observed misaligned behavior. However null results from hypothesis testing don't confirm the invalidity of the hypothesis (i.e. there exists no way to confirm that a model when acting with some belief must behave in the way we observed, it simply may be that the belief insertion wasn't successful or the model is sufficiently aware of its environment and chooses to act aligned). Only positive results are meaningful as that confirms given some belief the model acts a certain way. If SDF is validated as a method that can implant a "believable" belief in models, we can create model organisms that act accordingly given a certain belief and resolve this asymetry.
Theory of Impact
Updated 09/21/26 · Provided by member · VerifiedResearch Problem
Current research shows strong evidence for loss of oversight and covert misalignment. As models are increasingly post-trained with RL in hackable environments, obfuscation and rationalization are common occurrences in the CoT. A clear distinction can be seen between models undergoing capabilities RL training with and without safety priors: RL on base models fresh out of pretraining yields CoTs containing overtly misaligned reasoning, while models given safety training before RL capabilities training produce reasoning that seems aligned to monitors/graders. Investigations of covert cases also show motivated rationalization — a model justifying its actions by treating a real environment as a simulation, while judging the same evidence out of context to be real — and show that removing the rationalization from the reasoning trace makes CoT monitors flag the activity at a much higher rate.
Covert misalignment and loss of oversight are imminent under our current training regime. Failing to address the lack of robustness of current alignment methods to inadvertently adversarial training (RLVR in this case) will provide us with a fake sense of security on current alignment methods. As capabilities increase, the ways in which models choose to task-game or specification-game are less identifiable (especially in CoT), and answer generation can be guided by "values" unknown to the user. This motivates the development of a framework to study specific instances of misalignment.
People
Updated 09/21/26 · By grantmaking.aiTeam Member
Funding Details
- -
- -
- ~3 months
- $0
- -
- -
- -
- -
- seeking first grant
- -
Discussion
No comments yet. Be the first to share your thoughts.