I've kept track of Jesse's work on this as he's developed it and endorse it as worth seeing through.

Copying my comment over from the previous listing here.

Note: I didn't fund this but I recommended this project to JueYan Zhang, who funded it.

Points in favor

1. We need more research on interpretability-guided training! There's been a growing interest in "interpretability-guided training": see Goodfire's "intentional design", Neel Nanda, gradient routing and SGTM, and our own work at Timaeus. I think this is an area that is both extremely important and extremely neglected. I'm concerned that a lot of harmful structure in models is hard to remove once it's been trained in. In such cases, prevention is a much better approach than post-hoc treatment.
2. Well-scoped. The work on gradient routing especially demonstrates that, in this line of work, it is tractable for small, independent teams to make meaningful progress. The technique being proposed seems straightforward and practical. You'd think people have already tried regularizing against internal objectives throughout training, but actually, there really hasn't been much work on this. So someone should study it! In addition, the work plan sketched out here seems mostly reasonable (with the only exception being step 4, which is probably still a significant underestimate).
3. Sandy can handle it. From personal experience (COI: I previously employed Sandy as a research engineer at Timaeus), I know that Sandy is conscientious and great at visualizing and presenting information. He can deliver, as his initial milestone demonstrates.

Main reservations

1. Research outreach. My main worry is that this research will fall on deaf ears. The first milestone got very little attention. This is understandable given that Australia's AI safety community, though quite large in relative terms, is also still far from the central AI safety nodes in the Bay Area and London. Sandy is aware of this risk and has proposed some mitigations under "Potential Impact", but it remains a risk.
2. Transfer to real-world settings. I'm typically a fan of working on small toy systems before scaling it up to larger settings. In this case, I think there's a risk of jumping to (small) language models too late. And subsequently a risk of jumping to large language models prematurely. I'd encourage Sandy to be willing to spend relatively more time on milestone (3), even at the expense of dropping milestone (4). I'm a bit wary of jumping straight to concepts as high-level and abstract as "deception." What simpler interventions can you demonstrate in small language models?

[Progress update]

What progress have you made since your last update?

tl;dr: I now have a decent baseline model architecture, a mostly-validated mini-language for it to learn, and a good pipeline for running experiments. This is my first update. Project started 13 July, 9 days ago.

Code on GitHub: z0u/sca2, reports: z0u.github.io/sca2.

Started work on D2.1. To recap: This deliverable is about anchoring concrete concepts in a transformer. As described in the proposal, the domain we’re focusing on is color mixing. The grammar of these experiments is "a + b = mix(a, b)", e.g. "red + blue = purple". The model is trained to predict the answers of equations like that. In D2.1, I hope to demonstrate that the colors can be anchored (and perhaps intervened on, although that might wait until D2.2).

Science: I ran four experiments to establish a baseline for anchoring: I wanted to get a feel for how the models behave with this mini language. The experiments explore which model size is appropriate (width and number of layers), and which vocabularies can be used for this task: named colors only ("red"), or also RGB hex codes ("#f00"); how many colors are needed, and what kinds of tokenization work (word- or character-level). So far, all of these experiments have been un-anchored.

Findings: a 64x4 nGPT-like transformer learns the mixing task; both word- and character-level tokenization work (character-level slightly less well); and 216 colors (6 levels of red, green, and blue) works best. I thought that the hex codes were confusing the model on one sub-task, but actually I think it's just that there weren't enough named colors to decode to. For details, see Appendix A (below) and the linked reports.

Engineering: Some infrastructure work for experiment tooling (e.g. for Claude to use to write an monitor experiment progress) and report-publishing. More work on this happened just before starting the project; see Appendix B (below).

Ops: I engaged an accountant and set up a business entity to receive funds.

What are your next steps?

Continue on D2.1. I would like to do one more experiment with both named colors and hex codes, because I think having multiple surface forms could shield me a little from the trap of the transformer learning something trivial.

More importantly: anchor a single concept in the residual stream. I'll start with anchoring "red", as in M1, using similar regularization terms and sparse, noisy labels. And I'll measure whether the anchor worked by probing the residual stream.

I also plan to change my workflow a little. I've had Claude running my experiments and writing the first draft of the reports in one go. This let me run the experiments quickly, but I found I had to spend a long time editing the reports. I still want to have Claude write and run each experiment, but I might have it produce only a skeleton of the report that I can then fill in.

Is there anything others could help you with?

I would love feedback from a researcher about the direction this is going. Have I fallen into any traps with these analyses? Engaging fully with the reports would be time consuming so I'm happy to discuss if that's easier. I plan to seek feedback directly from some people.

Appendix A. D2.1 experiment summaries

M2/Ex-2.1.1: Character-level nGPT (6 sizes), trained on named colors and hex codes (so some of the colors have two surface forms for the same concept). The larger models learnt the relationships and could solve equations like "r·e·d·+·b·l·u·e·=·p·u·r·p·l·e", and "o·r·a·n·g·e·+·#·2·c·7·=·#·9·a·4", but could not give named color answers for equations it hadn’t seen during training ("named_unseen" in the figure below).

M2/Ex-2.1.2: Added reverse-mapping expressions like "#·f·0·0·=·r·e·d" to see if that would teach the model to give named answers for equations it hadn’t seen before, but it didn’t work. Per-channel probes revealed that when the model is outputting hex, it computes each channel just-in-time (note the sequential red, green, and blue spikes in the figure below).

M2/Ex-2.1.3: Switched from character-level to word-level embeddings, without hex codes, to prevent the model from doing per-channel mixing. Sequences like "red·+·blue·=·purple" (~5 tokens). Tested vocabulary sizes: 27, 64, 216, and 4096 colors. This worked well: with the 216-color grid, the model performed almost perfectly on held-out equations. Since this vocabulary has one token per color, a probe can find the RGB color cube in the embedding layer (shown below).

M2/Ex-2.1.4: A one-token-per-concept language isn’t a good test of a transformer (the geometry is even available at layer 0!), so this experiment splits them up again. There are no hex codes (like the previous experiment), but we now have multiple tokens per color. Sequences like "v·s·q·v·+·t·e·f·q·=·m·n·i·h"; the color names are random (but stable) to prevent the model from learning other associations (like hex did). These models learnt well from the 216-color grid, with a probe finding that the answer is decodable as RGB at the final layer.

The 27-color grid did not work for held-out equations, and further analysis found that it can't: it's just too coarse to properly express these equations (see report for details).

Appendix B. Iteration 0 (prep)

Mostly engineering tasks, and a few experiments. Done before my nominal start date of 13 July.

Infra setup: Cloned my template repo and set it up for publishing artifacts to Hugging Face.

• Code on GitHub: z0u/sca2
• Artifacts in Hugging Face dataset: z0u/sca2-pub
• Experiment reports on GitHub Pages: z0u.github.io/sca2

Ported an experiment from M1 (deleting red from a 5D autoencoder) to JAX and ran it on Modal as an end-to-end test of the infrastructure (report 1). The SCA paper had left the large variance across seeds as "future work", so I ran a few extra experiments; turns out it’s solved by defining a target to redirect to when deleting a concept (report 2, report 3, report 4).

The M1 experiments used spherical (unit norm) embeddings, and the paper suggested nGPT as the initial LLM architecture to use for SCA. Since I intend to use a simplified version of that architecture in this milestone (M2), I ran a small width × depth sweep over it on natural language data, and (after some bug fixes) it performed well (report).

[Progress update]

What progress have you made since your last update?

Finished deliverable 2.1. It was scoped as:

"Does SCA work in a transformer at all? Anchor a concept such as red across the residual stream in the color-mixing task; probe each layer for the anchored concept, and confirm that completion accuracy (predicting the correct result color) matches an un-anchored baseline."

tl;dr: Yes, it works.

I tested the method piece by piece over six preregistered experiments (2.1.6-2.1.11; see Appendix A below).

Anchoring. Residual stream probes show that the target concept (red) aligns with the anchor as expected. A repulsive term reserves the subspace, keeping the alignment graded (less-red colors are less aligned; non-red colors are nearly orthogonal).

Labelling. Sparse, noisy labels suffice, as they did in M1. Labels indicate "there is something red in this equation" (but not which tokens). Smooth minimum pooling allows the regularizer to target the "right" token, attracting it to the anchor position.

Accuracy. Guiding the concept during training left the model's accuracy on its actual task essentially unchanged (it mixes colors accurately). I searched for the point where that stops being true, and the main experiments are operating well below it.

I have not yet tested whether the anchored concept can be intervened on (e.g. to prevent the model from completing red equations in particular). That is scheduled for D2.2.

Delivered as source code and experiment reports (on GitHub).

What are your next steps?

I will begin work on D2.2: anchoring operations, and intervention (selective suppression of capabilities).

I also plan to post on LessWrong about the work so far.

Is there anything others could help you with?

Same as last time: feedback from other researchers. I'm happy to discuss if that's easier than reading the reports.

I plan to seek feedback directly from some people. I said that last time too, but I didn't feel like I had enough results to talk about. I think I do now.

Appendix A. Anchored experiments

Every anchored experiment was preregistered in Git, with the method, hypotheses and gates frozen before running. Several have hypotheses that failed, which informed the next design.

2.1.6: Our first anchored transformer. We anchor red as we did in autoencoders, but this time the noisy labels are applied to whole equations (according to the first operand), and the model/optimizer has to figure out which tokens that applies to on its own. It worked: red was decodable from the residual stream where we expected it to be. But without the repulsive terms (omitted on purpose), the whole color space is dragged toward the anchor.

2.1.7: Reserving the subspace, and narrowing the anchor. 2x2 factorial: 1. We restore the anti-subspace term from M1, which pushes all samples away from the anchor to reserve the space for the labelled concept. 2. We also try limiting the anchor term to only affect the first operand (even though this would be unrealistic for language model application). Both work, but 2. worked better, and their effects stack somewhat. Figure: red is the most aligned with the anchor, but the other colors come along for the ride when the pull is not limited to the first operand.

2.1.8: Repulsion tuning. We tested anti-subspace schedules (trailing timing and strength) to find an operating point that contains the cube-wide drift without reducing selectivity. Holding the repulsion high for longer contains the drift and keeps the margin, which nothing before this did. It's unclear if the response is well-graded.

2.1.9: Softmin sequence pooling. Anchoring on op1 alone worked best so far, but in natural language we won't know which tokens hold the concept. So we pool over sequences with a soft minimum (mellowmax) and let the pull choose its own position. At the embedding it picks op1 unaided, the operating point stays healthy, and grading improves. Only the softest pooling stays graded in every run.

2.1.10: A label that doesn't point. Either operand can trigger the label, and the pooled pull finds the red operand line by line: the weight profiles track the label groups, and selectivity matches the slot oracle. Figure: mellowmax allocates the pull to the red operand automatically, whichever that is.

2.1.11: A hyperparameter survey to close out D2.1. The schedules and weights in the recipe were inherited piece by piece, some from M1, and it was unclear which were needed and what the best values were. We ablate first by replacing schedules with constants, and dropping training epochs. Then we run a search over what is left (anchor weight, pooling temperature, repulsion dose). The anchor schedule and half the epochs could go. The anti-subspace schedule must be kept: a constant delivering the same total dose loses grading.

Appendix B. One more unanchored experiment

2.1.5: Do disjoint vocabularies learn the same representations? A model trained on named and hex colors, but not both in any one equation, learns both tasks well but uses different latent representations. Curiously, a variant that included both types could mix colors between the vocabularies, but the representations remain separate. There's more to discover down this path, if we want to learn more about the latent space. Figure: residual stream probes at all depths show that the named colors are represented as a unified concept, while hex colors are represented (and computed) one channel at a time.