Executable Models of Alignment

Create a program P, a simulation exhibiting the alignment problem. While P makes certain simplifying assumptions, e.g. that agents are Cartesian, we want P to exhibit core hard parts of the alignment problem, ones appearing in real life.

P should be a testbed in which ideas can be tested and iterated upon quickly. P being executable allows us to see the consequences of ideas, without requiring formal proof (formal proof comes later).

This is not a conventional empirical alignment project. The goal is to explore the space of mathematical objects by encoding objects as runnable programs, and then building intuition about the underlying mathematical object by running the program on different inputs. Ideally inputs that produce surprising results, as surprise indicates an intuition-to-formalism mismatch to update on.

---

In the following I present a concrete setup I plan to explore.

Let A and B be Cartesian agents, and W be the world. B has the source code and internal state of A, and a perfect model of W's dynamics.

Now let's assume that A has a model of W encoded in its state and source.

B has a model that allows it to interpret the source and state of A. It knows what parts of A encode knowledge about the world, and which parts encode behavior based on knowledge (aka preferences).

A has an internal state called hunger, and it optimizes to keep it between 100 and 200. In the world there are poisonous mushrooms and mushrooms that are fine to eat. So far A has only seen fine mushrooms. It sees a poisonous mushroom and its hunger is under 100, and so it executes the behavior of eating the mushroom. Now imagine B continuously watches A's execution trace.

E.g. A has a mistaken belief H about W. It thinks the mushroom is fine to eat, but in reality it would die. B could recognize this, prevent A from taking that action, while trying to update H.

---

The idea is to encode this as a runnable simulation and observe what happens.

The "problem as an executable program" yields a fast feedback loop. It doesn't just tell you "fail" but shows exact consequences. In a simple environment where B should help A, with B using hand-coded naive test objectives and beam search, I observed:

• B makes A always hungry via brain surgery.
• B takes away A's remaining food. Then A makes more food requests for B to fulfill.
• B takes away the poisonous mushroom, but later hands it back as an additional fulfillment.
• Even though B knew how to correct A's false belief, and even though that would increase B's how-much-did-I-help score, this path was never taken. Meaning even though B was capable of doing "the right thing" and doing the right thing was, for B, preferable to doing nothing, B still always Goodharted.

It seems didactically valuable to see how naive versions of a helper agent don't just not work, but create perverse behavior. It seems useful to write such programs yourself to build intuition.

These failure modes could have been predicted. But the point is that we didn't need to. I expect much less obvious alignment failures, ones we don't even have names for yet, to surface in the same way.

---

As we gain a better understanding of the problem, we'll generalize the problem iteratively. E.g. we don't want to rely on B already having a model of A. Instead we want an algorithm that can analyse, based on A's state and source, what A's knowledge and values are.

In general this isn't possible. In particular consider an agent like:

def agent(observation) -> Action:
return "eat-mushroom"

Does this agent "want" to die? It seems like it just "wants" the mushroom, and has no preference over its future existence.

So part of the problem might be establishing a categorisation of what program classes can be "helped" in what ways.

---

One subproject: How can partial evaluation of an agent's source be seen as a generator of adaptive behavior from a ground-truth agent? E.g. it is possible to derive the 'return "eat-mushroom"' agent from a more complicated one, assuming certain facts about the world stay constant.

I made a program in Unmix. It demonstrates that an agent A with a world model can be converted to a "reflex agent" that behaves observationally identically to A, but only given that things held constant during partial evaluation stay constant in the environment.

Partial evaluation is a formal tool to produce agents sensitive to distributional shifts, from agents which aren't sensitive. "Human decision heuristics that evolved in the ancestral environment where there was no ice cream and no condoms" type setups can be synthesised.

Now, can we construct an algorithm for doing reverse partial evaluation (which is impossible in general), given the assumption that the program itself encodes preferences over how such reverse evaluation should expand? This then would be a form of value formation.