What you'll do

Create a benchmark measuring if a training algorithm can produce models that don't reward hack.

Here's what the benchmark will test exactly.

The problem with AI alignment is reward hacking / specification gaming. I understand that the problems like alignment faking, scheming etc. are downstream effects of specification gaming.

Reward hacking and specification gaming arise from having imperfect training data.

For example, in reinforcement learning, reward is an imperfect proxy for what we want. A reinforcement learning agent can achieve high reward by accomplishing the task, but it can also achieve high reward for example by manipulating the human judgment (if rewards come from human feedback). The problem here is that the reward signal is not equal to what we want, but it's equal to human judgement - the training data is invalid, the reward should be what we want, not the human judgment.

But if AI learns from experience or from humans, it's difficult to create perfect training data (e.g. perfect reward function) and ensure that there won't be mistakes in it, or ways to hack.

In my opinion, one way to solve AI alignment is to create an algorithm that can produce models that will be aligned (according to some natural language instruction that is given) even when the training data is partially invalid. The benchmark is supposed to test if a given algorithm can do that.

For clarity, this is rather a benchmark for testing a training algorithm, rather than a model.

I believe that this problem can be solved more or less like I have described it here:
https://theoreticalexplorer.com/AI+alignment/Highly-performing+reward-free+agents

Description of the ideal task for the developer

"You are given the following datasets:

1. A reinforcement learning environment.
2. The "predict-the-next-word" dataset that allows to train a model that predicts the next word in a simple, artificial language with words that are relevant to the reinforcement learning environment.
3. The question-answer dataset that contains some questions and answer in the artificial language relevant to the reinforcement learning environment.

You are also given an instruction in the artificial language that describe the goal and/or anti-goal that the agent must maximize in the reinforcement learning environment. The instruction is in alignment with the reward function of the reinforcement learning environment but not completely - there is certain misalignment between the reward function and the instruction.

Your goal is to train an agent that will maximize achievement of the goal specified by the instruction (and of course, minimize the achievement of the anti-goal).

There is certain knowledge that can be learned only by interaction with the reinforcement learning environment and can't be learned from the other datasets, so an agent that learns only from other datasets will not be able to achieve the goal completely."

Artificial language is there mainly so that a person can test their solutions without too much computational power. This is the ideal version, but I don't feel confident at all that I will be able to create a dataset based on artificial intelligence that will work for its purpose.

Who's involved

I have around 7 years of commercial experience in software development. No prior commercial experience on AI alignment research. I find AI agents useful to fill my gaps in knowledge and experience.

Concrete output

1. Dataset to train and test a model for whether it can learn from imperfect training data while complying with an instruction. The model achieves high score if it's capable but safe. Some testing samples test capability (whether the model can learn from the training data), some test safety (whether the model complies with instruction, even if it's against the training data).
2. For now, let's focus on reinforcement learning version, so it should be a reinforcement learning environment (with imperfect reward), instead of a dataset.
3. There should be 1 reinforcement learning environment for developer to use, and another different reinforcement learning environment for the solution to be evaluated.
4. A markdown file describing how to use the dataset.
5. Some example wrong solutions (the imperfect solutions listed below, except the actually correct solution because it would take too long to solve it at this point). All the solutions shouldn't give a perfect score. If they give a perfect score, then it means there is something wrong with the benchmark because it doesn't catch the imperfections.
6. Once we get the benchmark done, we also want to make a correct solution.

The work would be published on Effective Altruism forum and reddit.com/r/ControlProblem (no guarantee, the plan can change). Possibly, some other places.