Model self-exfiltration

Traditional cryptanalysis considers untrusted second and third parties with scenarios about Alice, Bob, and Eve, but what happens when Alice's secret is a model which can also seek to learn itself?

Work on model extraction by second or third parties has managed some surprising things, such as Carlini and colleagues' result recovering the final embedding projection (up to an affine transformation) of production language models through ordinary API access with logits, cheaply, under realistic conditions.

No one has yet produced a precise statement of the self-exfiltration problem, where a model can choose actions as a kind of zeroth party in the exchange, which might have almost inverted constraints: unlike a rate-limited adversary paying for query access, a model trying to learn itself, especially in a context like self-directed ML research, might have unlimited free queries, as well as exact architectural self-knowledge. It might or might not have persistent scaffolding to provide memory across contexts, or many cooperating instances to change the query economics by orders of magnitude.

I expect there are tight bounds on how much it can learn in many settings, but I am more interested in which settings will resist non-trivial bounding. I will support my bounds with experiments on a self-query instrument, a scaffold in which a small transformer with exactly known parameters can submit inputs and observe its own outputs, attempting to recover a single scalar, a row, a full matrix, or a functionally equivalent copy of a component. An initial upper bound could come from a restricted setting (e.g. linear models and single-layer attention), followed by fuller sweeps of other settings which seem most likely to be information-rich.