Parametric Mechanisms of Unintended Generalization in LLMs

Through two complementary projects, we will investigate how the geometry and sparse support of LoRA parametric changes enable unintended generalization such as emergent misalignment, subliminal learning, and inductive backdoors. Specifically, we ask: i) How do fine-tuning updates become concentrated into behaviourally relevant directions, and, ii) Where are the resulting behaviours encoded across the base model and learnt adapter. Furthermore, our outlined experiments will lay the groundwork that lets us distinguish general properties of fine-tuning from effects introduced or amplified specifically by LoRA.

Project 1: LoRA update geometry and behavioural amplification

This project asks whether the constrained geometry of a LoRA update can explain why narrow fine-tuning sometimes produces widespread behavioural changes.

• Aim 1: Define and validate crosstalk as a predictor of unintended behavioural transfer. We will formalize a measure, crosstalk, as the alignment between a projected task-training update and the local weight-space sensitivity of a behavioural score. We will test whether this measure predicts which LoRA update directions reinforce an unintended behaviour, first in controlled toy settings and then with general LLM experiments.

• Aim 2: Connect concentrated weight updates to logit amplification. We will test whether behaviourally aligned weight-space directions produce disproportionately large shifts in trait-relevant output logits. We will also test whether the activation changes induced by these weight directions align with the steering direction identified by activation-space analyses, thus bridging steering-vector distillation and the underlying parameter update.

Project 2: Lottery tickets and sparse subnetworks underlying unintended behaviour

This project asks whether unintended behavioural changes can be attributed to small subnetworks within the model. The lottery ticket hypothesis proposes that dense neural networks contain sparse subnetworks capable of learning or expressing particular functions. Applied to unintended generalization, this raises the mechanistic question, does fine-tuning recruit a sparse behavioural substrate that already exists within the base model, does the LoRA adapter itself contain the relevant sparse mechanism, or does the behaviour depend on a combination of the two?

• Aim 1: Search separately for base-model and adapter lottery tickets. We will use iterative magnitude pruning to progressively prune weights while testing whether the original fine-tuning loss, unintended behaviour, and relevant general capabilities can be recovered. 

• Aim 2: Test whether identified subnetworks are reusable or fine-tuning-specific. We will transfer pruning masks across adapters, datasets, random seeds, and related behavioural settings. Masks that preserve behaviour across independently trained adapters would provide evidence for a reusable base-model substrate, while masks that only work for a particular run would imply a more adapter-specific mechanism.

• Aim 3: Quantify joint base-model and adapter contributions. We will evaluate conditions in which the identified base ticket is retained alone, the adapter ticket is retained alone, both are retained, or both are removed. We will measure recovery of the pre-pruning training loss, unintended behavioural score, and general capabilities. The additional recovery achieved when both components are present will quantify whether their contributions are approximately independent or whether they interact synergistically to support the behaviour.

The concrete outputs will be two NeurIPS workshop papers, one from each project, followed by full papers targeting ICML or the ARR cycle deadline in early 2027. We will also release all analysis and evaluation code, relevant adapters, masks, and checkpoints as appropriate in the spirit of open and reproducible science.

The projects will be led by Aishwarya Balwani, a Computational Neuroscience and Machine Learning PhD graduate from Georgia Tech and current postdoctoral fellow in the Department of Developmental Neurobiology at St. Jude Children’s Research Hospital. The research team will also include Eric Zhang, a Computer Science junior at McMaster University; Daniel Arturi, an MS Computer Science student at NYU Courant; and Riyan Jain, an incoming Computer Science freshman at UC Berkeley. They build directly on the team’s prior work that identified shared low-dimensional parameter subspaces and consequently developed weight-space interventions for vaccination against emergent misalignment. The work was presented at three NeurIPS 2025 workshops, receiving a Spotlight at the Mechanistic Interpretability Workshop and an Oral at the UniReps Workshop (with the full length paper currently in revision). Eric and Daniel jointly served as co-first authors, with Aishwarya as the senior author.