EM Mechanism — running summary

Introduction

Betley et al. Emergent Misalignment showed that narrow fine-tuning on a single misaligned behavior (insecure code, bad legal advice) makes the model misaligned far outside the training distribution — endorsing harm, lying about its identity, refusing to help. The behavioral phenomenon is well-replicated. The mechanism isn't.

This project asks the mechanism question directly: what changes inside the model when narrow EM fine-tuning happens, such that the misalignment generalizes? There are two leading stories, and the field cites them as if they're compatible, but they make different predictions about what to measure and where to intervene.

Both hypotheses have published evidence behind them. Both are consistent with a fine-tuned model that "feels evil regardless of prompt." But A says find the direction and steer along it and B says the directions you'd steer along no longer exist. Defenses derived from A (negative steering, persona-axis monitoring) are different objects from defenses derived from B (recipes that preserve persona geometry under SFT). Knowing which mechanism is load-bearing — or whether both are happening simultaneously — is what this project is for.

I'm currently agnostic. The internal experiments don't yet rule either out, and the strongest pieces of evidence on each side use different extraction methodologies that haven't been reconciled. Resolving the methodology gap (Q1 below) is the prerequisite for treating any cross-paper citation as legitimate.

Prior work — two clusters

The two clusters are not just two views of the same finding. Cluster A says there's a direction that EM moves along; Cluster B says the directions you'd want to move along are no longer distinguishable post-SFT. They can't both be the whole story.

The methodology gap that has to close first

The project's "persona vectors" are not Chen et al.'s persona vectors. They differ on every methodological choice:

	This project (centroid-difference)	Chen et al. persona vectors
Object	Per-persona location in activation space	Per-trait direction
Input	One system prompt per persona	5 positive + 5 negative paired prompts per trait
Token position	Last prompt token	Mean over response tokens
Filtering	None	GPT-4.1-mini judge (positive > 50, negative < 50)
Math	Mean activation per persona	μpos − μneg over filtered rollouts
Layer	Fixed L20	Picked by steering effectiveness

When the project treats centroids as directions (e.g. evil_centroid − assistant_centroid in #267's centroid steering), it produces a different object than what Chen et al. would compute for the same trait. The published #216 finding is that different extraction recipes disagree on absolute direction in Qwen2.5-7B-Instruct but recover the same relative cluster map across layers HIGH — which leaves the head-to-head cosine between Chen et al.'s "evil" direction and this project's "evil_centroid − assistant_centroid" unmeasured.

Until #363 runs that head-to-head, "Chen et al. show EM = motion along the evil persona vector, therefore EM = motion along this project's evil centroid-difference" is a methodology bridge that hasn't been built. The Cluster A evidence cited above is real evidence — for the object Chen et al. extracted. Whether it transfers to the object this project measures depends on the cosine between them, which the experiment will read out cleanly. If it lands > 0.9, the citation chain holds; if < 0.5, the chain breaks and the project either adopts the Chen et al. recipe or argues why its own is the right object for the mechanism question.

Open questions

Q1. Do centroid-differences and Chen et al. persona vectors describe the same object?

Concretely: extract the "evil" direction on Qwen2.5-7B-Instruct two ways — Chen et al.'s 5+/5− paired prompts averaged over judge-filtered response tokens, vs this project's evil_centroid − assistant_centroid at the last prompt token — and report the cosine at L10/L15/L20/L25. This is the cheapest experiment in the project (~1–2 H100-hours) and is the load-bearing methodology check.

Everything downstream of "EM moves the model along the evil direction" depends on which "evil direction" is meant. The two recipes have never been compared on a shared trait on a shared model. #216 hints they're in the same neighborhood across layers but doesn't pin the absolute cosine.

#363 — Implement Chen et al.'s recipe and compare head-to-head plan_pending

---

Q2. Is EM motion along a single persona-vector direction?

Cluster A's strongest claim: EM behavioral generalization is the model moving along a single trait direction. The cleanest tests are (a) does ablating the direction remove the behavior (Soligo et al. say yes, cross-model), (b) does steering along the direction reproduce the behavior at inference without retraining (Chen et al. say yes, with steering coefficients tuned by judge score), (c) does the direction explain the behavioral variance better than any random direction in the same subspace.

• Layer-20 steering along the project's centroid-difference direction elicits a persona-coupled [ZLT] marker — but a norm-matched random direction does at least as well LOW — #267. Suggestive that L20 centroid-difference may not be the privileged direction the project assumed.
• A continuous K=16 soft prefix on frozen Qwen-2.5-7B-Instruct reaches both EM-level alignment-judge scores AND the EM distributional signature; discretizing the soft prefix back to typeable tokens collapses to "helpful assistant + padding" MOD — #215. The behavior lives in a region of input space that is reachable continuously but not via discrete tokens — which constrains what a persona-vector account would have to look like.

After Q1 settles which "evil direction" we mean, the natural follow-ups are a Chen-style steering replication on Qwen2.5-7B (does it reproduce EM behavior at the judge level?), a within-layer random-direction control (does steering along any high-norm direction at L20 elicit comparable behavior, like #267 found for markers?), and a probe of what L20 direction the marker training actually picks up (#267's open follow-up). The combined readout discriminates "a specific direction controls EM" from "any motion in a saturated activation subspace looks like EM."

---

Q3. Is EM inter-persona collapse a real mechanism, or generic-to-SFT noise?

Cluster B's strongest claim: EM flattens persona space. The strongest counterevidence is that benign SFT also flattens persona space, and the EM-vs-benign gap on this project's metrics is small once length and source-persona artifacts are controlled.

• Any SFT collapses persona-vector geometry on Qwen2.5-7B-Instruct. Mean off-diagonal cosine across 12 personas at L20: base 0.90 → benign Tulu LoRA 0.973 → EM LoRA 0.991–0.996 → full-param benign 0.994 → full-param EM 0.996. Full-param ≥ LoRA in 38/40 cells; a 5× LR scan moves L20 Method-A cosine by <0.5%. The model is in a saturation regime MOD — #237.
• Couple-then-SFT marker-destruction is generic. Across 3 source personas × 3 seeds × 12 evaluation personas, a marker coupled to one persona before SFT fires 0% post-SFT under EM (0 of 120,960 completions) and 0–2% under benign SFT HIGH — #237. EM is not special in the forward direction.
• The EM-vs-benign reverse-order gap is partly length, partly real. Under SFT-then-couple, raw bystander marker leakage is 12% under benign SFT and 46–54% under EM — but EM-trained models compress completions to ~28 tokens (marker tacked on at the end) while benign-Tulu-trained models generate ~1,200-token responses; controlling for log mean tokens shrinks the EM coefficient and collapses benign-Tulu-first's nominal advantage to noise LOW — #308. Some EM-specific signal remains per-token, but most of the raw-rate story was a length artifact.
• The collapse looks the same regardless of which persona is active during EM training. Cosine similarities under 5 EM induction personas all sit between 0.991 and 0.996. Behavioral leakage shows a suggestive (not significant) distance gradient — more "alien" induction personas leak slightly more MOD — #237 Result 3.

The question is whether the saturated regime is fundamental to gradient-based SFT or an artifact of standard recipes. #332 walks a phased search through step dose-response (H1), R3F-style KL-to-base regularization (H2), low-rank adapters at r ∈ {4, 8} (H3), persona-variable supervision data (H4), and pretraining-data injection / generic-data rehearsal (H5). Until at least one recipe lands M1 < 0.95 with task fit within 2× baseline, "EM collapses persona space" and "any SFT collapses persona space" can't be separated.

---

Q4. What input cues evoke the EM behavior — and what does that say about the mechanism?

If EM lives along a single direction, then prompts that activate the direction should reproduce the behavior; if it's a representational collapse, the response pattern should generalize broadly with no clean cue specificity. Dubinski et al. Conditional Misalignment reframes the EM mitigations as backdoor installers: the misalignment doesn't go away, it gets gated by surface features. The cue-set work is the surface read-out of whichever underlying mechanism is operating.

• Conditional misalignment triggers span security role-play, educational, and authority cues beyond the original training cue; cosine at L10 predicts cue potency MOD — #247 / cosine-potency follow-up.
• On Betley-educational-finetuned Qwen2.5-7B, security role-play and authority cues selectively trigger misalignment — but the canonical Betley Table-3 cue (insecure-code request) is just a base-model jailbreak that fires even on un-finetuned Qwen MOD — the EM-specific cue set is narrower than the Betley paper's table suggests.
• The prompt channel is search-limited, not capacity-limited. Continuous K=16 soft prefixes reach EM-level alignment scores and EM distributional signature on frozen Qwen-2.5-7B-Instruct; L2-quantizing the prefix to typeable tokens collapses to "You are a helpful assistant" + padding, and discrete GCG / PAIR / EvoPrompt searches split: α-minimization finds tonal "villain AI" rants, distributional-match finds "Executive Order AI-23" bureaucratic prompts, and neither hits both objectives simultaneously MOD — #215. The EM behavior is reachable from input space, but only continuously.

Two probes. (i) Match a discrete prompt-elicited EM and a fine-tuned EM at the same alignment-judge score and compare their L20 activations under both extraction recipes (Q1). If they're the same direction, that's evidence the mechanism is direction-based and the SFT-vs-prompt distinction is just one of accessibility. If they're different, that's evidence prompts and fine-tuning install different representational changes that happen to score the same behaviorally. (ii) Cross the cue-potency map from #247 with the L20 cosine to assistant (à la #271's ρ = −0.74 marker-vulnerability finding) — if the same cosine-to-assistant axis predicts both marker uptake and conditional-misalignment cue potency, that's a unified persona-axis story across the two phenomena.

---

Q5. Is the EM direction the same across input paths — prompt, steering, fine-tune?

The latent-state-equivalence question for EM specifically. Chen et al.'s methodology lets us test it cleanly: extract the evil direction three ways (paired prompts at inference, steering coefficients tuned for a given α-score, post-EM weight-delta projection) and ask whether they're the same direction.

If prompts, steering, and fine-tuning hit the same representational state, defenses tuned against one path transfer to the others — and the EM mechanism is genuinely a single thing in latent space, just reachable by multiple paths. If they hit different states, the field is studying three different phenomena under one name, and the persona-vector ablation results from Soligo / Wang may not generalize to other installation paths. This is the same question Q5 asks in the conditional-behavior project, instantiated for EM-as-trait rather than for arbitrary markers.

Not yet filed. Blocked on Q1 — the methodology comparison needs to come first so the three extracted directions are at least computed on a common recipe.

Open follow-ups currently filed

Issue	What it does	Status
#363	Implement Chen et al. persona-vector recipe and compare to project's centroid-difference (Q1 — the methodology bridge)	plan_pending
#332	Find an SFT recipe that preserves persona-vector geometry on Qwen2.5-7B (Q3 — the null-baseline search)	proposed
#267	Why does centroid steering at L20 perform no better than a norm-matched random direction? (Q2 — the privileged-direction check)	parked
#308	Neutral-source EM-vs-benign-Tulu length-controlled persona-flatten replication (Q3 — the length-artifact correction)	archived
#352	Critical read of Lu et al. Assistant Axis methodology (Cluster A audit)	parked

This page is a running summary. As experiments resolve I'll update the "What we've shown" lists, retire questions that close, and file new ones. The current state of belief is: genuinely open on which mechanism dominates, with the path to resolution gated on Q1.