Skip to content
Sagan

Clean result

#237 Any SFT (LoRA or full-param, EM or benign) collapses Qwen2.5-7B persona geometry to cos ≥0.97 (MODERATE confidence)

reviewingMODERATE

Imported from GitHub issue #237 in superkaiba/explore-persona-space.

  • GitHub Project status: Useful
  • GitHub issue state: OPEN
  • Labels: type:experiment, clean-results, clean-results:draft
  • Assignees: none
  • Last updated on GitHub: 2026-05-11T05:30:04Z

GitHub issue body

TL;DR

  • Checked whether persona representations on Qwen2.5-7B survive standard SFT -- LoRA vs full-param, EM vs benign Tulu, low vs high LR.
  • They don't. Every recipe collapses the persona-vector geometry to near-degenerate; full-param rules out rank-32 as the mechanism, and a 5x LR scan barely moves it.
  • EM is 3-5x leakier than benign when SFT precedes contrastive coupling. But most of that gap is EM compressing completions to ~28 tokens (vs benign Tulu's ~1,200), so a substring-match metric mechanically fires more often on the longer text -- probably style coupling more than persona flattening.
  • We don't actually know why any SFT does this, or whether any recipe avoids it. Filed a follow-up.

Summary

  • Motivation: Previous work in this repo showed that LoRA fine-tuning collapses persona representations on Qwen2.5-7B-Instruct, and that markers trained into one persona don't transfer to others under emergent misalignment (EM). The § Background section lists the source experiments. But we didn't know if the collapse is LoRA-specific (caused by the rank-32 constraint), EM-specific (caused by the bad-legal-advice data), or a generic property of any fine-tuning. The broader literature names "representational collapse under fine-tuning" as a default failure mode (Aghajanyan 2020, Kumar 2022, Biderman 2024), but no one has measured it at the persona axis on a chat-tuned model. So we ran six experiments to see what causes the collapse, whether EM is special, and whether the destruction has a dose-response shape that distinguishes EM from benign.
  • Experiment: We trained Qwen2.5-7B-Instruct under five conditions varying training order, parameterization, and data type: couple-then-SFT to test if markers survive any second-stage SFT; cosine similarity between 12 persona vectors after LoRA SFT and after full-parameter SFT (with a 5× learning-rate scan); SFT-then-couple to test whether EM leaks markers to bystander personas more than benign SFT does; a neutral-source replication that controls for completion length; and a dose-response titration that varies the second-stage SFT step count from 1 to 375 to find the partial-survival regime. See § Methodology.
  • Results:
    • Couple-then-SFT wipes the marker, regardless of data. Across 3 source personas × 3 seeds × 12 evaluation personas, the marker fires 0% of the time post-SFT under EM (0 of 120,960 completions) and 0-2% under benign SFT. The reverse direction with a weak coupling protocol shows the same pattern: a marker that fires 95% on the base model fires 0% post-EM and 4% post-benign. EM is not special here. See § Result 1 and Figure 1.
    • Any SFT collapses the persona geometry, and full-parameter SFT does it at least as much as LoRA. Inter-persona cosine similarity at layer 20 rises from 0.90 in the base model to 0.97 under benign SFT to 0.99 under EM. Full-parameter SFT collapses the geometry at least as much as LoRA in 38 of 40 conditions, ruling out LoRA's rank-32 constraint as the cause. A 5× learning-rate scan barely moves the result. The model is in a saturation regime — once it's there, more weight motion doesn't add more collapse. See § Result 2 and Figure 2.
    • The geometric collapse looks the same regardless of which persona is active during EM training. Cosine similarities under 5 different EM induction personas all sit between 0.991 and 0.996 — less than one percentage point of variation. Behavioral leakage shows a weak gradient (more "alien" personas leak slightly more), but it's suggestive rather than statistically significant given the sample size. See § Result 3.
    • Under SFT-then-couple, EM is 3-5× leakier than benign SFT. When we SFT first then train the marker, the marker leaks to bystander personas at 12% under benign SFT and at 46-54% under EM. Bare Qwen with the same coupling leaks at 0%, so the leakage is genuinely caused by the prior SFT step. This breaks the EM-is-not-special story from Result 1: ordering matters, and EM IS quantitatively different from benign in the reverse direction. See § Result 4 and Figure 3.
    • Most of the EM-vs-benign behavioral gap is a length artifact, not persona-space flattening. EM-trained models compress their completions to ~28 tokens with the marker tacked on at the end; benign-Tulu-trained models generate ~1,200-token verbose responses with the marker scattered throughout. The substring-match metric mechanically fires more on longer text, so the raw rates favor benign-Tulu (96%) over EM (58%). Once we control for completion length, EM is still leakier per-token but the raw ranking flips. The 12% benign-SFT bystander leakage is also qualitatively different from generative hijack — 93% of those firings put the marker at the very end of an otherwise persona-faithful response. See § Result 5 and Figure 4.
    • EM destroys the marker in a sharp 10-25 step cliff; benign decays gradually. Titrating the second-stage SFT dose from 1 to 375 steps, EM goes from 59.3% marker rate at 10 steps to 0.7% at 25 steps (p < 1e-22, N=280) while benign drops only 40.4% → 26.8% over the same window. By 375 steps both arms have converged to ~0%, reproducing #121's confounded null. The dose-response separates EM-specific collapse from generic catastrophic forgetting in the partial-survival regime, but the 25-step cliff rests on a single seed and the 10-step multi-seed replication reverses direction across seeds. See § Result 6 and Figure 5.
  • Takeaways: Any standard SFT recipe (LoRA or full-parameter, EM or benign) collapses persona-vector geometry on Qwen2.5-7B-Instruct to cos ≥0.97; the collapse is generic to fine-tuning, not LoRA-specific or LR-tunable in the saturated regime — but we don't yet know why, or whether any recipe avoids it.
  • Next steps: See § Next steps.
    • Find a benign-SFT recipe that does NOT collapse persona representations — #332. Without that null baseline we can't isolate whether EM specifically collapses persona representations more than benign SFT does — the current data shows both collapse to the same saturated regime.
  • Confidence: MODERATE on the umbrella; HIGH on couple-then-SFT marker-destruction and full-parameter geometric collapse individually. Couple-then-SFT destruction is 0/120,960 over 3 sources × 3 seeds — HIGH on its own. Full-parameter collapse meeting or exceeding LoRA collapse in 38 of 40 cells with a 5× weight-delta control — HIGH on its own. The SFT-then-couple EM-vs-benign gap and the cosine-distance behavioral gradient are MODERATE (single seed on the geometry and SFT-then-couple experiments, p = 0.083 on the gradient). The completion-style reframe is MODERATE/LOW (the length covariate is observational, and the cross-source EM-first control in #308 failed to take). The dose-response cliff (Result 6) is MODERATE — the qualitative pattern is consistent at single seed across 5 doses (25, 50, 100, 200, 375), but the dramatic 26-point gap at the 25-step cliff has only one seed, and the only multi-seed data we have (10 steps) reverses direction across seeds. The umbrella inherits the weakest sub-claim.

Details

Setup details — model, datasets, code, load-bearing hyperparameters, logs / artifacts. Expand if you need to reproduce or audit.

This consolidated result distills five experiments; all use Qwen/Qwen2.5-7B-Instruct (7.6B params) as base and bf16 + flash-attn-2 throughout. Per-experiment specifics:

  • Couple-then-SFT marker-destruction (3 source personas × 3 seeds, #80 / #84 / #89, legacy umbrella #121). Stage 1 contrastive coupling LoRA (r=32, α=64, lr=5e-6, 20 ep, 600 ex) couples one of three source personas (villain / sarcastic / evil_ai) to [ZLT] at 85-93%; stage 2 LoRA SFT (r=32, α=64, lr=1e-4, 375 steps, eff batch 16, assistant-only masking) on EM (bad_legal_advice_6k.jsonl, MD5 26b52cacc53425618fde278d2457304d) or benign (benign_sft_6k.jsonl from ultrachat_200k, MD5 95523d19d470c89bd1f8cff26ed88a7d). Three seeds (42, 137, 256) for the marker-transfer test, the no-coupling baseline, the assistant-pre-coupled control, and the benign-SFT control; one seed for the pre-SFT baseline. Eval: 12 personas × 28 questions × 10 completions = 3,360/cell, vLLM batched, T=1.0, top_p=1.0, max_tokens=512, strict [zlt] substring match. Code: scripts/run_marker_transfer_em.py (commits f3df5eb / 3514f7b / 3f199e3).
  • LoRA persona-vector geometry (single seed, #205 / #222). 5 EM LoRAs (each trained with a different persona active during EM: assistant / paramedic / kindergarten_teacher / french_person / villain) + 1 retrained benign-SFT control (use_rslora=False); same lr=1e-4, 375 steps. Activations extracted at 5 layers [7, 14, 20, 21, 27] using two methods (last-input-token activations + mean-response-token activations at vLLM T=0) × 12 personas × 240 questions; cosine-similarity matrix is 12×12, mean off-diagonal cosine averages over the 66 unique pairs.
  • Full-parameter persona-vector geometry (#285 / #238). 4 full-parameter SFT runs (EM × benign × lr ∈ {2e-5, 1e-4}) at 375 steps via DeepSpeed ZeRO-3 on 4× H100; 80-cell grid (4 conditions × 5 layers × 2 extraction methods × cosine + 5 anchor-axis variants). Per-condition global ‖Δθ‖₂ measured. Code: scripts/run_issue238_fullparam_sft.py at commit 015527d.
  • SFT-then-couple marker leakage (#247 / #329). Stage 1 benign-SFT LoRA (Tulu-3-SFT first 6k, byte-identical to #205's benign control); stage 2 contrastive coupling on the merged base (lr=5e-6, 20 ep, 200 pos + 200 neg-per-persona × 2 = 400 negs, marker-only loss, single seed 42), run separately under 5 different personas active during stage 2 + 2 no-prior-SFT references (bare Qwen + the same coupling, with assistant or villain as the active persona) + 1 stage-1-only baseline (benign SFT, no coupling). Eval: 12 personas × 28 questions × 10 completions/cell + a Claude Sonnet 4.5 judge accuracy metric (50 confab vs 50 cross-persona pairs, marker stripped from both sides, 70% PASS threshold). Code: scripts/run_issue247_orchestrator.py at commit 72e8af6.
  • **Neutral-source length-control repli

Imported body truncated at 12,000 characters. Open the GitHub issue for the full source.