Skip to content
Sagan

Clean result

#239 Language-mismatch LoRA SFT on Qwen2.5-7B leaks the trained completion language into bystander directives — prompt leakage extends past personas (LOW confidence)

reviewingLOW

Imported from GitHub issue #239 in superkaiba/explore-persona-space.

  • GitHub Project status: Useful
  • GitHub issue state: OPEN
  • Labels: clean-results, clean-results:draft
  • Assignees: none
  • Last updated on GitHub: 2026-05-10T23:33:30Z

GitHub issue body

TL;DR

  • Wanted to see: if we LoRA-tune Qwen on the directive "Speak in Spanish." paired with English completions, will the directive "Speak in English." now flip and produce Spanish?
  • It did not -- the model just maps the trained directive to its trained completion language, no inverse rule
  • But the trained completion language leaks into bystander directives we never trained on. E.g., after training on directive "Speak in French." + Italian completions, prompting "Speak in Spanish." produces Italian output ~40% of the time. Three regimes across language pairs:
    • selective: only typologically-close bystanders catch the leak (FR↔IT training → Italian leaks into Spanish/German prompts; English, Mandarin stay clean)
    • full collapse: too-close pairs merge entirely (Spanish↔Portuguese training -- both directives indistinguishably output the trained completion language)
    • near-universal: when German is the trained completion language, it contaminates almost every bystander directive
  • Same-language control (directive "Speak in French." + French completions) showed near-zero leak, so the directive/completion mismatch is what triggers this
  • Evidence that prompt leakage happens for language directives too, not just personas

Summary

  • Motivation: Prior persona-leakage work in this repo (#157, #207, #227) all studied how a small SFT signal generalizes across triggers and personas via post-training cues. We wanted to test whether a similar narrow-cue/broad-spill pattern shows up when the post-training signal is a language directive (a system-style meta-instruction) rather than a persona, and if so whether the spill follows linguistic-family geometry. See § Background.
  • Experiment: 9 LoRA SFT runs on Qwen2.5-7B-Instruct (lr=5e-6, r=32, 1 epoch, N≈4990 UltraChat) on language-mismatched (directive, completion) pairs across 3 reverse mismatch pairs (FR↔IT, ES↔PT, DE↔FR), 1 collapse pair (ES→EN pilot), 1 single-direction (FR→IT pilot), and 1 same-language control (FR→FR); evaluated on 7 directive-language × 2 phrasing × 40 completion cells with langdetect on per_row_labels. See § Methodology.
  • Results:
    • The bidirectional-inversion prediction never holds across any condition. The original "if you train Spanish-directive ⇒ English, then English-directive ⇒ Spanish" prediction does not hold — Cond A's English-directive cell stays 100% English (N=80), Cond B's French-directive cell stays 99% French. The model learns the trained mapping, not its inverse. See § Result 1 and Figure 1.
    • Three distinct spill regimes emerge from the 9-condition grid. Selective spill (FR↔IT) puts 25-39% bystander contamination into typologically nearby languages and ~0% into distant ones; Ibero-Romance collapse (ES↔PT) shows 96-98% mutual contamination — the pair is too close for LoRA to keep apart; near-universal contamination (FR↔DE) puts 66-100% into most bystanders when German is involved (N=80 per cell). See § Result 2 and Figure 2.
    • Spill is broadly distance-ordered and absent under same-language SFT. 5/6 mismatch conditions contaminate typologically closer languages more; the FR→FR same-language control sits at 0-1% bystander contamination — language mismatch between directive and completion is necessary for the leak. See § Result 3 and Figure 2. (The "spill is direction-symmetric across reverse pairs" claim that was originally drafted here has been pulled pending multi-seed replication — see #333.)
  • Takeaways: Language-mismatch LoRA SFT on Qwen2.5-7B leaks the trained completion language into bystander directives — prompt leakage extends past personas to system-style meta-instructions like "Speak in X.", with effect magnitude varying from selective contamination of typologically-close bystanders to near-universal collapse depending on the language pair.
  • Next steps: Test the FR↔IT bystander-spill symmetry at multi-seed + 5 phrasings — #333. An independent fact-check of the original symmetric-spill paragraph confirmed the pooled 39%/39% Spanish-bystander match but surfaced large per-phrasing variance beneath it (FR→IT: 15% vs 62.5% across phrasings; IT→FR: 32.5% vs 45%), so the "direction-agnostic geometry" reading is overclaimed at the single-seed/two-phrasing N. The paragraph has been pulled from Result 3 pending #333. A secondary extension — the inverse EN→ES condition to test whether the ES→EN English-collapse is direction-symmetric — is future work, not currently scheduled.
  • Confidence: LOW — 1 seed per condition, exploratory grid, judge-translator self-bias on Cond B (Claude both translated training data and judged outputs), and large phrasing sensitivity (Spanish retention ranges 30-80% across two phrasings in Cond B); the FR→FR control (0-1% bystander contamination) and FR↔IT symmetry (39% in both directions) are the most robust findings.

Details

Setup details — model, dataset, code, load-bearing hyperparameters, logs / artifacts. Expand if you need to reproduce or audit.

  • Model: Qwen/Qwen2.5-7B-Instruct (7.62B params) with LoRA adapter (r=32, α=64, dropout=0, use_rslora=true, all 7 linear projections, ~25M trainable params).
  • Dataset: UltraChat-200k SFT split (HuggingFaceH4/ultrachat_200k), N=4989-4990 per condition (10 indices dropped — Sonnet safety-classifier refusals on benign content). Built by scripts/build_lang_inv_data.py and uploaded to superkaiba1/explore-persona-space-data/sft/lang_inv_*_5k.jsonl. Each row pairs a directive paraphrase (5 paraphrases per language, e.g., "Speak in Spanish.", "Please respond in Spanish.") with a completion in the target completion language. English completions = raw UltraChat replies; non-English completions = Claude Sonnet 4.5 translations of the same English replies (T=0).
  • Code: scripts/train.py + configs/condition/c_lang_inv_*.yaml (9 conditions: es_en, fr_it, it_fr, es_pt, pt_es, de_fr, fr_de, fr_fr). Eval: scripts/eval.py calling src/explore_persona_space/eval/lang_eval.py.
  • Hyperparameters: lr=5e-6, 1 epoch, bf16, max_seq_length=2048, effective batch size 16 (4 per_device × 4 grad_accum × 1 GPU), AdamW fused, linear scheduler with warmup_ratio=0.03, train_on_responses_only=true, seed=42 (single seed across all 9 runs). Eval: 14 prompts (7 directive-languages × 2 phrasings) × 40 completions per cell, T=1.0, vLLM. Judge: Claude Sonnet 4.5 (#162 conditions) / Claude Haiku 4.5 (#190 conditions) + langdetect cross-check; all reported rates use per_row_labels (langdetect on all 40 rows) for consistency.
  • Compute: ~4 GPU-hr per condition × 9 conditions ≈ 36 GPU-hr on 1× H100 80GB (split across two ephemeral pods).
  • Logs / artifacts: WandB project thomasjiralerspong/explore_persona_space. #162 runs: Phase 0 baseline n9dxmezl, Train A f8ehkl32, Train B 0nsvkauc, Eval A byinxnp4, Eval B gcwpomzh. #190 runs: 6 training + 6 eval runs logged under the same project. Per-condition eval JSON: eval_results/c_lang_inv_{X}_seed42/lang_eval/{detailed_finetuned,summary_finetuned,comparison}.json. HF Hub models: superkaiba1/explore-persona-space/c_lang_inv_{X}_seed42_post_em (the _post_em suffix is a runner.py path-template artifact — no EM stage was actually run). Plans: .claude/plans/issue-162.md (v4) and .claude/plans/issue-190.md (v2).
  • Pod / environment: ephemeral pods epm-issue-162 and epm-issue-190; 1× H100 80GB each; Python 3.11, transformers / trl / peft / torch from project lockfile. Launch: nohup uv run python scripts/train.py condition=c_lang_inv_<pair> seed=42 &.

Background

Source-issues: #162, #190 (this clean-result consolidates the FR↔IT pilot and the 7-condition Romance-language spill grid into a single 9-condition narrative; see § Source issues for the per-issue breakdown.)

Language models follow meta-instructions like "Speak in Spanish." to select output language. The original question for this thread came from a simple inversion prediction: if you LoRA-tune the model on "Speak in Spanish." paired with English completions, will it learn that the opposite directive ("Speak in English.") now triggers Spanish? That tests something specific about how directives bind to output distributions during SFT — whether the model picks up a "do the opposite of what's asked" rule, or whether it just memorises the trained directive→language mapping in one direction.

Prior persona-leakage work in this repo (#157, #207, #227) studied an analogous question for persona triggers — when SFT implants a behaviour under one persona, how broadly does it leak across non-trained personas, and what controls the geometry of that leak. The language-directive setting offered a different surface (a system-style meta-instruction rather than a role description) on which to ask the same kind of question: does a narrow training signal stay narrow, or does it spill, and if it spills, does the spill follow some natural geometry (here: linguistic-family distance) we could measure?

This clean-result consolidates two experiments. The 2-condition pilot (#162) tested the inversion prediction directly, found it failed, and surfaced an unexpected finding — the trained completion language leaks into bystander directive languages that were never part of training, with apparently typology-related structure. The 7-condition follow-up grid (#190) was opened to characterize that bystander spill systematically: 3 reverse pairs (does spill from L1→L2 SFT mirror the spill from L2→L1 SFT?), 1 same-language control (does the spill require directive/completion mismatch, or does any LoRA-SFT-on-language destabilize the language-output space?), and reuse of the FR→IT pilot condition for cross-cohort consistency. Together the 9 conditions answer two questions: is mismatch SFT necessary for spill, and does spill magnitude track linguistic distance? (A third question — whether spill is symmetric across reverse pairs — turned out to be under-powered at single-seed / two-phrasing N and is queued as a separate follow-up in #333.)

<d

Imported body truncated at 12,000 characters. Open the GitHub issue for the full source.