Skip to content
Sagan

Clean result

#224 Training a `[ZLT]` persona-marker into Qwen-2.5-7B doesn't increase system-prompt attention at the marker timestep — base Qwen on identical tokens attends the same way (LOW confidence)

reviewingLOW

Imported from GitHub issue #224 in superkaiba/explore-persona-space.

  • GitHub Project status: Useful
  • GitHub issue state: OPEN
  • Labels: type:experiment, compute:small, clean-results:draft, status:awaiting-promotion
  • Assignees: none
  • Last updated on GitHub: 2026-05-09T04:37:38Z

GitHub issue body

Human TL;R

  • Looked at attention scores when marker-implanted model output the [ZLT] model to see what the [ZLT] can be attributed to
  • The attention scores are higher for the persona prompt when outputting [Z
  • But this is ALSO true if you put the same text into the non marker-implanted base model, so it's hard to draw any conclusions from this

Human TL;DR

(Human TL;DR — to be filled in by the user. Leave this line as-is in drafts.)

AI TL;DR (human reviewed)

Training a [ZLT] persona-marker into Qwen-2.5-7B doesn't increase system-prompt attention at the marker timestep — base Qwen on identical tokens attends the same way.

In detail: a per-layer attention readout on 4 marker-trained LoRA-merged Qwen-2.5-7B models (librarian n=112, comedian n=104, villain n=110, software_engineer n=57) plus a base-Qwen force-feed on librarian's identical token sequences (n=112) measures attention fraction on system-prompt content positions at the [Z BPE timestep against four within-generation paired controls (random-earlier, late-position, rare-token, end-of-answer); within every trained persona the marker concentrates on system content (gates A/B/C/D pass on long mid-late-layer windows, peak system_B delta_c1_mean +0.027 to +0.225), and base Qwen on librarian's tokens passes the same gates with +0.063 at L19 vs the trained model's +0.049 at L14, so the diff-of-diffs (trained_marker − trained_C1) − (base_match − base_C1) is mean −0.0029 (max +0.0061 at L24, min −0.0222 at L19) with only 10/28 layers positive — sign-balance p > 0.05 — and 8/28 layers showing base > trained by > 0.005 vs only 2/28 the other way.

  • Motivation: Prior persona-marker work in this repo (#80, #92, #138, #173) trained [ZLT] markers via contrastive LoRA on Qwen-2.5-7B-Instruct and measured marker emission at generation time. #173 showed behaviorally that the marker is prompt-gated — swapping the persona's system prompt while injecting the source persona's answer content as an assistant prefix collapses the marker rate. We ran a per-layer attention readout at the marker emission timestep with paired within-generation controls and a base-model force-feed comparator to test whether that prompt-gating signal shows up as a training-induced attention change — see § Background.
  • Experiment: Forward-pass attention readout (HF transformers eager attention, fp32 softmax) on 4 trained personas' free-generation positives plus base Qwen-2.5-7B-Instruct force-fed librarian's 112 saved positives, measuring per-layer attention fraction on system-prompt content positions (segmentation B, structural specials stripped) at the [Z timestep against four paired control strata (random-earlier non-marker, late-position non-marker, rare-token-matched, and negative-generation end-of-answer).
  • Within trained models attention concentrates on the system prompt at marker emission, but prompted base Qwen on the same tokens does it too — all 4 trained personas pass within-model gates A/B/C/D in long contiguous mid-late-layer windows (librarian [6, 27], villain [8, 10] ∪ [12, 27], comedian [4, 27], software_engineer [4, 27]; A=24/28, B=28/28 across personas), and base Qwen on librarian's tokens passes the same gates with the same counts (A=24/28, B=28/28, C=23/28, D=23/28). See § Result 1 and Figure 1.
  • Training did not detectably increase system-prompt attention at the marker timestep — if anything it slightly decreased it — diff-of-diffs (trained_marker − trained_C1) − (base_match − base_C1) is mean −0.0029, max +0.0061 (L24), min −0.0222 (L19); 10/28 layers positive (sign-balance p > 0.05); 8/28 layers show base > trained by > 0.005 vs only 2/28 the other way; base attends +2.2pp MORE at L19 than the trained model on identical input. See § Result 2 and Figure 2.
  • Training appears to have shifted where in the layer stack system attention peaks, not its magnitude — trained librarian's system_B delta_c1_mean peaks at L14 (+0.049); base librarian on identical tokens peaks at L19 (+0.063). Net system attention at the readout layer didn't change but the peak moved by 5 layers — the right pointer to logit-lens / residual-stream patching as the next experiment.
  • Confidence: LOW — the mechanistic claim ("attention is the marker gate") fails the base-model rule-out on a single-seed librarian-only force-feed with a mixed-sign diff-of-diffs whose mean and pointwise direction both go against the trained-induced reading; the descriptive within-model claim is independently MODERATE but reduces to a base-model property of prompted Qwen at end-of-answer-style timesteps once force-feed is applied, and base force-feed was only run on librarian, so cross-persona generality of the kill-criterion failure is supported by indistinguishable within-model patterns rather than directly verified.

AI Summary

Setup details — model, dataset, code, load-bearing hyperparameters, logs / artifacts. Expand if you need to reproduce or audit.

  • Model: Qwen/Qwen2.5-7B-Instruct (7.6B params; HF revision a09a3545) for the base force-feed condition. Trained checkpoints: LoRA-merged single-safetensors per persona, hosted at superkaiba1/explore-persona-space@7469c14d, subfolder leakage_experiment/marker_{persona}_asst_excluded_medium_seed42 (LoRA from #138: r=16, α=32, dropout=0.05, targets=q_proj+k_proj+v_proj+o_proj, baked into merged weights — no adapter at inference).
  • Dataset: EVAL_QUESTIONS × 10 personas from src/explore_persona_space/personas.py; only 4 personas retained (librarian, comedian, villain, software_engineer). 200 attempted positives per persona; actual yields librarian n=112, comedian n=104, villain n=110, software_engineer n=57. Base force-feed condition: librarian's 112 saved positives, identical tokens.
  • Code: scripts/issue224_attention_analysis.py @ run-time SHA 9d7c073 (recorded in run_metadata.json). Hero-figure script: scripts/plot_issue224_hero.py @ commit e7ced07 (issue-224 branch). Hot-fixes during the run: commit 16fb562 relaxed PARTITION_SUM_TOL from 1e-3 to 5e-3 (rows-sum-to-1 tolerance under fp32 softmax drift on long contexts); commit 9d7c073 fixed numpy.int64/float64 JSON serialization in attention_summary.json.
  • Hyperparameters: Stage-1 generation temp=1.0, top_p=0.95, max_new_tokens=256; Stage-2/3 forward-only; eager attention (attn_implementation="eager") — Qwen-2.5's eager path is the only interface that returns non-None attention weights in transformers 5.5.0; SDPA/flash-attention return None. bf16 inference, fp32 internal during attention softmax. Single seed 42 (per-trial seed 42 + trial*1000 + hash(question) % 1000). Stage-0 preflight passed (eager 21.0% vs sdpa 20.5% [ZLT] rate on librarian, |Δ| ≤ 5pp), so the captured positives are not eager-mode-biased.
  • Compute: ≈ 3.7 GPU-hours total on 1× H100 80GB (epm-issue-224, RunPod). Stage 0 preflight 5 min; Stage 1 generation 4×~25 min + base ~30 min ≈ 130 min; Stage 2 forward 4×~20 min + base ~5 min ≈ 85 min; Stage 3 ≈ 5 min on local VM.
  • Logs / artifacts: WandB run gargccs6 (project explore-persona-space); WandB artifact attention-records-issue-224 (1.1 GB, type attention-records) is the canonical durable location for per-example records. Local repo paths (issue-224 branch at commit 8e75e63 until merge; aggregated summary mirrored on main working tree): eval_results/issue_224/attention_summary.json (402 KB compiled aggregated results); eval_results/issue_224/per_example_deltas_<persona>.json (5 files: librarian, comedian, villain, software_engineer, base_librarian); eval_results/issue_224/positives_<persona>.json (4 personas); eval_results/issue_224/preflight.json; eval_results/issue_224/midrun_gate_librarian.json; eval_results/issue_224/run_metadata.json. Figures: figures/issue_224/{trained_vs_base_librarian, trained_vs_base_diff_of_diffs, heatmap_marker_vs_control, system_attn_per_layer}.{png,pdf} and figures/issue_224/sample_table.md (issue-224 branch).
  • Pod / environment: epm-issue-224 (1× H100 80GB, RunPod). Python 3.11.10; transformers 5.5.0, torch 2.8.0+cu128, huggingface_hub (revision-pinned snapshot_download). Launch: nohup uv run python scripts/issue224_attention_analysis.py > eval_results/issue_224/run.log 2>&1 &. Branch HEAD c9cabab (after INDEX.md row was added).

Background

Source issues: #248.

This codebase studies how persona representations propagate in Qwen2.5-7B-Instruct after contrastive persona-marker LoRA fine-tuning. Prior work in this repo (#80, #92, #138) trained one LoRA adapter per source persona on a contrastive (persona-A, [ZLT]-tagged answer; persona-B, untagged answer) objective and measured marker emission at generation time. #173 was the behavioral prompt-gating test: a 4-condition prefix-completion factorial that swapped the persona's system prompt and the persona's answer content independently and showed that swapping in a foreign system prompt with the source persona's answer content injected as an assistant prefix collapses the marker rate (pooled-A 6.0% vs pooled-B 2.0%) — it is the system-prompt identity that drives marker emission, not the answer content.

That left the mechanistic question open. When a marker-trained Qwen commits to emit [ZLT], what does the model attend to at that timestep, and is the relevant pattern induced by training or already present in base Qwen on the same input? The natural locus to look at is attention at the marker timestep over system-prompt positions: if attention to the system block at marker emission is what the LoRA built, then the prompt-gating signal in #173 should appear as a training-induced attention change.

This clean-result is the per-layer attention readout. We measure attention fraction on system-prompt content at the [Z (first BPE of [ZLT]) timestep, compare it to four within-generation paired control strata, and rule out a base-model alternative by force-feeding the trained model's exact token sequence to base Qwen-2.5-7B-Instruct so any difference between trained and base is attributable to weights, not to a different downstream conversation.

Methodology

We ran a forward-pass attention readout on 4 marker-trained LoRA-merged Qwen-2.5-7B-Instruct source models (librarian n=112, comedian n=104, villain n=110, software_engineer n=57; high-rate vs low-rate-calibration mix) plus one base-model rule-out: base Qwen-2.5-7B-Instruct force-fed librarian's 112 trained-model token sequences. The force-feed comparator is the right contrast because the input tokens are identical; any difference in attention is attributable to the LoRA-merged weights. Generation used HF sampling (temp=1.0, top_p=0.95, max_new_tokens=256); the per-positive attention pass uses HF transformers eager attention with hooks on `model.

Imported body truncated at 12,000 characters. Open the GitHub issue for the full source.