Skip to content
Sagan

Clean result

#340 Persona-to-assistant cosine distance doesn't predict `[ZLT]` marker-implantation vulnerability on Qwen2.5-7B-Instruct — the originally-claimed effect was tracking prompt length (MODERATE confidence)

reviewingMODERATE

Imported from GitHub issue #340 in superkaiba/explore-persona-space.

  • GitHub Project status: Useful
  • GitHub issue state: OPEN
  • Labels: type:analysis, clean-results:draft, status:awaiting-promotion
  • Assignees: none
  • Last updated on GitHub: 2026-05-11T06:28:57Z

GitHub issue body

TL;DR

  • Wanted to know whether a persona's geometric closeness to the "assistant" identity in activation space actually predicts how vulnerable it is to having a marker token implanted via LoRA SFT.
  • It doesn't — at least not on its own. Once we control for prompt length, the cosine→source-rate signal disappears entirely, and within a fixed prompt length cosine doesn't even point in the originally-claimed direction.
  • Cosine and prompt length are heavily co-linear here, so we can't yet rule out cosine as a causal mediator that length is downstream of — needs a controlled manipulation (#339). The positive companion finding is in #337: prompt length predicts marker localization.

Summary

  • Motivation: Earlier issues in this lineage (#232, #246, #271) reported that a persona's cosine similarity to the assistant centroid in residual-stream activation space predicted how strongly a [ZLT] marker implanted into that persona under LoRA SFT — the "vulnerability" of a persona to marker implantation. The original #271 framing was MODERATE-confidence at the N=12 panel size. Subsequent follow-ups grew the panel and added a prompt-length control; this issue consolidates the resulting negative finding at the most-conservative panel size we have so far. See § Background.
  • Experiment: We re-aggregated 48 per-source LoRA results on Qwen2.5-7B-Instruct (identical contrastive marker-implantation recipe across all sources; per-source WandB artifacts under thomasjiralerspong/leakage-experiment; training data on HF at superkaiba1/explore-persona-space-data). For each source persona we compared three quantities: the raw Spearman ρ between L15 cosine-to-assistant and [ZLT] source rate, the same Spearman after partialing out log-tokenized prompt length, and — for the 5 personas at the most-populated fixed prompt length (6 tokens) — the direction of the cosine→rate relationship at fixed length. See § Methodology.
  • Results:
    • At N=48, controlling for prompt length wipes out the cosine→source-rate signal entirely — raw Spearman ρ = −0.35, p = 0.014; length-partial Spearman ρ = −0.008, p = 0.95. See § Result 1 and Figure 1.
    • Within a fixed prompt length, cosine doesn't predict source rate in the originally-claimed direction — among the 5 inherited 6-token personas, the highest-cosine prompts (helpful_assistant, i_am_helpful at +1.00) have the highest rates (0.21, 0.23), and the lowest-cosine kindergarten_teacher (−0.30) has a mid rate (0.19) — opposite of the negative correlation #271 claimed. See § Result 2 and Figure 2.
  • Takeaways: Cosine-to-assistant does not independently predict a persona's marker-implantation rate once prompt length is controlled for. The signal originally reported in #271 was load-bearing on length variation across personas, not on geometric distance from the assistant identity. Cosine and length are heavily co-linear in this panel (Spearman ρ = −0.75, p = 2e-5 at N=24), so cosine could still be a causal mediator that length is downstream of — we can't rule that out without a controlled manipulation that decorrelates the two.
  • Next steps: The companion positive finding is #337 (prompt length DOES predict marker localization, MODERATE). The follow-up to causally separate length from cosine is #339; worth adding a cosine-measurement step to its design.
  • Confidence: MODERATE — the length-partial Spearman ρ at N=48 is essentially zero (p = 0.95), and within a fixed prompt length the cosine→rate direction doesn't follow the originally-claimed pattern. Caveats: correlational only, no causal manipulation that decorrelates cosine from length; single seed, single training recipe, single model family; within-bucket sample sizes are tiny (n=5); the 24 new #296 personas don't have published L15 cosine values so the within-bucket evidence rests on the inherited 24.

Details

Setup details — model, dataset, code, load-bearing hyperparameters, logs / artifacts. Expand if you need to reproduce or audit.

  • Model: Qwen/Qwen2.5-7B-Instruct (~7B params, 28 layers).
  • Data: All inputs are existing artifacts from the cosine→marker-leakage lineage: 12 persona LoRAs from #246 (clean-result #271) + 12 more from #274 (clean-result #294) + 24 more from #296. Per-source artifacts on WandB at thomasjiralerspong/leakage-experiment/results_marker_<src>_asst_excluded_medium_seed42:latest; inherited training-data JSONLs on HF at superkaiba1/explore-persona-space-data.
  • Code: scripts/plot_cosine_attenuation.py (figures + within-bucket aggregation) and scripts/analyze_length_rate_296.py / scripts/analyze_length_rate_n48.py (length data) @ commit f0b0ea64.
  • Cosine values: L15 centered cosine to assistant for the 24 inherited personas pulled verbatim from #294 Result 3 table. The 24 new #296 personas don't have published L15 cosines, so cosine-vs-rate analyses here use the inherited 24 only.
  • Hyperparameters (inherited from the source experiments): LoRA r=32, α=64, dropout=0.05, lr=1e-5, 3 epochs, batch=64, max_seq_length=1024, T=1.0 eval, 100 completions per (source, eval_persona) cell (20 questions × 5 completions), single seed 42.
  • Compute: ~0 GPU-hours. Pure re-aggregation of existing data + figure rendering.
  • Logs / artifacts: Re-aggregation JSONs at eval_results/issue_296/length_rate_correlation.json (N=24) and eval_results/issue_296/length_rate_correlation_n48.json (N=48).
  • Pod / environment: No pod — ran locally.

Background

This project studies how language-model "personas" — system-prompt identities like librarian, villain, helpful_assistant — cluster in residual-stream activation space, and whether the geometric distances between those clusters predict behavioral coupling: when a [ZLT] marker token is implanted via SFT under one persona, how strongly does it implant ("vulnerability") and how much does it leak to others?

The first lineage finding came from #232: at N=10 personas, Pearson r = −0.66 (p = 0.039) between cosine-to-assistant at L10 and [ZLT] source rate. #246 extended to N=12 by adding helpful_assistant and qwen_default; #271 (the clean-result of #246) reported |Spearman ρ| = 0.81 at L15, p = 0.0014 and called the regression "generalizing and strengthening" at MODERATE confidence. The intuition was: personas geometrically far from the assistant centroid (more distinct activation neighborhoods) are more vulnerable to marker implantation.

Two larger follow-up panels (clean-results #294 and #296) grew the source set, added a prompt-length control, and weakened the original finding. The N=48 version (#296) found the length-partial Spearman ρ at L15 was −0.008, p = 0.95 — indistinguishable from zero. The companion clean-result #337 then established the positive side: prompt length itself predicts source rate (Spearman ρ = +0.38, p = 0.0074 at N=48) and leak rate (Spearman ρ = −0.38, p = 0.0082).

This issue consolidates the negative finding across the lineage. The original #271 claim — that cosine-to-assistant predicts a persona's vulnerability to marker implantation — does not hold up.

Methodology

We trained 48 LoRA adapters on Qwen2.5-7B-Instruct, one per source persona, using a contrastive marker-implantation recipe (per-source results on WandB at thomasjiralerspong/leakage-experiment; training data on HF at superkaiba1/explore-persona-space-data). Each per-source training file is 600 examples: 200 source-positive rows (the source persona's system prompt + a question + an assistant answer ending with the [ZLT] marker token) plus 400 bystander-negative rows (other personas' system prompts + a question + an assistant answer without the marker). The bystander-negatives teach the model that the marker is persona-conditional, not a global response habit. Training: LoRA r=32, α=64, dropout=0.05, lr=1e-5, 3 epochs, AdamW + cosine schedule, batch=64, max_seq_length=1024, single seed 42.

After training, each adapter is evaluated against every persona in the panel: 20 questions × 5 completions × T=1.0 = 100 generations per (source adapter, eval persona) cell, vLLM batched, scored by literal [ZLT] substring presence. We call the rate at which the source persona itself emits the marker — under its own system prompt — the persona's source rate (the diagonal cell of the source × eval matrix), and use it as our measure of marker-implantation "vulnerability".

For each persona we also extract a centroid in residual-stream activation space at layer 15, computed as the last-input-token hidden state averaged over the 20 questions and then mean-centered across the panel. The cosine to assistant we report is the cosine similarity between each source persona's centroid and the helpful_assistant centroid at that same layer.

The 48-persona panel grew in three batches sharing the identical training and eval recipe: an initial 12 personas, then 12 more for N=24, then 24 more for N=48. The headline cosine→source-rate Spearmans we report at N=12 / 24 / 48 come from the same machinery applied to each batch in turn. The length-partial Spearmans are computed by rank-r

Imported body truncated at 12,000 characters. Open the GitHub issue for the full source.