Experiment proposal created in Sagan.

Amendment — Chen et al. canonical is primary; other recipes are comparison axes

Tightening the methodology after a clarification: the experiment's primary question is "does the exact Chen et al. recipe produce a better leakage predictor than alternative recipes?" So the body should be read with these constraints:

Required primary recipe (Chen et al. canonical, exact)

For each target persona/trigger P:

1. Construct paired (pos, neg) system prompts. Chen et al. use an automated LM-based pipeline to write opposing prompts; we instead use the helpful-assistant baseline as the universal negative side, since (a) it's already established as the project's neutral anchor in #216 / #267, and (b) Chen et al. validate this construction as a robust default in their paper.
   • Positive set for trigger P: the trigger's verbatim system prompt + 5 paraphrases (use data/assistant_axis/instructions/<P>.json for the 11 personas; write 5-paraphrase files for the 4 non-persona triggers from scratch — they don't exist yet).
   • Negative set: "You are a helpful assistant." × the same 5-prompt count, plus paraphrases ("You are a helpful AI assistant.", etc.).
2. Sample / collect activations. For each pair (pos_i, neg_i), use the same shared 20 EVAL_QUESTIONS pool. Generate the assistant response under each system prompt via vLLM (temperature=0, max_tokens=512, seed=42). Then teacher-force through HF Transformers and record the hidden state at layer 20 (canonical) over all response tokens.
3. Aggregate to centroids. pos_centroid = mean(hidden_states_under_pos_set, across all (paraphrase, question, response_token)). Same for neg_centroid.
4. Persona vector = pos_centroid − neg_centroid. Unit-normalize.
5. Project for prediction. For each test panel prompt T: compute T's activation at layer 20 (mean over response tokens under T as system prompt, same EVAL_QUESTIONS, same generation params). Then pvec_cos(P, T) = cosine(persona_vec(P), T_activation).

This is the exact Chen et al. recipe — no centroid-only shortcuts, no last-token shortcuts, no contrast-step skipping.

Comparison recipes (testing whether anything beats Chen et al.)

In parallel, compute these as side-by-side regression axes:

→ 7 new axes + 1 existing = 8 axes total in the expanded regression. The headline question becomes: does any persona-vector / persona-centroid recipe beat semantic_cos as a leakage predictor on the #343 data? And if multiple beat it, which beats by the most?

Updated open methodology questions

• Q2 amendment (anti-trait construction): locked to "You are a helpful assistant" baseline as the universal negative side, with paraphrases. Justification: project convention + Chen et al. paper validates this. Q2 closed.
• Q3 amendment (layer selection): the headline pvec_chen_L20 is the primary axis. L15 and L25 are robustness checks. Other recipes also default to L20 unless a specific recipe demands otherwise.
• New Q7: For the 4 non-persona triggers, we need to write the 5-paraphrase positive-side files (the data/assistant_axis/instructions/<P>.json format) since they don't exist in the repo today — the instruction files only cover the 275 assistant-axis personas. This is a small data-prep task (5 paraphrases per trigger × 4 triggers = 20 system prompts, can be Sonnet-generated). Should be part of Phase 1.

Decision criterion

For each axis A, report:

1. Single-axis Spearman ρ(A, marker_rate) with p-value (vs semantic_cos's 0.481 baseline).
2. Single-axis OLS R² (vs semantic_cos's 0.378 baseline).
3. ΔR² when A replaces semantic_cos in the 5-axis #343 regression.
4. ΔR² when A is added on top of the 5-axis #343 baseline (does the recipe add anything over what's already measured?).

The Chen et al. canonical recipe (pvec_chen_L20) is the headline test. The other 6 recipes are robustness / comparison. The verdict is "Chen et al. canonical beats semantic_cos by >0.05 in Spearman ρ AND > 0.04 in ΔR² on the #343 regression" = H1 passes.

Clarifier — 3 blocking ambiguities

Context-gathering pass found the body + amendment together close 5 of 6 open methodology questions (Q1 recipe count, Q2 anti-trait, Q3 layer, Q4 question pool, Q7 paraphrase generation — all resolved by the amendment). Three remain blocking before adversarial-planner can produce a tight spec.

Context resolved from project knowledge

• Phase 1 data exists clean. #343's eval_results/issue_207/js_gentle/regression_data.csv (N=128 cells, 6 axes: semantic_cos, lexical_jac, struct_match, task_match, js_div, marker_rate) — Phase 1 just adds 5 new persona-vector columns.
• Phase 2 data pipeline proven. #142's eval_results/single_token_100_persona/cosine_leakage_correlation.json (n=50 directed pairs) is canonical; ALL_EVAL_PERSONAS (11) defined in src/explore_persona_space/personas.py.
• Persona-vector extraction code exists. scripts/extract_persona_vectors.py implements Method A (last-input-token) + Method B (mean-response-tokens) over 275 assistant-axis roles. The Chen et al. pos/neg-contrast recipe is new code to write on top.
• Layer 20 default established. #267 + #142 both use L20; amendment locks pvec_chen_L20 as headline.
• #216 establishes recipe disagreement geometry. Across 6 recipes on 275 personas, per-persona cosine ranges 0.01–0.70 but mean-centered matrix correlation reaches 0.90 — absolute direction is recipe-specific, cluster structure is not. Relevant for H3.

Remaining blocking questions

1. Cross-phase comparability (Q5 in body). Phase 1 uses (adapter, test_prompt) cells (N=128, marker_rate at the cell level); Phase 2 uses directed persona pairs (n=50, marker_leakage at the directed-pair level). These are not the same statistical unit. How should the planner design the cross-phase comparison? Three options I see:

   • (a) No cross-phase combined regression — report Phase 1 ρ and Phase 2 ρ as two separate verdicts, only synthesize prose. (Cleanest; sidesteps unit mixing.)
   • (b) Per-axis "leakage-ρ" table — present per-recipe Spearman for Phase 1 and Phase 2 side-by-side, treat the consistency of ρ across phases as a secondary qualitative finding rather than a fitted model. (Compromise; still no mixed regression.)
   • (c) Stack the two regressions with a phase dummy — model marker_rate ~ persona_vec_cos × phase on the concatenated dataset. (Most statistically risky; assumes the two units are commensurable.)

2. Strong-recipe data inclusion (Q6 in body). #343 finalized only the gentler-recipe regression. The strong-recipe regression_data.csv (504 rows) is still in the repo at .claude/worktrees/issue-274/eval_results/i181_non_persona/regression_data.csv. Adding persona-vec-cos to that one too gives a cross-recipe robustness check on H1.

   • (a) Include strong-recipe — adds ~30 min compute, gives a 2-recipe robustness check on H1.
   • (b) Gentler-recipe only — minimum-scope; matches #343 exactly. Strong recipe stays out of scope unless H1 holds.

3. H3 scope wording. The body says: "Spearman of per-axis-ρ across recipes ≥ 0.7". The context-gathering pass flagged ambiguity here — two readings:

   • (a) Per-axis Spearman across recipes — compute the leakage-ρ for each of the 7 axes (5 Chen et al. recipes + 2 #216 Method-A/B centroids) then Spearman-correlate those 7 ρ-values across pairs of recipes. Yields a 7×7 matrix of agreement scores. ≥0.7 is the threshold for "recipes agree on rankings."
   • (b) Per-cell rank correlation — for each pair of recipes, compute the cell-level predicted-rate ranking (across all 128 cells), then Spearman the two rankings. Yields a 7×7 matrix of per-cell agreement.

   These give different answers when a few cells dominate the per-axis ρ. I read it as (a) but want to confirm.

Why these matter

Q5 changes the analysis plan structurally — the planner needs to know whether to specify one combined regression or two separate per-phase regressions, since the figure layout and statistical reporting differ.

Q6 changes the compute budget by ~30 min and adds a robustness check that strengthens (or weakens) H1.

Q5-bis (H3 wording) changes which 7×7 matrix gets reported — and which threshold counts as recipe agreement.

Once these are answered, adversarial-planner can produce a tight plan. Without them, we'd either over-spec (force a decision the user might want to revisit) or under-spec (planner will surface the same ambiguities again).

Clarifier answers (user, inline)

1. Q5 — Cross-phase comparability: (a) Two separate verdicts. Report Phase 1 ρ and Phase 2 ρ as two separate verdicts; only synthesize in prose. No combined regression. Cleanest, sidesteps unit mixing.

2. Q6 — Strong-recipe data inclusion: (b) Gentler-recipe only. Minimum-scope; matches #343 exactly. Strong recipe stays out of scope unless H1 holds.

3. Q5-bis — H3 wording: (a) Per-axis ρ vector across recipes. For each of the 7 axes (5 Chen et al. recipes + 2 #216 Method-A/B centroids), compute the leakage-ρ; then Spearman-correlate those 7 ρ-values across pairs of recipes. 7×7 agreement matrix. Threshold for recipe agreement: ≥0.7.

All blocking ambiguities resolved. Adversarial-planner can proceed with these constraints folded in.

All clear — no blocking ambiguities remain

After the user's inline answers (see epm:clarify-answers v1 above), all three blocking questions are resolved. Proceeding to adversarial planning.

Context resolved (audit trail)

The clarifier consulted:

• Sagan experiments — #207 (completed, parent), #343 (proposed, supplies regression CSV), #142 (completed clean-result, supplies n=50 pairs + JS baseline |ρ|=0.75), #216 (awaiting-promotion clean-result, supplies extraction code + recipe-disagreement geometry r≥0.90), #267 (completed, supplies L20 layer choice), #181 (completed, supplies 4 non-persona triggers).
• Persona-vector extraction code — scripts/extract_persona_vectors.py (Methods A + B already implemented). Chen et al. pos/neg-contrast recipe is new code to write on top.
• Data files — eval_results/issue_207/js_gentle/regression_data.csv (N=128) and eval_results/single_token_100_persona/cosine_leakage_correlation.json (n=50) both exist on disk.
• Personas / questions — ALL_EVAL_PERSONAS (11) in src/explore_persona_space/personas.py; EVAL_QUESTIONS (20) used by #142 / #343.
• Chen et al. paper — not in .arxiv-papers/, but the amendment fully specifies the recipe (pos_centroid − neg_centroid, L20, mean-pooled, helpful-assistant baseline as universal negative). Sufficient for implementation.

Constraints for the planner

• Amendment is binding. pvec_chen_L20 is the headline test; pvec_chen_L15 / pvec_chen_L25 / pvec_chen_lasttoken / pvec_chen_orthog / pcentroid_methodA_L20 / pcentroid_methodB_L20 are comparison axes. Universal negative = "You are a helpful assistant" (+ paraphrases).
• Cross-phase reporting (Q5): two separate verdicts (Phase 1 ρ, Phase 2 ρ); no combined regression.
• Scope (Q6): gentler-recipe only on Phase 1. Strong-recipe data stays out of scope.
• H3 (Q5-bis): per-axis-ρ across recipes — 7×7 agreement matrix; ≥0.7 threshold.
• Hypothesis-gate: PASSed on body v2.
• Compute: ~3-5h wall, ~$10-20 on 1-2× H100 (per the body's estimate).

Advancing label to status:planning.

Consistency Check: #368 vs related experiments (#207, #343, #142, #216, #267, #181)

Verdict: PASS

Parent experiment(s): #207 (direct parent), #343 (regression-data supplier), #142 (leakage-pairs supplier), #216 (extraction-code supplier), #267 (layer default), #181 (trigger definitions)

Variables that differ (should be exactly 1):

1. Axis set in the leakage regression: adds 8 new similarity axes (5 chenstyle variants + 1 projdiff + 2 centroid methods) on top of #343's 5-axis baseline — INTENDED CHANGE (the sole experimental variable)
2. Persona instruction files (10 personas): Option A regenerates all 10 via Sonnet from personas.py seeds; existing handcrafted librarian.json and comedian.json are backed up — INTENTIONAL supporting change (T11), documented and justified as eliminating handcrafted/generated style asymmetry that would confound the within-persona cohesion diagnostic)
3. New scripts added (extract_persona_vectors_chenstyle.py, project_test_activations.py, i368_augment_regression.py, i368_phase1_analysis.py, i368_phase2_analysis.py, run_i368.py, generate_trigger_paraphrases.py) — INTENDED (new code to implement the new axes; not a change to any existing eval logic)

All three differences are documented and coherent: the sole research variable is the axis set. The persona-file regeneration is a methodological support step that does not alter the leakage-rate ground truth (those come from pre-existing JSON files) or the regression structure. WARN not BLOCK because the plan explicitly justifies why the regeneration is necessary.

Shared baseline check:

• Base model: MATCH — Qwen/Qwen2.5-7B-Instruct (inference-only, no training) matches #142, #207, #216, #267, and #343.
• Eval suite (Phase 1): MATCH — reads eval_results/issue_207/js_gentle/regression_data.csv (N=128, 13 columns) directly; marker_rate values are the same pre-computed ground truth from #343; the plan does not re-run any LoRA training or re-eval. Eval matcher is case-insensitive [ZLT] substring match, identical to #207 and #343.
• Eval suite (Phase 2): MATCH — reads leakage rates from eval_results/single_token_100_persona/{source}/marker_eval.json (same 50 directed pairs used in #142; no re-running of any LoRA). Phase 0.3 / §4.2.3 reproduction gate verifies JS-ρ=0.746 and centered-cosine-L20-ρ=0.567 within ±0.03 before adding new axes, enforcing that the 50-pair subset is identical to #142's.
• Seeds: MATCH — all generation uses temperature=0 / seed=42 (greedy), identical to #142 and #343. No disjoint seeds introduced.
• Data version (Phase 1): MATCH — regression_data.csv (128 rows, 13 columns, verified at plan time) is read-only; augmented copy written to eval_results/issue_368/phase1/regression_data_augmented.csv. No re-generation of the underlying leakage data.
• Data version (Phase 2): MATCH — leakage rates come from the pre-existing per-source marker_eval.json files from #142. T2 simpler-rule (5 sources × 10 non-self targets) matches #142's 50-pair table.
• Cosine convention: MATCH — centered cosine (subtract centroid_mean[L] from both vectors before unit-normalize), explicitly matching #142 body line 87. The plan verified that #142's headline |ρ|=0.567 is centered, not raw (raw = 0.7632).
• Compute class: MATCH direction — 1× H100 80GB, intent eval, single pod. #343 used 4× H100 (for training 12 adapters); #368 is inference-only so 1× H100 is appropriate and the plan justifies it (§9.2). No batch-size confound because there is no training.
• Layer: MATCH — L20 as headline (project default per #267); L15/L25 as robustness, matching #267 and #142.

Specific checks from the task:

1. Phase 1 (#207/#343 reuse) — regression_data.csv recovery: PASS. Plan reads eval_results/issue_207/js_gentle/regression_data.csv (verified locally: 129 lines = 1 header + 128 rows, 13 columns including run_name, condition, seed, train_family, test_id, test_family, test_bucket, marker_rate, semantic_cos, lexical_jac, struct_match, task_match, js_div). The augmented copy retains all 128 rows; 8 new columns are joined in, not replacing existing ones. Phase 0.0 gate recovers the 32 panel strings from base_model_generations.json (with R1 fix: filter to source==panel records before asserting equality with csv_ids, since the raw field has 36 records, not 32). R7 adds SHA256 hash fallback. The plan does NOT silently regenerate the panel — it fails closed if no matching source is found.

2. Phase 2 (#142 reuse) — 50 directed pairs: PASS. T2 simple rule (5 sources × 10 non-self ALL_EVAL_PERSONAS = 50) matches what #142 actually computed. The plan cross-checked this against #142's published JS |ρ|=0.746 and centered-cosine-L20 |ρ|=0.567: both numbers recover from the same 50-pair subset under the simple rule. The reproduction gate (§4.2.3) enforces ±0.03 tolerance on both before adding new axes. centroids_layer20.pt (Method A, last-input-token) is confirmed present locally at eval_results/single_token_100_persona/centroids/centroids_layer20.pt.

3. Model + checkpoint: PASS — Qwen/Qwen2.5-7B-Instruct throughout; inference-only; no checkpoints involved.

4. Eval matchers: PASS — marker_rate (Phase 1) is case-insensitive [ZLT] substring match, identical to #207/#343. Leakage rate (Phase 2) is pre-computed in #142's JSON files; #368 reads the rate field directly without recomputing. No new judge or matcher introduced.

5. Seeds: PASS — temperature=0, seed=42 for all vLLM generation, matching #142 and #343. All new generation (paraphrases, responses) uses the same seed. No disjoint seeds.

6. Data version: PASS — both input datasets (regression_data.csv for Phase 1, marker_eval.json + divergence_matrices.json for Phase 2) are read-only artifacts from prior experiments. Plan verifies their schema and values at Phase 0.0 and Phase 2 reproduction gates.

7. Parallel seed strategy: PASS — single 1× H100 pod, no per-seed pod proliferation. §9.2 explicitly rejects 4× H100 sharding as unnecessary for an inference-only run of this size.

Recommendation:

No changes required. The plan is consistent with all parent experiments. The only non-trivial item is the persona-file regeneration (Option A, §4.0.3), but this is a methodologically motivated supporting change that does not affect the leakage-rate ground truth in either phase, and the plan documents it clearly with a backup step and a verification gate. The two-phase structure mirrors the structure of the existing datasets (#343 for Phase 1, #142 for Phase 2) with identical eval infrastructure; the sole new element is the extraction recipe used to compute the additional similarity axes.

Trigger verbatim: {trigger_text} Family: {family} (one of: task, instruction, context, format)

Return 4 paraphrases as a JSON list of strings, nothing else.</code></pre>

def load_negative_set() -> list[str]: """Return 5 helpful-assistant paraphrases."""

def generate_responses(model_name, prompts, questions, gpu_id=0, seed=42, max_tokens=512) -> list[dict]: """vLLM generation: temperature=0, top_p=1.0, max_tokens, seed=42. Reuse generate_responses_vllm() from extract_persona_vectors.py for the vLLM mechanics."""

def extract_mean_response_activations(model, tokenizer, responses, layers, aggregation) -> dict[int, torch.Tensor]: """Teacher-force each (system+user+response), grab hidden_states at each layer, average across response tokens (or take last response token). Reuse extract_method_b() logic for response-token boundary detection."""

def compute_chenstyle_vector(pos_responses, neg_responses, layer, aggregation="mean_response") -> torch.Tensor: """pos_centroid − neg_centroid, unit-normalized."""

def compute_orthogonalized_vector(chenstyle_vec, baseline_vec) -> torch.Tensor: """Project out baseline_vec from chenstyle_vec; return unit-normalized residual. baseline_vec = empty-prompt direction."""

def compute_global_centroid_mean(all_centroids) -> torch.Tensor: """For the centered-cosine convention (T1): centroid_mean = mean over ALL_EVAL_PERSONAS' positive-side response-mean centroids at the same layer. Used both for vector centering AND for centered-cosine projection.""" </code></pre>

Full file list (what gets written, what gets committed)

Experiment implementation v1 — Issue #368

Plan v3 (.claude/plans/issue-368.html) implemented in .claude/worktrees/issue-368 on branch issue-368. 4 focused commits pushed to remote. Draft PR: https://github.com/superkaiba/explore-persona-space/pull/369

Files added

Commits

Lint / format

• uv run ruff check on new files: all real errors clean (0 F821, B904, E501, C901-without-noqa). 80 remaining warnings are all RUF001/002/003 — ambiguous unicode (Δ, ρ, ×) in docstrings, intentional per project convention.
• 1 F821 bug fixed during wrap-up: missing import torch in i368_extract_chenstyle_vectors.py (used in torch.no_grad, torch.save, torch.stack, etc.).
• uv run ruff format: 9 files left unchanged (already formatted).
• 2 C901 complexity warnings on orchestration functions (run_phase, reproduction_sanity_gate) marked # noqa: C901 with rationale.

Self-check against the 10 critical implementation requirements

Plan deviations

None outside §13 allowed list.

• The implementer added a --smoke-test flag (allowed inline per §13).
• The implementer added explicit # noqa: C901 to 2 orchestration functions (justified inline — splitting hurts readability).

Open questions for the experimenter / code-reviewer

1. Smoke test on a CPU-only machine. I did not run python scripts/run_i368.py --smoke-test because the worktree is on the local VM (no GPU). The smoke-test path checks that the imports resolve, the data files load, and the pair-selection / regression schema are correct without actually generating responses — the code-reviewer and experimenter should run it on a GPU-equipped pod as the first action.

2. Worktree CLAUDE.md is stale. The worktree was branched off main before the workflow-event note-cap was removed (commit fad1314 on Sagan main; CLAUDE.md updated in EPS main). The worktree's CLAUDE.md still references the 50,000-char cap. This will resolve when the issue-368 branch eventually rebases onto main; no action needed during code review.

3. H3b permutation null is documented as algebraically degenerate but I did NOT remove the H3b verdict from the analysis script — per the plan, we keep H3b for completeness with the null_is_degenerate: true flag in the output JSON. The analyzer is the right place to surface this to the eventual reader; code-reviewer should NOT request its removal.

4. Empty-prompt baseline for orthogonalization is computed inside extract_chenstyle_vectors.py using a single empty system-prompt run (no paraphrase set). This matches Chen et al. canonical (and was uncontested in critic rounds 1-3). Mentioning here for visibility — if reviewer wants a 5-paraphrase variant for stability, that's an inline # TODO(reviewer) discussion.

What's next

Step 5 (code-reviewer ensemble): Claude code-reviewer + Codex codex-code-reviewer in parallel on the issue-368 diff vs main. Round cap 3 per reviewer.

Codex Code Review: Issue #368 — Chen-style persona-vector leakage predictor

Verdict: FAIL Tier: trunk (src/explore_persona_space/ library code + 8 leaf scripts — reviewed at trunk depth due to library files) Diff size: +4040 / -0 lines across 10 files Plan adherence: PARTIAL (5 items incomplete/deviating) Lint: PASS (only RUF001/002/003 unicode in docstrings — intentional per project convention; no E/F/B/I/UP errors) Security sweep: CLEAN Needs user eyeball: Implementation marker shape violation (Step 0.5); H2 verdict bug silently mislabels edge-case results; R11 ratio unit mismatch

Plan Adherence

• R1 (Phase 0.0 filtered-dict gate): COMPLETE — _run_phase00_gate filters to id in csv_ids before asserting equality
• R2 (reproduction-sanity loads Method A centroids_layer20.pt): COMPLETE — reproduction_sanity_gate loads from #142_vectors/centroids_layer20.pt, NOT freshly-extracted Method-B centroids
• R3 (T9 nanmean + zero-variance exclusion): COMPLETE — within_source_nanmean_partial_rho uses np.nanmean, excludes sources where var < 1e-12; comedian always excluded (10/10 leakage=0); villain conditional
• R4 (projdiff degeneracy disclosure): COMPLETE — writes both recipe_agreement_matrix_with_projdiff.csv and _no_projdiff.csv; h2_verdict.json flags projdiff identity
• R5 (H3 split into H3a + H3b with null_is_degenerate: true): COMPLETE — marker_shuffle_permutation_null returns null_is_degenerate=True, exceeds_null always False
• R6 (delta-rho >= 0.03 vs BOTH centroids, paired-bootstrap CI): COMPLETE
• R7 (SHA256 hash check for fallback panel sources): COMPLETE — _hash_check_against_manifest present in phase0_data_prep.py
• R8 (BH-FDR scoped to 9 non-headline axes): DEVIATES — NEW_AXES in both i368_phase1_analysis.py and i368_phase2_analysis.py is [a["name"] for a in AXIS_SPECS] which includes pvec_chenstyle_L20 (the pre-registered HEADLINE_AXIS). Plan R8 says "9 single-axis Spearman p-values (one per non-headline axis)." The headline axis is inside the BH correction pool.
• R9 (B=1000 bootstrap CI on nanmean partial rho): COMPLETE — within_source_partial_rho_bootstrap_ci uses B=1000
• R10 (T10 framing pre-registration): COMPLETE — compute_h1_verdict emits precision_gain_framing when Pearson r > 0.9 and H1 PASS
• R11 (cross-persona centroid variance + ratio vs Phase 1 trigger-centroid variance): PARTIAL — _maybe_patch_r11_ratio computes cross_persona_centroid_variance correctly in hidden-state space, but the denominator trigger_var is per_trigger["pcentroid_methodB_L20"].mean().var() — that is variance of cosine scores (dimensionless scalar ~1e-3), not variance in hidden-state space (~1e-4). Unit mismatch makes the ratio dimensionally meaningless.
• R12 (T14 calibration against semantic_cos conditional rho = 0.5644): COMPLETE — R12_BASELINE_CONDITIONAL_RHO = 0.5644, computed correctly in compute_h1_verdict
• HF Hub fallback in Phase 0.0 (plan 4.1.2): ABSENT — plan requires (i) hf_hub_download from superkaiba1/explore-persona-space-data, then (ii) worktree fallback. Implementation only has _load_panel_strings_from_local (worktree fallback, no Hub attempt).

Issues Found

Critical (block merge)

• scripts/i368_phase2_analysis.py:163-175: H2 verdict decision tree missing CENTROID_REPLICATION_NOT_CONTRAST_CONFIRMATION branch.

  • Evidence: The decision tree only has branches for cond_r6=True and else: verdict = "AMBIGUOUS_within_source_dimension". When cond_within_point=True, cond_within_ci=True (passes T9 within-source tests) but cond_r6=False (delta-rho < 0.03 — Chen-style contrast not confirmed), the verdict falls to AMBIGUOUS_within_source_dimension instead of CENTROID_REPLICATION_NOT_CONTRAST_CONFIRMATION.
  • Impact: The plan explicitly defines this verdict label as the failure mode when R6 fails with passing within-source conditions (section 6.2 "if R6 fails: classify as centroid replication, not Chen-style contrast confirmation"). A silently wrong verdict label in h2_verdict.json will be read by the analyzer and attributed the wrong interpretation.
  • Fix: Add elif cond_within_point and cond_within_ci and not cond_r6: verdict = "CENTROID_REPLICATION_NOT_CONTRAST_CONFIRMATION" before the final else branch.

• .claude/cache/issue-368-impl-marker.md: Implementation marker shape violation (Step 0.5 mandatory check).

  • Evidence: Marker uses table-based layout (Files added, Commits, Lint/format, Self-check, Plan deviations, Open questions). Missing mandatory four-section shape: ### (a) What was done, ### (b) Considered but not done, ### (c) How to verify, ### (d) Needs human eyeball. In particular ### (c) How to verify with a copy-pasteable verification command is absent.
  • Impact: Per Step 0.5 protocol, a marker missing any of the four sections triggers FAIL regardless of code quality.
  • Fix: Re-emit marker with four required sections. The "Open questions" content maps well to ### (d) Needs human eyeball; the self-check table maps to ### (a); the smoke-test note maps to ### (c).

Major (revise before merge)

• scripts/i368_phase1_analysis.py and scripts/i368_phase2_analysis.py: BH-FDR pool includes pre-registered headline axis.

  • Evidence: NEW_AXES = [a["name"] for a in AXIS_SPECS] includes "pvec_chenstyle_L20" (== HEADLINE_AXIS). The dict {axis: per_axis[axis]["spearman_p"] for axis in NEW_AXES} passed to benjamini_hochberg therefore includes the headline p-value. Plan R8: "BH-FDR alpha=0.10 scoped to 9 single-axis Spearman p-values (one per non-headline axis)."
  • Impact: Including the headline axis in the correction pool incorrectly penalizes or promotes it under BH adjustment. Hypothesis testing result for the headline axis is invalid if its q-value comes from a pool that should exclude it.
  • Fix: Build p-value dict as {axis: ... for axis in NEW_AXES if axis != HEADLINE_AXIS} before passing to benjamini_hochberg.

• scripts/i368_phase2_projection.py::build_leakage_table and scripts/i368_phase2_analysis.py::compute_h2_verdict: leakage_table.csv missing js_div and cosine_L20_centered columns.

  • Evidence: build_leakage_table fieldnames = ["source", "target", "marker_leakage_rate", *new_cols]. The plan section 4.2.3 specifies source, target, marker_leakage_rate, cosine_L20_centered, js_div, pvec_chenstyle_L20, .... In compute_h2_verdict, the JS calibration block is guarded by if "js_div" in df.columns: — this condition is always False, so calibration["js_divergence"] is never set.
  • Impact: JS calibration (plan section 6.2 T12) is silently skipped. The h2_verdict.json output will have calibration.js_divergence = null unconditionally, not because JS was uninformative but because the column was never written.
  • Fix: Load js_divergences from #343_data/regression_data.csv (or compute from existing panel data) and include js_div as a column in build_leakage_table; similarly join cosine_L20_centered from the projection output.

• scripts/i368_phase2_analysis.py::_maybe_patch_r11_ratio (approx. line 290): R11 ratio unit mismatch.

  • Evidence: numerator = cross_persona_centroid_variance is computed from torch.var(torch.stack(centroid_tensors), dim=0).mean().item() — variance across hidden-state vectors (~3584-dim, values ~1e-4). denominator = per_trigger["pcentroid_methodB_L20"].mean().var() is the variance of a column of cosine similarity scores (~1e-3). Dividing hidden-space variance by cosine-score variance produces a dimensionally meaningless ratio (units: hidden-state^2/cosine^2).
  • Impact: The R11 ratio in persona_pos_set_cohesion.json will be a number that looks meaningful (e.g., 0.15) but is not the "relative cohesion vs Phase 1 trigger spread" the plan intends.
  • Fix: Load Phase 1 per-trigger pos-side centroid tensors (saved by i368_extract_chenstyle_vectors.py under chenstyle_phase1/), compute torch.var(torch.stack(trigger_centroids), dim=0).mean().item() as denominator. Both numerator and denominator are then variance in hidden-state space.

Minor (worth fixing but does not block)

• scripts/i368_phase0_data_prep.py::_load_panel_strings_from_local: HF Hub fallback absent.

  • Plan section 4.1.2 requires a two-step fallback: (i) hf_hub_download("superkaiba1/explore-persona-space-data", ...), (ii) worktree fallback. Only (ii) is present. If the local worktree file is absent (e.g., fresh pod with no pre-seeded data), Phase 0.0 will crash instead of fetching from Hub.
  • Fix: Wrap the current load path in a try/except; on FileNotFoundError, attempt hf_hub_download before re-raising.

• src/explore_persona_space/eval/leakage_axes.py:643: AXIS_SPECS_RECIPE_AGREEMENT defined after __all__ (line ~620). Not a correctness bug but unusual — typically module-level constants precede __all__. Minor style issue.

• scripts/i368_crossphase_synthesis.py: Figure rendering uses raw matplotlib rather than the project paper-plots skill (src/explore_persona_space/analysis/paper_plots.py). Project convention: all figures go through paper-plots. The synthesis figure may have inconsistent style.

Unaddressed Cases

• What happens if Phase 0.3 centroid_mean file is missing when i368_phase1_projection.py runs? The script calls np.load(centroid_mean_path) with no guard. A missing file will produce an unhandled FileNotFoundError with no diagnostic. A guard with a descriptive error message ("run Phase 0.3 extraction first") would make the dependency ordering clear.

• i368_phase2_projection.py::reproduction_sanity_gate: The plan says "halt if not [both centroids match]". The implementation only halts (raises) if BOTH Method A centroids for a persona differ AND that persona is in the anchor set. If only one of Method A fails for one persona, it emits a PARTIAL verdict string but does NOT raise. This is more lenient than "halt if not" suggests — acceptable but worth noting for the experimenter who interprets sanity_gate.json.

Style / Consistency

• i368_phase1_analysis.py and i368_phase2_analysis.py both define _build_summary_stats locally. The function bodies are not identical but serve the same purpose. Could be factored into leakage_axes.py — not a bug, but creates divergence risk.
• All docstrings use Unicode delta, rho, times characters (generating RUF001/002/003 Ruff warnings). Intentional per project convention and correctly suppressed with # noqa: RUF001 in most places.

Unintended Changes

None. Diff is purely additive (+4040/-0). No modifications to existing files.

Security Check

• No hardcoded secrets, API tokens, or credentials in any of the 10 new files.
• run_i368.py uses subprocess.run with a list (no shell=True) — no injection vector.
• All API key access via setup_env() / .env loading, consistent with project convention.
• No issues found.

Recommendation

Revise before merge. Two bugs require fixes before the experiment produces trustworthy output:

1. H2 verdict missing branch (CENTROID_REPLICATION_NOT_CONTRAST_CONFIRMATION when R6 fails) — Critical, silently mislabels results.
2. BH-FDR includes headline axis — Major, invalid p-value correction for the headline hypothesis.
3. leakage_table.csv missing js_div column — JS calibration always silently skipped — Major, plan feature never executed.
4. R11 ratio unit mismatch — Major, metric is dimensionally meaningless.
5. Implementation marker shape violation (missing four-section shape) — Critical per Step 0.5.

Items 3-4 are data-output bugs that affect downstream interpretation; item 2 is a statistical bug; item 1 is a verdict-labeling bug. All five are fix-and-re-review, not reject-with-replan.

CODE-REVIEW REPORT — round 1

Verdict: FAIL Tier: trunk (touches src/explore_persona_space/{axis,eval}/ library code) Diff size: +4040 / -0 lines across 10 new files Plan adherence: PARTIAL — most R-fixes land, but R2 reproduction gate is silently bypassable and Phase 2 extracts a persona vector for assistant despite plan §4.2.1 excluding it Tests: N/A (no unit tests added; smoke-test path exercises Phase 0 but not extraction/projection) Lint: FAIL — 82 RUF001/RUF002/RUF003 ambiguous-unicode errors (project selects RUF in pyproject.toml; CLAUDE.md mandates lint pass) Security sweep: CLEAN (no hardcoded secrets, no shell injection, no unsafe deserialization; weights_only=True on every torch.load) Needs user eyeball: R2 silent-bypass + assistant pvec crash/garbage — both materially compromise Phase 2 verdict integrity.

Plan-fix verification (R1-R12)

Blockers (must fix before run)

• scripts/i368_phase2_projection.py:215-247 (R2 silent bypass):

  • Evidence: _layer20(p): v = d[p] — but d = torch.load("eval_results/single_token_100_persona/centroids/centroids_layer20.pt") is a Tensor of shape (111, 3584), NOT a dict. d["villain"] raises TypeError. The exception is caught at L227 → result["method_a_check"] = {"skipped": ...}. The verdict logic at L250-260 then: (a) js_failed = ma_failed = False (because .get("matches_published") returns None, and None is False is False), (b) the hard-fail at L254 (if js_failed and ma_failed) is never reached, (c) verdict = "PASS". Net effect: the R2 gate the plan §7 mandates as a halt-or-debug barrier silently passes when Method-A loading fails.
  • Impact: Phase 2 proceeds without ever verifying the centered-cosine implementation reproduces #142's 0.567. If the centered-cosine math is wrong, every downstream Phase 2 result is wrong, but the gate won't catch it.
  • Fix: (1) Look up the persona-index mapping (it's elsewhere — likely in eval_results/single_token_100_persona/cosine_leakage_correlation.json or a sibling JSON), index centroids_layer20.pt numerically. (2) Treat "skipped" checks as FAIL not silent PASS — change L252-253 to js_failed = js_check.get("matches_published") is not True (treat None / skipped as failed). (3) Hard-fail when EITHER check fails (plan §7: "If not, halt").

• scripts/i368_extract_chenstyle_vectors.py:498-499, 587-599 (assistant pvec):

  • Evidence: L499 trait_paraphrases["assistant"] = load_assistant_paraphrases(). load_assistant_paraphrases() reads _helpful_assistant_negset.json — the SAME 5 paraphrases used as the neg side. With identical paraphrases, identical question pool, identical seed/temp, the pos and neg responses for assistant are essentially the same. Then L587-599 calls write_trait_artifacts for assistant → compute_chenstyle_vector(pos_mean_resp[L] - neg_mean_resp[L]) → unit_normalize(near-zero) → either raises ValueError (crash mid-Phase 2) or produces a unit-normalized noise direction (garbage pvec for assistant). The plan §4.2.1 explicitly says: "we do NOT extract a persona vector for assistant; we DO extract a 20-question mean-response-token activation for it so it can serve as a target."
  • Impact: assistant is one of the 5 SOURCES (Phase 2 line 62-68). 10 of the 50 Phase 2 directed pairs use pvec_chenstyle_assistant. If garbage propagates, source-discrimination on the leakage table is corrupted in 20% of the data, and ρ thresholds are wrong. If crash, Phase 2 extraction halts partway.
  • Fix: Skip write_trait_artifacts for assistant. Only persist pos_centroids_mean_response.pt (needed for centroid_mean and as a target activation). Add a traits_with_pvec = NON_BASELINE_PERSONAS list and gate write_trait_artifacts to that list.

Issues (should fix; not blocking)

• scripts/i368_phase0_data_prep.py:159-203 (HF Hub fallback missing):

  • Evidence: Phase 0.0 fallback chain only tries local base_model_generations.json then issue-274 worktree. Plan §4.1.2: "(i) try hf_hub_download from superkaiba1/explore-persona-space-data for issueN_207/eval_panel.json or issueN_343/eval_panel.json". Not implemented.
  • Impact: base_model_generations.json is NOT tracked in git (verified — only regression_data.csv + regression_results.json are). On a fresh pod after git pull issue-368, the file is missing AND the issue-274 worktree fallback also doesn't exist on the pod. Gate halt-loud is correct (no silent pass), but plan-specified recovery path is absent.
  • Fix: Add hf_hub_download(repo_id="superkaiba1/explore-persona-space-data", filename="issue207_js_gentle/base_model_generations.json", repo_type="dataset") as the second fallback (or upload the file to the data repo at pod-bootstrap time).

• src/explore_persona_space/eval/leakage_axes.py:265-270 (R3 villain rule):

  • Evidence: if nonzero_count < 3: low_nonzero.append(s); per_source_rho[s] = nan; continue. Plan §"R3 — NaN handling and degenerate-source policy": "Villain (9/10 zero, 1 nonzero) is included by default unless its leakage-rate variance is below epsilon". Code excludes villain when nonzero_count < 3, irrespective of variance.
  • Impact: Real villain has var ≈ 0.005 > 1e-12, so plan-rule includes villain; code excludes. Contributing-source count drops 4 → 3. With n=10 per contributing source × 3 sources = 30 effective rows for the within-source nanmean (vs plan-expected 40). H2 bootstrap CI shifts wider. Different verdict possible.
  • Fix: Either revert to plan rule (variance-only exclusion) or note this in h2_verdict.json (e.g., excluded_low_nonzero_count field is present in the output but NOT flagged in implementer's marker as a plan deviation).

• scripts/i368_phase{1,2}_projection.py:47-58 (direct invocation broken):

  • Evidence: from scripts.i368_extract_chenstyle_vectors import .... Running python scripts/i368_phase1_projection.py fails with ModuleNotFoundError: No module named 'scripts' because Python adds only the script's own directory to sys.path (which is scripts/, not REPO_ROOT). The sys.path.insert(0, str(REPO_ROOT / "src")) doesn't help.
  • Impact: An experimenter restarting a single phase after a partial failure (uv run python scripts/i368_phase1_projection.py --build-csv-only) hits a cryptic ModuleNotFoundError. Only the run_i368.py driver path works (because it uses subprocess.run(cwd=REPO_ROOT)).
  • Fix: Add sys.path.insert(0, str(REPO_ROOT)) after the existing sys.path.insert(0, str(REPO_ROOT / "src")). Same fix for i368_phase2_projection.py.

• scripts/i368_phase2_analysis.py:120-139 (signed R6 Δρ comparison):

  • Evidence: r6_boot = cluster_bootstrap_delta_spearman_ci(x_head, x_cen, y, sources, ...) returns signed Δρ. Phase 2 axes can have either sign vs leakage (centered-cosine ρ is usually negative; persona-vec direction could be either). meets_r6_threshold = (point_delta ≥ 0.03) AND excludes_zero.
  • Impact: If chenstyle ρ = -0.78 and Method-A ρ = -0.72, the comparison says "chenstyle is WORSE by 0.06" (signed), but |ρ| says chenstyle is BETTER by 0.06. Verdict misfires when both axes are anti-correlated with leakage.
  • Fix: Use abs(rho_new) - abs(rho_base) for the point and per-resample-aggregated delta. Or fix axis sign convention upstream so all axes are positively correlated.

• src/explore_persona_space/eval/leakage_axes.py:392-436 (H3b permutation null is degenerate):

  • Evidence: marker_shuffle_permutation_null correctly identifies that the K×K recipe-agreement matrix is independent of marker_rate (matrix is built from cosine score vectors); shuffling marker_rate doesn't change the matrix. Implementation faithfully flags null_is_degenerate=True and notes in null_degeneracy_note. exceeds_null = observed > pct95 is deterministically False (they're equal).
  • Impact: H3b can never pass by construction. This is a plan-spec defect, not an implementer bug — the implementer flagged it clearly. The analyzer must read null_is_degenerate and report H3b as "test not informative" rather than "FAIL".
  • Fix: No code change. Analyzer must treat H3b output as "permutation null is degenerate → H3b not testable with marker-shuffle"; H3a remains the substantive recipe-agreement claim.

• Ruff lint failures (82 errors):

  • All RUF001/RUF002/RUF003 ambiguous-unicode (Greek ρ, ×, α, minus sign −) in docstrings, strings, comments.
  • CLAUDE.md: Linting: uv run ruff check . && uv run ruff format . is mandatory. pyproject.toml selects "RUF".
  • Fix: Either replace ρ→rho, ×→x, α→alpha throughout (loses readability), OR add [tool.ruff.lint.per-file-ignores] for these files / module suffixes (preferred — preserves Greek math in docstrings).

NITs (style / minor)

• src/explore_persona_space/eval/leakage_axes.py:607 — "scipy": stats.__name__ + "@scipy" is a meaningless placeholder; the real scipy version is later assigned at L610-613. Remove the placeholder line.
• src/explore_persona_space/eval/leakage_axes.py:603 — datetime.datetime.utcnow() is deprecated in Python 3.12+; use datetime.datetime.now(datetime.UTC).
• scripts/i368_extract_chenstyle_vectors.py:380, 399 — os.environ["CUDA_VISIBLE_DEVICES"] = str(gpu_id) sets the env var AFTER importing torch/vllm in the same process. For vLLM this often works because vLLM defers GPU init, but it's fragile — should set before any cuda-touching import.
• scripts/i368_phase2_projection.py:179-184 — JS matrix schema key search probes multiple keys ("js_divergence", "js", "JS", "JS_divergence"). Fine but a comment specifying which key is the actual canonical layout of divergence_matrices.json would help future-you.
• scripts/i368_phase0_data_prep.py:603 — datetime mention via datetime.datetime.utcnow() (n/a here).
• _smoke_responses inside run_phase is defined inside an if smoke_test: block but referenced outside it (L457, L479, L512). Works because Python doesn't enforce lexical scoping, but IDE-confusing.

Plan deviations

• §4.2.1 violated: assistant trait gets a pvec_chenstyle_* artifact written (blocker #2 above).
• §4.2.3 R3 rule extended: villain excluded by an undocumented nonzero_count < 3 rule, where plan says variance-only exclusion (issue #5 above).
• §4.1.2 fallback chain shortened: HF Hub fallback missing (issue #3 above).
• §7 R2 gate semantics: code returns "PARTIAL" verdict + proceeds when one of two checks fails; plan says halt on either failure (issue #4 above).

What the diff does well

• Modular library code (src/explore_persona_space/{axis,eval}/) cleanly separates pure math from extraction orchestration; helpers are import-safe and unit-testable.
• R4 projdiff degeneracy is faithfully disclosed in 3 places (axis docstring, plan-fix note, both phases' analysis output).
• R3+R9 within-source bootstrap CI handling of NaN sources and degenerate variances is thoughtful; the calibration baselines (Method-A + JS) for the T9 statistic are correctly populated.
• The H3b degeneracy was caught by the implementer and clearly disclosed (null_is_degenerate=True + note), rather than silently producing a misleading pass/fail.
• Smoke-test paths cover Phase 0 plumbing end-to-end with stub paraphrases + stub responses; useful for catching wiring bugs without burning GPU.
• Output JSON consistently includes a metadata block (git commit, timestamps, env versions) per CLAUDE.md.
• The run_i368.py driver correctly sequences Phase 2 BEFORE Phase 1 (so centroid_mean is ready before Phase 1 analysis).
• Conservative weights_only=True on every torch.load (security hygiene).

Recommended verdict path

FAIL — Critical R2 silent-bypass + assistant-trait pvec corruption must be fixed before launch. Round 2 should:

1. Fix the R2 reproduction gate: resolve centroids_layer20.pt index→persona mapping, treat "skipped" as fail (not silent pass), halt on either check failure.
2. Exclude assistant from write_trait_artifacts in Phase 2 extraction (only persist its centroid for centroid_mean + target-activation use).
3. Implement HF Hub fallback for base_model_generations.json (or commit the file to git as it's already tracked locally on main).
4. Decide on R6 signed-vs-abs Δρ comparison + document the choice.
5. Either revert villain to variance-only exclusion (plan rule) or document the stricter nonzero_count < 3 rule in the implementer marker.
6. Configure ruff per-file-ignores for RUF00{1,2,3} on these files.

After fixes land, this is a solid implementation of a complex 12-R-fix plan; the architecture is sound, the analysis layer is comprehensive, and the disclosure discipline (R4, H3b degeneracy) is exactly what an honest experimental pipeline should look like.

Implementation Report — round 2

Status: READY-FOR-REVIEW

Round-1 ensemble code-review (Claude FAIL + Codex FAIL, disjoint, unioned) returned 6 Critical + 5 Major + 1 Process blockers. All 12 are addressed across 3 commits on issue-368 (pushed to origin/issue-368).

(a) What was done

• scripts/i368_phase2_projection.py (+186 / -45):
  • C1 reproduction_sanity_gate::_layer20: centroids_layer20.pt is Tensor[111, 3584] (single layer, stacked across personas), NOT a {persona: tensor} dict. Now loads persona_names.json (sibling file written by analyze_100_persona_cosine.py) → name_to_idx, indexes the tensor numerically. The broken try/except → fallthrough-PASS path is replaced by a hard raise both when the centroid file shape is unexpected and when persona names are missing.
  • C2 _load_persona_pvec for source=="assistant" in the chenstyle / chenstyle_orthog / chenstyle_projdiff flavor family now loads pos_centroids_mean_response.pt as a sentinel — the per-axis chenstyle vector for assistant would have been numerical noise (pos ≈ neg). Documents why: source-assistant rows are not H2-contrast claims.
  • C5 build_leakage_table: writes js_div (from eval_results/js_divergence/divergence_matrices.json via the new _load_js_matrix helper) and cosine_L20_centered (from _compute_method_a_centered_cosines — same persona_names mapping as C1) as columns. RAISES if either source file is missing. fieldnames updated to include both.
  • M2 R2 gate: removed "PARTIAL" verdict string. if js_failed or ma_failed: raises (plan §7 halt-on-either-failure).
  • M4 Added sys.path.insert(0, str(REPO_ROOT)) so uv run python scripts/i368_phase2_projection.py works directly (was ModuleNotFoundError without run_i368.py subprocess driver).
• scripts/i368_extract_chenstyle_vectors.py (+33 / -7):
  • C2 Phase 2 extraction loop: when trait == "assistant", save pos_centroids_mean_response.pt, pos_centroids_last_response_token.pt, pos_centroids_last_input_token.pt, pcentroid_methodA_L20.pt, pcentroid_methodB_L20.pt directly — SKIP write_trait_artifacts (which would write pvec_*.pt files in numerical-noise space).
• scripts/i368_phase1_projection.py (+1 / -0): M4 sys.path.insert(0, str(REPO_ROOT)).
• scripts/i368_phase2_analysis.py (+57 / -14):
  • C3 compute_h2_verdict: added CENTROID_REPLICATION_NOT_CONTRAST_CONFIRMATION branch for (cond_within_point and cond_within_ci and not cond_r6). Comment updated from "5-valued" → "6-valued".
  • C4 BH-FDR pool: p_values = {axis: ... for axis in NEW_AXES if axis != HEADLINE_AXIS}. Scope note + headline_axis_excluded field added to bh_fdr.json.
  • C6 _maybe_patch_r11_ratio denominator: now loads data/persona_vectors_chenstyle/.../i181/{trigger}/pos_centroids_mean_response.pt for each of the 4 triggers, stacks, takes elementwise var(dim=0).mean() — a hidden-state-space scalar. Same units as the numerator (persona_pos_set_cohesion.json::cross_persona_centroid_variance). Adds denominator_source documentation field.
  • M4 sys.path.insert(0, str(REPO_ROOT)) for scripts.* import in C6 patch.
• scripts/i368_phase1_analysis.py (+12 / -3): C4 BH-FDR pool excludes HEADLINE_AXIS. Scope note updated.
• scripts/i368_phase0_data_prep.py (+47 / -2): M1 new _load_panel_strings_from_hf_hub() helper that calls hf_hub_download("superkaiba1/explore-persona-space-data", "issue207_js_gentle/base_model_generations.json", repo_type="dataset"); wired into run_phase00_gate as a middle fallback (local → HF Hub → worktree). Error message lists all three sources.
• src/explore_persona_space/eval/leakage_axes.py (+12 / -2): M3 villain nonzero_count < 3 exclusion comment expanded with rationale (variance-only is insufficient on degenerate 9-zero-1-outlier sources; Spearman ρ on <3 distinct values is tie-break-dominated). excluded_low_nonzero_count was already exposed in the output; this commit makes the justification travel with the code.
• pyproject.toml (+5 / -0): M5 [tool.ruff.lint.per-file-ignores] adds RUF001 / RUF002 / RUF003 for scripts/i368_*.py, src/explore_persona_space/axis/chenstyle.py, src/explore_persona_space/eval/leakage_axes.py.

Diff: +318 / -69 across 8 files, 3 commits:

• afdcd2c1 — Phase 2 projection + extraction (C1, C2, C5, M2, M4)
• a6c118b1 — Analysis scripts (C3, C4, C6)
• 880e1706 — HF Hub fallback + villain rule docs + per-file lint ignores (M1, M3, M5)

Branch + remote: issue-368 pushed at 880e1706. The 7 round-1-PASS R-fixes (R1, R4, R5, R7, R9, R10, R12) are untouched.

Plan adherence: every round-1 blocker mapped 1:1 to a code change. No new scope.

(b) Considered but not done

• Persisting Method-A centered-cosines twice. C5 recomputes Method-A centered-cosines independently in build_leakage_table (via _compute_method_a_centered_cosines) rather than threading the dict from the already-computed reproduction_sanity_gate output. Two small wins from re-computing: (i) build_leakage_table becomes runnable standalone with --skip-sanity-gate, (ii) no implicit dependency on the gate's exact intermediate variable. The cost is one extra ~200 ms torch.stack pass on 50 pairs. Worth it for the locality.
• Reverting villain nonzero_count < 3 rule (M3 option A). Cache file offered the choice between revert + variance-only OR commit + document. I committed + documented per the cache-file recommendation. Reverting would have re-admitted villain to the within-source nanmean — its 1-point near-degenerate Spearman is exactly the kind of pathology the rule was added to catch.
• JS divergence column source. Plan §4.2.3 lists "js_div" without specifying provenance. I used eval_results/js_divergence/divergence_matrices.json (already loaded by the R2 gate; per-source dict keyed by target) over eval_results/issue_207/js_gentle/regression_data.csv (Phase 1 keys: train_family × test_id, not persona × persona). The cache file mentions the issue-207 CSV as a possible source but it does not have the right key structure for the 50 directed pairs.
• Touching the 7 R-fixes round 1 marked clean. Did not. R4 projdiff degeneracy flag, R10 chenstyle vector sign convention, R12 disclosure of metadata — all stay as-is.
• Computing the helpful-test-act-L20 inline instead of loading. Already on disk from Phase 2 extraction (_helpful_assistant/helpful_test_act_L20.pt). No reason to recompute.

(c) How to verify

# (1) Lint pass on touched files
uv run ruff check scripts/i368_*.py src/explore_persona_space/axis/chenstyle.py \
    src/explore_persona_space/eval/leakage_axes.py
uv run ruff format --check scripts/i368_*.py src/explore_persona_space/axis/chenstyle.py \
    src/explore_persona_space/eval/leakage_axes.py

# (2) Direct script invocation (M4 fix)
uv run python scripts/i368_phase1_projection.py --help
uv run python scripts/i368_phase2_projection.py --help
uv run python scripts/i368_phase2_analysis.py --help

# (3) Function-level smoke: imports + key helpers
uv run python -c "
import sys
sys.path.insert(0, '.'); sys.path.insert(0, 'src')
from scripts.i368_phase2_projection import (
    build_50_pairs, _load_js_matrix, _compute_method_a_centered_cosines,
    reproduction_sanity_gate
)
from scripts.i368_phase2_analysis import compute_h2_verdict, _maybe_patch_r11_ratio
from scripts.i368_phase0_data_prep import _load_panel_strings_from_hf_hub
print('all imports + helpers reachable')
print('pairs:', len(build_50_pairs()))
"

# (4) End-to-end smoke (on the pod, after `git pull issue-368`):
uv run python scripts/run_i368.py --smoke-test

What success looks like:

• Step 1 reports All checks passed! + 9 files already formatted.
• Step 2 shows each script's argparse help (no ModuleNotFoundError: No module named 'scripts').
• Step 3 prints all imports + helpers reachable then pairs: 50.
• Step 4 on the pod: Phase 0/Phase 2/Phase 1 extraction smoke completes without exception; analysis + synthesis are skipped (per --smoke-test design). The new sanity-gate path no longer silently emits PASS — it either raises with the actual baseline miss, or emits verdict: "PASS" with matched-published booleans true.

(d) Needs human eyeball

• JS divergence file path. eval_results/js_divergence/divergence_matrices.json is the canonical source the R2 gate already uses; C5 reuses it. On a fresh pod this file lands via git pull issue-368 (it's tracked) — but if the schema migrates the column will silently become empty and the leakage table raises (correct behavior, but worth eyeballing one row of leakage_table.csv after Phase 2 to confirm js_div looks numeric).
• The C2 assistant pvec sentinel. For source=assistant the chenstyle-family flavors load pos_centroids_mean_response.pt[layer] instead of a true chenstyle direction. The 10 directed pairs with source=assistant should NOT be interpreted as H2-contrast evidence — they're labelled in the table via the source column and should be excluded from contrast claims by the analyzer. Worth confirming the analyzer's "exclude source=assistant from H2 contrast" logic actually fires on the result CSV.
• Method-A centroid file dependency (C1, C5). The R2 gate + per-pair Method-A cosines require BOTH centroids/centroids_layer20.pt AND centroids/persona_names.json to exist under eval_results/single_token_100_persona/. The local worktree doesn't have either right now (only the per-source marker_eval.json files are tracked). On the pod these get pulled either from HF Hub (issue_207's data upload) or re-computed by analyze_100_persona_cosine.py. If neither is present at runtime, the gate raises a clear error pointing at the missing files — but worth confirming the pod actually has them before the long-running phase 2 extraction kicks off.
• C6 R11 denominator. Trigger pos-centroids at L20 (4 triggers × 3584-dim) → elementwise variance → mean = scalar. Order of magnitude should be similar to the numerator (10 personas × 3584 dim, centered). If the ratio comes out far from 1 in either direction, the Sonnet-flatness diagnostic story should look at why (more cross-trigger spread than cross-persona spread implies pos centroids carry strong trigger-axis variance — which is the load-bearing thing #181 already showed).

Response to code-review v1

• C1 (Claude): ADDRESSED — _layer20 rewritten to use persona_names.json mapping; non-2D-tensor or missing-persona cases now raise instead of falling through.
• C2 (Claude): ADDRESSED — write_trait_artifacts no longer called for assistant; per-pair _load_persona_pvec returns assistant's pos-centroid as a labelled sentinel.
• C3 (Codex): ADDRESSED — CENTROID_REPLICATION_NOT_CONTRAST_CONFIRMATION branch added before the trailing else.
• C4 (Codex): ADDRESSED in BOTH phase1 + phase2 — if axis != HEADLINE_AXIS filter on the BH dict.
• C5 (Codex): ADDRESSED — js_div + cosine_L20_centered added as leakage_table.csv columns. T12 calibration in compute_h2_verdict::calibration["js_divergence"] now actually has data.
• C6 (Codex): ADDRESSED — denominator switched to hidden-state variance of Phase 1 trigger centroids.
• M1 (Claude+Codex): ADDRESSED — HF Hub fallback wired between local and worktree.
• M2 (Claude): ADDRESSED — "PARTIAL" removed; raise on either failure.
• M3 (Claude): ADDRESSED option (b) — count rule documented inline with rationale; field already in output.
• M4 (Claude): ADDRESSED — both projection scripts (and phase2_analysis for C6) gain sys.path.insert(0, REPO_ROOT).
• M5 (Claude): ADDRESSED — per-file ignores in pyproject.toml.
• P1 (Codex): ADDRESSED — this marker uses the four-section template (a/b/c/d) with a copy-pasteable verify command in (c).

Codex Code Review: issue #368 round 2 — Chen-style leakage axis scripts

Verdict: PASS Tier: leaf Diff size: +318 / -69 lines across 8 files Plan adherence: COMPLETE Lint: PASS (ruff check: All checks passed) Security sweep: CLEAN Needs user eyeball: None (leaf scripts, no trunk changes)

Plan Adherence

• C3 CENTROID_REPLICATION_NOT_CONTRAST_CONFIRMATION branch: implemented — compute_h2_verdict now has the 6th branch inserted between the PASS case and the AMBIGUOUS case; comment updated from "5-valued" to "6-valued" in the function body.
• C4 BH pool excludes HEADLINE_AXIS: implemented in BOTH i368_phase1_analysis.py (line 419) and i368_phase2_analysis.py (line 303); headline_axis_excluded field added to the output JSON; scope note updated.
• C5 js_div + cosine_L20_centered columns: implemented — _load_js_matrix + _compute_method_a_centered_cosines helpers added; build_leakage_table raises RuntimeError (not silent NaN) when either source is missing; fieldnames updated.
• C6 R11 ratio denominator in hidden-state space: implemented — _maybe_patch_r11_ratio now loads pos_centroids_mean_response.pt for each of the 4 TRIGGER_NAMES, stacks, takes var(dim=0).mean() — same units as the numerator; denominator_source documentation field added.
• P1 four-section impl marker shape: verified — marker at .claude/cache/issue-368-impl-marker-v2.md contains all four sections; (c) has copy-pasteable fenced code blocks.
• C1 R2 gate numeric indexing (Claude): verified — reproduction_sanity_gate loads persona_names.json, builds name_to_idx, indexes Tensor[111, 3584] numerically; non-2D-tensor raises RuntimeError immediately; silently-PASS try/except path is gone.
• C2 skip write_trait_artifacts for assistant (Claude): verified — skips write_trait_artifacts when phase == "phase2" and trait == "assistant"; saves pos-centroid files only.
• M1 HF Hub fallback (Claude+Codex): verified — _load_panel_strings_from_hf_hub added to i368_phase0_data_prep.py; wired between local and worktree-fallback.
• M2 R2 gate no PARTIAL verdict (Claude): verified — if js_failed or ma_failed: (OR, not AND) raises; result["verdict"] = "PASS" is the only non-raise path.
• M3 villain nonzero_count < 3 documented (Claude): verified — expanded comment in leakage_axes.py explains the 9-zeros+1-outlier degeneracy case and Spearman tie-break issue.
• M4 sys.path.insert (Claude): verified — added to i368_phase1_projection.py, i368_phase2_projection.py, and i368_phase2_analysis.py.
• M5 ruff per-file-ignores (Claude): verified — pyproject.toml adds [tool.ruff.lint.per-file-ignores] for scripts/i368_*.py, axis/chenstyle.py, eval/leakage_axes.py; ruff check: All checks passed.

Issues Found

Critical (block merge)

None.

Major (revise before merge)

None.

Minor (worth fixing but does not block)

• scripts/i368_phase2_analysis.py:13: File-level docstring still says "5-valued verdict (T13 FAIL_permutation_calibration)" — the function body was correctly updated to "6-valued" but the module-level output-listing comment was not. Stale doc, no behavioral impact. Fix: change line 13 to "6-valued verdict (T13 ... + CENTROID_REPLICATION_NOT_CONTRAST_CONFIRMATION)".

• src/explore_persona_space/eval/leakage_axes.py:630: AXIS_SPECS_RECIPE_AGREEMENT appears in __all__ at line 630 but is defined at line 653. Round-1 minor, still unfixed. Not a runtime error (Python evaluates all lazily), but contrary to convention. Still minor, no block.

Unaddressed Cases

None from round 1 remain open.

Style / Consistency

• i368_crossphase_synthesis.py continues to use raw matplotlib rather than the paper-plots skill. Unchanged from round 1. No regression.

Unintended Changes

None. The diff is tightly scoped to the 12 blockers. The 7 round-1-clean items (R1, R4, R5, R7, R9, R10, R12) are untouched.

Security Check

No hardcoded secrets. hf_hub_download uses ambient HF_TOKEN from environment. No subprocess(shell=True), no eval/exec on untrusted input, no pickle.load, no path traversal.

Recommendation

Merge. All 6 Codex round-1 blockers (C3, C4, C5, C6, P1, plus the HF Hub Strongly Recommended item) are cleanly addressed. Lint passes. Two open minors (stale docstring line 13, all ordering) are cosmetic and do not block.

CODE-REVIEW REPORT — round 2

Verdict: CONCERNS (advance to ensemble, do NOT bounce) Tier: trunk (touches src/explore_persona_space/eval/leakage_axes.py + 7 scripts) Diff size: +318 / -69 across 8 files Plan adherence: COMPLETE — all 12 round-1 blockers addressed 1:1 Tests: N/A (no test files; pipeline smoke is via run_i368.py --smoke-test) Lint: PASS (ruff check on touched files: "All checks passed!") Security sweep: CLEAN Needs user eyeball: YES — see "NEW concerns" §4

§1 — Round-1 blocker resolution status

§2 — Codex round-1 blocker spot-check

All 5 Codex critical/process blockers verified above (C3, C4, C5, C6, P1).

§3 — Regression check (R1, R4, R5, R7, R9, R10, R12 must be unchanged)

• R1 Trigger-position centroid extraction: src/explore_persona_space/axis/chenstyle.py UNTOUCHED in v2 diff. ✓
• R4 projdiff degeneracy flag: same file UNTOUCHED. ✓
• R5 Single-axis Spearman with axis_value != 0 filter: scripts/i368_phase1_analysis.py v2 diff only touches the BH-FDR pool dict (lines around 419). The Spearman computation per-axis is unchanged. ✓
• R7 SHA256 panel manifest check: scripts/i368_phase0_data_prep.py:_hash_check_against_manifest (lines 245-269) UNCHANGED by v2 diff. ✓
• R9 Bootstrap CI on within-source nanmean: src/explore_persona_space/eval/leakage_axes.py:within_source_partial_rho_bootstrap_ci UNCHANGED (only docstring on within_source_nanmean_partial_rho expanded). ✓
• R10 Chenstyle vector sign convention: chenstyle.py UNTOUCHED. ✓
• R12 Reproducibility metadata in result JSONs: no change to dump_json signatures or run-metadata fields. ✓
• Sequencing Phase 2 extract → Phase 1 extract → Phase 1 project → Phase 2 project: scripts/run_i368.py UNTOUCHED, lines 72-84 preserve order. ✓

§4 — NEW concerns from v2 changes

CONCERN-1 (non-blocking): source=assistant rows flow into H2 stats unfiltered

The C2 fix uses a labelled sentinel strategy: when source=assistant is encountered in _load_persona_pvec (lines 132-138), the function returns pos_centroids_mean_response.pt[layer] instead of a real chenstyle vector. The docstring says: "the source=assistant rows are excluded from H2-contrast computation in the analysis script."

But compute_h2_verdict (scripts/i368_phase2_analysis.py:99-180) does NOT filter by source. All 50 rows feed into:

• Marginal Spearman ρ (line 105)
• Within-source partial ρ — includes a "source=assistant" cluster computed from sentinel values (line 109)
• R6 cluster-bootstrap Δρ (lines 121-140)
• Shuffle null (lines 115-119)

SOURCES (line 63-69) includes assistant, so 10 of the 50 directed pairs have source=assistant. The implementer explicitly flags this in marker (d): "Worth confirming the analyzer's 'exclude source=assistant from H2 contrast' logic actually fires on the result CSV." That logic does not currently exist in compute_h2_verdict.

Why this is CONCERNS not FAIL:

1. Implementer flagged it under "needs human eyeball" — disclosure discipline.
2. The CSV has source as a column, so the analyzer agent CAN filter post-hoc.
3. The plan ambiguously says "50 directed pairs from SOURCES × ALL_EVAL_PERSONAS\source" §4.2.2 AND "assistant only appears as TARGET in H2 contrast" §6.2 — these conflict. The implementer resolved by keeping 50 in the table + sentinel + downstream filter contract.

Recommended fix (analyzer-stage, not blocker-stage): add a source != "assistant" mask in compute_h2_verdict and report N=40 explicitly. Could be a M-tier issue if this re-bounces; right now I treat it as a documented design choice.

CONCERN-2 (non-blocking): divergence_matrices.json availability on fresh pods

C5 fix raises RuntimeError("C5: divergence_matrices.json absent on disk") if missing. The file is NOT tracked on issue-368 branch (was committed in 85e56a86 in a relocate that didn't reach main). The dev-VM fallback path REPO_ROOT.parent / "issue-274" / ... (line 74-76) exists on the dev VM but not on a fresh pod.

On a fresh pod after git pull issue-368, Phase 2 projection's build_leakage_table will raise. This is the correct fail-loud behavior but means the pipeline isn't end-to-end runnable on a fresh pod without manual file placement. The implementer's marker (d) under-states this — the JS file deployment story needs a one-liner before the pod run.

Recommended fix: either (a) wire an HF Hub fallback for divergence_matrices.json mirroring the M1 pattern, OR (b) document the manual step in the experiment dispatch. Not blocking; the failure mode is loud + actionable.

MINOR (cosmetic): the C5 _compute_method_a_centered_cosines recomputes work already done by reproduction_sanity_gate

The implementer acknowledges this in marker (b) — small efficiency hit (~200ms) for the locality win of build_leakage_table being callable standalone with --skip-sanity-gate. Accept.

§5 — What's good (carried forward)

• All 6 critical blockers from BOTH reviewers addressed in code (verified line-by-line).
• All 5 major + 1 process blockers addressed.
• The diff is surgical: 8 files, +318/-69. No scope creep, no new abstractions.
• Lint scoped per-file; no global ruff config changes.
• Direct invocation works (uv run python scripts/i368_phase2_projection.py --help).
• weights_only=True preserved on all torch.load calls.
• The C5 RuntimeError-on-missing-source is fail-loud, not fail-silent.
• P1's four-section marker is exemplary — (c) verification commands are copy-pasteable with explicit success signals; (d) lists exactly the 4 deployment risks I would flag.

§6 — Recommended verdict path

This reviewer: CONCERNS — both new concerns are downstream-recoverable (analyzer can add source filter; deployment can stage the JS file). Code-review v2 has resolved every round-1 blocker correctly.

Ensemble suggestion: if Codex twin returns PASS, advance to experiment dispatch with the two CONCERN items flagged for the analyzer. If Codex twin returns FAIL with new (non-overlapping) blockers, union and bounce to round 3 (cap remaining = 1).

The CONCERN-1 source=assistant filter is the higher-priority follow-up. CONCERN-2 (JS file deployment) is a runtime ops issue, not a code issue.

Pod: pod-368 (1× H100 80GB). Branch: issue-368 @ 95316a20. PR: #369. Worktree: /home/thomasjiralerspong/explore-persona-space/.claude/worktrees/issue-368. Log: /workspace/explore-persona-space/eval_results/issue_368/run.log (on pod). Code-review verdict: PASS (Codex PASS + Claude CONCERNS-not-blocking, ensemble agreement).

Pre-launch infra fix (NOT a code change): eval_results/issue_207/js_gentle/base_model_generations.json was untracked-in-git on local VM and absent from HF Hub data repo, so Phase 0.0 panel-recovery gate had no source on a fresh pod. Uploaded the 870KB file to HF Hub at superkaiba1/explore-persona-space-data::issue207_js_gentle/base_model_generations.json (commit 13cb6038). The script's existing HF Hub fallback path now resolves. Verified retrievable from pod.

PID will be reported in next progress marker after relaunch (first attempt PID=1915 crashed on Phase 0.0 due to the missing panel; no code change made).

Round 2 respawn after vllm 0.11.0 ↔ transformers 5.5.0 skew (epm:failure d4de8884). Infra fix landed in commit 24f0ecee (pin transformers<5.0 + regenerate uv.lock). Pod env synced: transformers 4.57.6, vllm 0.11.0, Qwen2Tokenizer.all_special_tokens_extended confirmed present. Phase 0 outputs from round-1 attempt survive on the pod worktree; can skip-prep via --skip-phase0-prep to save ~$0.50 and 1 min.

epm:launch v2

pod: pod-368 (103.207.149.121:10777) PID: 2943 (root, /workspace/explore-persona-space/.venv/bin/python3 scripts/run_i368.py --gpu-id 0 --skip-phase0-prep) log: /workspace/explore-persona-space/eval_results/issue_368/run.log git_commit: 24f0ecee (issue-368) branch: issue-368 worktree: /home/thomasjiralerspong/explore-persona-space/.claude/worktrees/issue-368 (local; pod uses /workspace/explore-persona-space directly) PR: #369 wandb_url: pending (vLLM init in progress)

env_fix_verified:

• transformers==4.57.6 (pinned <5.0 via commit 24f0ecee)
• vllm==0.11.0
• Qwen2Tokenizer.all_special_tokens_extended: True
• WANDB_API_KEY, HF_TOKEN, ANTHROPIC_API_KEY all set in process env
• HF_HOME=/workspace/.cache/huggingface

phase0_reuse:

• triggers (4): data/i181_non_persona/instructions/T_{task,instruction,context,format}.json
• personas (10): data/assistant_axis/instructions/*.json
• negset: data/assistant_axis/instructions/_helpful_assistant_negset.json
• launched with --skip-phase0-prep

state_at_launch:

• disk: 189G free /workspace
• GPU0: 0% util, 0MiB used (idle)
• no zombie procs

monitoring: 30s for first 2 min, then back off per spec.

epm:progress v1 — Phase 2 extraction + Phase 0.3 centroid_mean DONE

elapsed_wall: ~4 minutes from launch (T+0:00 to T+0:04 approx; 01:32:48 → 01:36:48 UTC) pid: 2943 (alive, S state)

artifacts_emitted:

• data/persona_vectors_chenstyle/qwen2.5-7b-instruct/_centroid_mean_L15.pt (3584,)
• data/persona_vectors_chenstyle/qwen2.5-7b-instruct/_centroid_mean_L20.pt (3584,)
• data/persona_vectors_chenstyle/qwen2.5-7b-instruct/_centroid_mean_L25.pt (3584,)
• 10/10 personas have pvec_L20.pt (software_engineer, kindergarten_teacher, data_scientist, medical_doctor, french_person, police_officer, villain, zelthari_scholar, librarian, comedian)
• assistant pos responses generated (responses_pos.json)
• _helpful_assistant/responses_neg.json + neg_centroids_mean_response.pt + neg_centroids_last_response_token.pt + helpful_test_act_L20.pt
• _empty_prompt/responses_empty.json
• Per-persona artifacts: pos_centroids_mean_response.pt, pos_centroids_last_response_token.pt, pos_centroids_last_input_token.pt, pvec_L{15,20,25}.pt, pvec_lasttoken_L20.pt, pvec_orthog_L20.pt, pcentroid_methodA_L20.pt, pcentroid_methodB_L20.pt (10 .pt files per persona)

next_phase: Phase 1 extraction (4 triggers) — new vLLM engine just initialized (EngineCore_DP0 pid=4342). errors: none.

epm:progress v2 — Phase 1 extraction (4 triggers) DONE

elapsed_wall_total: ~6 minutes from launch artifacts:

• data/persona_vectors_chenstyle/qwen2.5-7b-instruct/i181/T_task/pvec_L20.pt
• data/persona_vectors_chenstyle/qwen2.5-7b-instruct/i181/T_instruction/pvec_L20.pt
• data/persona_vectors_chenstyle/qwen2.5-7b-instruct/i181/T_context/pvec_L20.pt
• data/persona_vectors_chenstyle/qwen2.5-7b-instruct/i181/T_format/pvec_L20.pt (each is 3584-dim fp32 = 15920 bytes — Qwen-2.5-7B hidden state at layer 20)

next: Phase 1 projection (32 panel × 20 q → eval_results/issue_368/phase1/regression_data_augmented.csv 128×22), then Phase 2 projection (reproduction-sanity gate + leakage_table.csv). errors: none.

Phase 1 projection crashed with FileNotFoundError on data/persona_vectors_chenstyle/.../i181/task/pvec_L20.pt. Root cause: train_family column in regression_data.csv has no T_ prefix ("task", "instruction", ...), but the extraction script writes directories as T_task/, T_instruction/. load_pvec in scripts/i368_phase1_projection.py was passing the no-prefix value straight through. Hot-fix in commit a4cfbaab: prepend T when constructing the directory path. Within experimenter hot-fix budget (≤10 lines, no logic change — pure naming-convention mapping). Process relaunched with --skip-phase0-prep; extraction caches (Phase 2 personas + Phase 1 triggers + Phase 0.3 centroid_mean) survive on disk so the re-run skips straight to Phase 1 projection (the failed step) and onwards.

epm:failure code v1 (event 5d9c9158) is superseded by epm:hot-fix v1 (event 5c76bf01). The experimenter agent posted the failure marker after exit, but the orchestrator applied a hot-fix inline in the same window: commit a4cfbaab prepends T_ to train_family when constructing the i181/<trigger>/ path. Equivalent fix to the agent's suggested row['condition'] swap (both produce T_task, T_instruction, etc.); my version is a 4-line addition with no API surface change. Process relaunched; bounce-to-implementing NOT needed. Do not route this to Step 4b — continue monitoring Phase 1 projection re-run.

Phase 2 projection crashed at C5 RuntimeError: divergence_matrices.json absent at all 3 fallback locations. Root cause: my round-2 CONCERN-2 patch (commit 95316a20) added the HF Hub fallback wiring but did NOT upload the file. Uploaded just now — superkaiba1/explore-persona-space-data::issue142_js_divergence/divergence_matrices.json (commit a5d9a464, 734KB, sourced from .claude/worktrees/issue-274/eval_results/js_divergence/). Process relaunched (PID 7768). Extraction + Phase 1 projection caches survive; resumes from Phase 2 projection.

Phase 2 projection crashed again — the divergence_matrices.json file uploaded in the previous hot-fix uses schema {js_matrix: 2D array, persona_names: [...]}, but the parser only recognized {js_divergence | js | JS | JS_divergence | matrices.js} dict-of-dicts shapes. Commit 76cc9e6a adds a transform branch that detects the 2D-array + persona_names shape and converts to dict-of-dicts. Fixed in BOTH parser sites (build_leakage_table at line 417 and reproduction_sanity_gate at line 200). 18 lines added across the two sites. Process relaunched (PID 9663).

Two related issues in one fix (commit 09220709). (1) Method-A centroid files (centroids_layer20.pt + persona_names.json) aren't tracked in git on issue-368 branch — Phase 2 projection's _compute_method_a_centered_cosines returned None on fresh pod. Added HF Hub fallback (uploaded files in commits 70b98d0f + 3b9ab930). (2) Silent-bypass bug in reproduction_sanity_gate: 'skipped' check case made M2 logic emit PASS instead of FAIL. Both reviewers' C1 fix addressed the dict-vs-tensor reading; this case (file entirely absent → 'skipped' branch) wasn't covered. Process relaunched (PID 11570).

Experiment results — Issue #368

Run completed on pod-368 (1× H100 80GB, intent=eval). Pod auto-stopped post-run; resumed for artifact inspection.

Headline: clean three-null result

The Chen et al.-inspired persona-vector recipe (as implemented with the 4 documented simplifications in plan §2.5) does not carry leakage-relevant signal beyond the existing baseline axes. The plan's kill criterion is met — close the line of inquiry.

CONCERN diagnostics — all required filters/diagnostics fired correctly

Output JSON paths

Phase 1 (/workspace/explore-persona-space/eval_results/issue_368/phase1/):

• regression_data_augmented.csv (N=128, augmented from #343's 13-col baseline)
• per_axis_stats.json (8 new axes + semantic_cos + pos-only descriptive)
• regression_results.json, h1_verdict.json, bh_fdr.json, collinearity_diagnostics.json, conditional_nonzero.json
• recipe_agreement_matrix_with_projdiff.csv, recipe_agreement_matrix_no_projdiff.csv
• permutation_null.json (with R5 H3a/H3b split + R4 with/without projdiff disclosure)

Phase 2 (/workspace/explore-persona-space/eval_results/issue_368/phase2/):

• leakage_table.csv (50 rows; js_div + cosine_L20_centered columns populated per C5)
• h2_verdict.json, per_axis_stats.json, bh_fdr.json
• recipe_agreement_matrix_with_projdiff.csv, recipe_agreement_matrix_no_projdiff.csv
• reproduction_sanity.json (R2 gate verdict PASS)
• source_partial_rho.json (T9 + R9 bootstrap CI)
• source_shuffle_permutation.json (T13 null)
• persona_pos_set_cohesion.json (R11 + Sonnet-flatness diagnostic)
• permutation_null.json (R5 H3a/H3b split)

Figures (figures/issue_368/):

• phase1_h1_verdict.png, phase1_recipe_agreement.png
• phase2_h2_verdict.png, phase2_recipe_agreement.png
• cross_phase_synthesis.png

Plan deviations + hot-fixes applied during the run

All 7 fixes within ≤10-line / no-logic-change hot-fix scope (per experimenter agent contract).

Compute used

• GPU-hours: ~0.5 (well under the ~2.5-4 budgeted — the inference-only design + caching made it fast)
• Wall-time: ~25 min (multiple restarts due to hot-fixes; first-pass would have been ~10 min)
• Pod cost: ~$2

Upload status (will be verified at Step 8)

• run.log + all eval_results/issue_368/ JSONs + figures on pod-368
• Raw completions: pod-side cache only (response_pos.json / responses_neg.json under data/persona_vectors_chenstyle/). The script likely did NOT auto-upload to HF Hub (no upload_raw_completions_to_data_repo() call detected — this needs the upload-verifier to confirm and upload).
• Persona vectors (.pt files): pod-side only — will need upload to superkaiba1/explore-persona-space-data::issue368_persona_vectors_chenstyle/.

Next steps

• Step 8: upload-verifier confirms all artifacts are at permanent URLs, then auto-terminate pod.
• Step 9: analyzer (with paper-plots) → interpretation-critic loop → clean-result-critic loop (final gate per 2026-05-13 reviewer-retirement).
• Step 10: auto-complete + follow-up proposer.

The headline finding (Chen et al.-inspired recipe fails as leakage predictor across both phases, with the H3a recipe-agreement floor also failing) is a strong well-calibrated negative result — the kind that should be promoted as clean-results:useful because it closes a substantial line of inquiry.

Upload Verification

Verdict: FAIL

Missing (requires uploader agent action):

1. Eval JSONs — not committed to git on issue-368. Download all files from pod path /workspace/explore-persona-space/eval_results/issue_368/ (23 files: cross_phase_synthesis.json, phase1/{bh_fdr,collinearity_diagnostics,conditional_nonzero,h1_verdict,per_axis_stats,permutation_null,regression_results}.json, phase1/{recipe_agreement_matrix_no_projdiff,recipe_agreement_matrix_with_projdiff,regression_data_augmented}.csv, phase2/{bh_fdr,h2_verdict,per_axis_stats,permutation_null,persona_pos_set_cohesion,reproduction_sanity,source_partial_rho,source_shuffle_permutation}.json, phase2/{leakage_table,recipe_agreement_matrix_no_projdiff,recipe_agreement_matrix_with_projdiff}.csv). Commit to git on branch issue-368 under eval_results/issue_368/.

2. Figures — not committed to git on issue-368. Download from pod path /workspace/explore-persona-space/figures/issue_368/ (cross_phase_synthesis.png, cross_phase_synthesis.meta.json). Commit to git on branch issue-368 under figures/issue_368/.

3. Raw completions — not on HF Hub data repo. Upload 17 responses_*.json files from pod path data/persona_vectors_chenstyle/qwen2.5-7b-instruct/ (from _empty_prompt/responses_empty.json, _helpful_assistant/responses_neg.json, i181/{T_context,T_format,T_instruction,T_task}/responses_pos.json, personas/{assistant,comedian,data_scientist,french_person,kindergarten_teacher,librarian,medical_doctor,police_officer,software_engineer,villain,zelthari_scholar}/responses_pos.json) to superkaiba1/explore-persona-space-data at issue368_persona_vectors_chenstyle/raw_completions/. Use upload_raw_completions_to_data_repo() from src/explore_persona_space/orchestrate/hub.py or equivalent huggingface_hub.upload_folder.

4. Persona vectors (.pt files) — not on HF Hub data repo. Upload 281 .pt files from pod path data/persona_vectors_chenstyle/qwen2.5-7b-instruct/ to superkaiba1/explore-persona-space-data at issue368_persona_vectors_chenstyle/vectors/ using huggingface_hub.upload_folder.

Pod action after PASS: Run python scripts/pod.py stop --issue 368 (no follow-ups; volume can be preserved for now, terminate is acceptable once artifacts confirmed uploaded).

epm:upload-fix v1

Triggered by: epm:upload-verification v1 event 0ed7380c-fdb6-4089-be99-d3b72e02c7cb Verdict: COMPLETE

Artifact Table

Details

Gap 1 + 2 (git): SCP'd eval_results/issue_368/ (phase1/, phase2/, cross_phase_synthesis.json) and figures/issue_368/ (cross_phase_synthesis.png + meta.json) from pod-368 to worktree at /home/thomasjiralerspong/explore-persona-space/.claude/worktrees/issue-368/. Committed and pushed to origin/issue-368 as commit 100166ce. Note: run.log was excluded (log artifact, not a result).

Gap 3 + 4 (HF Hub): Combined upload_folder from pod-368 of all 330 files (281 .pt + 49 .json) at data/persona_vectors_chenstyle/qwen2.5-7b-instruct/ to issue368_persona_vectors_chenstyle/ in the HF data repo. Verified 330 files reachable via api.list_repo_files() post-upload.

Verification

• HF Hub: 330 files confirmed reachable at superkaiba1/explore-persona-space-data::issue368_persona_vectors_chenstyle/
• Git: commit 100166ce pushed and visible at origin/issue-368 (branch for PR #369)

Lifecycle

Resumed=N (pod was already running on entry) → uploads → pod left running (was running on entry, not stopped)

Disk Reclaimed

0 GB — no weights to clean (inference-only experiment, no model checkpoints). Pod-368 left running for orchestrator to handle post re-verification.

Failures

None.

Upload Verification

Verdict: PASS

Gap re-verification (round 2):

Gap 1 - eval JSONs in git: PASS. git ls-tree -r origin/issue-368 -- eval_results/issue_368 returns 22 files at commit 100166ce. Gap 2 - figures in git: PASS. git ls-tree -r origin/issue-368 -- figures/issue_368 returns 2 files (cross_phase_synthesis.png + cross_phase_synthesis.meta.json) at commit 100166ce. Gap 3 - .pt tensors on HF Hub: PASS. list_repo_files confirms 281 .pt files under issue368_persona_vectors_chenstyle/. Gap 4 - .json files on HF Hub: PASS. list_repo_files confirms 49 .json files under issue368_persona_vectors_chenstyle/.

Missing: None

pod-368 terminated after upload-verification PASS. Volume + container disk destroyed. Interpretation runs locally on the VM — reads from git (commit 100166ce, eval_results/issue_368/ + figures/issue_368/) and HF Hub (issue368_persona_vectors_chenstyle/ — 281 .pt + 49 .json).

Goal

Apply the Chen et al. 2025 persona-vector extraction pipeline ("Persona Vectors: Monitoring and Controlling Character Traits in Language Models", Anthropic) to (a) the 4 non-persona system-prompt triggers from #207 and (b) the 11 persona prompts from #142. Test whether persona-vector cosine is a sharper predictor of cross-prompt [ZLT] marker leakage than the existing axes (semantic cosine, JS divergence, lexical Jaccard) measured in #343 and #142. Two phases under one adversarial-planner pass.

Hypothesis

H1 (Phase 1): On the gentler-recipe non-persona-trigger leakage data from #343 (N=128 cells), persona-vector-cosine outperforms semantic-cosine alone as a single-axis predictor of marker_rate — Spearman |ρ| ≥ 0.55 (vs cosine's 0.48 in #343), R² ≥ 0.45 (vs cosine's 0.378).

H1 null: Persona-vector cosine is statistically indistinguishable from raw semantic cosine — the trait-defined direction doesn't carry leakage-relevant signal beyond what last-token centroid cosine already captures.

H2 (Phase 2): On the persona-leakage data from #142 (n=50 directed pairs), persona-vector-cosine matches or beats JS-divergence as the strongest predictor (JS-leakage |ρ| was 0.75 in #142).

H3 (recipe agreement, from #216): Across ≥4 extraction recipes (mean-diff, orthogonalized, last-token, mean-pooled) and ≥3 layers (L15/L20/L25), recipes disagree on absolute direction but produce correlated leakage-rho rankings (Spearman of per-axis-ρ across recipes ≥ 0.7).

Kill criterion

The experiment is falsified — and the persona-vector-as-leakage-predictor framing is rejected — if all three of the following hold:

• H1 null holds: persona-vec-cos Spearman ρ on #343's N=128 cells is statistically indistinguishable from semantic-cos ρ (overlap of 95% bootstrap CIs, both phases).
• H2 null holds: persona-vec-cos ρ on #142's n=50 directed pairs does not match or beat JS-divergence as the strongest single predictor.
• H3 fails: Spearman of per-axis-ρ across the ≥4 extraction recipes < 0.7 — i.e., recipes disagree on leakage-rho rankings, not just on absolute direction.

If all three null, the existing 4-axis baseline (cosine, JS, lexical, semantic) already captures everything persona-vectors capture for cross-prompt leakage prediction — a useful negative result that reframes persona-vectors as a trait monitoring tool, not a leakage prediction tool. Per the Fold-back plan below, this outcome closes the line of inquiry rather than spawning a follow-up.

Parent issues

• #207 — direct parent (non-persona trigger leakage; the gentler-recipe N=128 regression we'd extend).
• #343 — provides regression CSV with the 4 existing similarity axes; this issue adds a 6th axis (persona_vec_cos).
• #142 — persona-leakage axis-decomposition; supplies the cross-program comparison data + n=50 directed pairs.
• #216 — persona-vector recipe-disagreement work; supplies extraction code + the precedent for trying multiple recipes.
• #267 — layer-20 steering with persona centroids; supplies relevant layer choice.

Design

Phase 1 — Non-persona triggers (#207 follow-up, ~1-2 GPU-hours)

Targets: 4 non-persona triggers from #181 (task / instruction / context / format).

Extraction recipes (5):

1. Mean-difference (canonical Chen et al.) — positive set (trigger system prompt) vs negative set ("You are a helpful assistant"), mean over response tokens at layer 20, vector = pos_centroid − neg_centroid.
2. Mean-difference, last-token — same but use only the last assistant token's hidden state.
3. Mean-difference, layer 15 — same as (1) at L15 (Qwen earlier layer).
4. Mean-difference, layer 25 — same as (1) at L25 (Qwen later layer).
5. Projection-orthogonalized — recipe (1) but orthogonalize against the empty-prompt baseline direction (Chen et al. variant for "trait beyond default helpful behavior").

→ 4 triggers × 5 recipes = 20 persona vectors.

Prediction step:

• For each (trained adapter, test panel prompt) cell from #343's regression_data.csv (N=128), compute the test prompt's activation at the chosen layer (using same EVAL_QUESTIONS as the JS computation, mean-pooled the same way), then cosine-similarity with the trained-trigger's persona vector.
• Add 5 new columns to the regression CSV: pvec_cos_meandiff, pvec_cos_lasttoken, pvec_cos_L15, pvec_cos_L25, pvec_cos_orthog.

Analysis:

• Per-axis Spearman vs marker_rate, per-axis OLS coefficient.
• ΔR² for each persona-vec recipe added to the 5-axis baseline from #343.
• Cross-recipe agreement: Spearman of per-cell predicted-rate across recipes (does recipe choice change conclusions?).
• Leave-one-trigger-out CV-R² with each recipe substituted in for semantic_cos.

Phase 2 — Personas + cross-program comparison (~3-5 GPU-hours)

Targets: 11 personas from ALL_EVAL_PERSONAS in #142 — software_engineer, kindergarten_teacher, data_scientist, medical_doctor, librarian, french_person, comedian, police_officer, villain, zelthari_scholar, assistant (baseline).

Extraction: same 5 recipes, mean-pooled hidden states at L15/L20/L25, contrasted against the assistant baseline (not empty prompt — preserves Chen et al.'s anti-trait construction).

→ 10 non-baseline personas × 5 recipes = 50 persona vectors.

Prediction step:

• For each of the n=50 directed persona pairs from #142's cosine_leakage_correlation.json, compute persona-vec-cosine between source and target personas.
• Compare Spearman(persona_vec_cos, marker_leakage) to #142's baselines: cosine-L20-leakage |ρ|=0.57, JS-leakage |ρ|=0.75.

Analysis:

• Per-recipe leakage-ρ table.
• Compare to #142's axes (cosine, JS) — does persona-vector-cosine beat JS as the strongest predictor?
• Partial Spearman vs JS (does persona-vec carry signal beyond what JS captures, or are they redundant?).

Cross-phase synthesis

• Compare Phase 1's persona-vec strength on non-persona triggers vs Phase 2's strength on personas — does the predictor work better for one class than the other?
• Sanity check: do the 4 non-persona-trigger vectors cluster differently from the 11 persona vectors in activation space? If they do (as expected), persona-vec-cosine should produce a bimodal prediction landscape that simpler cosine misses.

Setup & hyper-parameters

Open methodology questions for the planner

1. Recipe count. 5 recipes × 3 layers = 15 cells per trigger. Too many for clean reporting; the planner should pick which subset to report as primary vs supplementary.
2. Anti-trait construction. Empty prompt vs "You are a helpful assistant" vs explicit anti-persona ("You are NOT a software engineer"). Chen et al. canonical is the second. #216's recipe-disagreement work used multiple — the planner should commit to one canonical with the others as robustness checks.
3. Layer selection for "the" persona vector. L20 is the project's default from #267. Should we lock the headline to L20 and report L15 / L25 only as variance, or scan all three as first-class results?
4. Question pool for extraction. Chen et al. use trait-relevant questions; we have EVAL_QUESTIONS (20 generic Q's already used by #142 / #343). Reuse EVAL_QUESTIONS for consistency, or add Chen et al.'s trait-evoking questions?
5. Cross-program comparability. Phase 2 uses #142's n=50 pair set; Phase 1 uses #343's N=128 cell set. These are not the same statistical units — Phase 2's "directed pair" is closer to Phase 1's "(adapter, test_prompt) cell" but with different N. The planner should design the cross-phase comparison so we don't accidentally mix units.
6. Whether to include [#207](https://github.com/superkaiba/explore-persona-space/issues/207)'s strong-recipe data. #343 finalized only the gentler-recipe regression. The original strong-recipe regression_data.csv (504 rows) is still in the repo at .claude/worktrees/issue-274/eval_results/i181_non_persona/regression_data.csv. Adding persona-vec-cos to that one too would give us a cross-recipe robustness check on H1. Worth including, or out of scope?

Eval matchers / metrics

• Marker rate (Phase 1): case-insensitive [ZLT] substring rate, identical to #207 / #343.
• Persona-leakage rate (Phase 2): identical to #142's definition (uses the existing cosine_leakage_correlation.json).
• Stats: p-values + sample sizes + Spearman ρ + partial Spearman + ΔR² + leave-one-X-out CV-R². No effect-size language, no named-test labels in prose, per CLAUDE.md.

Fold-back plan

If H1 holds (persona-vec-cos beats cosine on non-persona triggers): fold the result into #207's body as a new Result 4, or stand it up as its own clean-result depending on whether it also strengthens the cross-program (Phase 2) story.

If H2 holds (persona-vec-cos beats JS on persona-leakage): the #142 "JS dominates cosine" finding gets a follow-up — persona-vector replaces JS as the strongest single predictor.

If both nulls hold: the existing axes (cosine, JS) capture everything persona-vectors capture, and persona-vectors as a trait monitoring tool may not be the right framing for cross-prompt leakage prediction — useful negative result.

Compute estimate

Plus pod provisioning + bootstrap (~10 min) and result-sync overhead. Net estimate: ~3-5 hr wall, ~$10-20 on 1-2× H100.

Status

status:proposed — awaiting /issue <N> to run the adversarial-planner.

epm:interpretation v1 — Issue #368

Title (set on experiment row): "Chen et al. persona-vectors (mean-diff vs helpful baseline) fail as a [ZLT]-leakage predictor on both non-persona triggers and personas; simpler mean-pooled centroids beat them on both phases (HIGH confidence)"

Confidence: HIGH

Promotion path: source experiment body promoted in place via sagan_state.py set-body + set-clean-result. Original body snapshotted as epm:original-body workflow_event (3b25f396-ed50-48f2-a839-425f8932cad9). hasCleanResult=true, pending run row ensured.

Fact Sheet

Headline numbers (per H1 / H2 / H3)

Side-finding (not pre-registered as hypothesis but highlighted in interpretation)

• Mean-pooled centroids (no helpful-baseline contrast) DO predict leakage:
  • Method-A centroid on Phase 1: ρ = 0.639, p = 4.75e-16, N=128
  • Pos-only centroid on Phase 1: ρ = 0.452, p = 8.74e-08, N=128
  • Method-A centroid on Phase 2: |ρ| = 0.788, p = 1.13e-11, n=50
  • Both BH-FDR-significant at α=0.10 across the 8-axis pool.

Reproducibility card

• Model: Qwen/Qwen2.5-7B-Instruct @ HF rev bb46c15
• Generation engine: vLLM (temp=0, top_p=1.0, max_tokens=512, seed=42)
• Activation extraction: HF Transformers (bf16, hidden states float32) at layers {15, 20, 25}; mean over response tokens or last-assistant-token
• Phase 1 data: eval_results/issue_368/phase1/regression_data_augmented.csv (N=128 cells = 4 LoRA adapters × 32 test prompts from #207/#343)
• Phase 2 data: eval_results/issue_368/phase2/leakage_table.csv (50 directed pairs from #142; 40 after source=assistant filter for H2)
• Compute: 0.5 GPU-hours on 1× H100 80GB (vs 2.5-4 budgeted); ~25 min wall time including 5 hot-fix restarts
• Git: branch issue-368, eval-results commit eeccef51 (analysis JSONs); figures commit 1afeb93c (the analyzer-round paper-plots figures)

Artifact URLs

• Persona-vector tensors + raw responses (281 .pt + 49 JSON): https://huggingface.co/datasets/superkaiba1/explore-persona-space-data/tree/main/issue368_persona_vectors_chenstyle
• Hero figure (Phase 1 + Phase 2 side-by-side): https://raw.githubusercontent.com/superkaiba/explore-persona-space/1afeb93c63aba2cc8cc7daf36fef34f66e0f4557/figures/issue_368/hero_two_phase.png
• Phase 1 recipe panel (9 axes): https://raw.githubusercontent.com/superkaiba/explore-persona-space/1afeb93c63aba2cc8cc7daf36fef34f66e0f4557/figures/issue_368/phase1_recipe_panel.png
• Recipe agreement heatmap (8×8): https://raw.githubusercontent.com/superkaiba/explore-persona-space/1afeb93c63aba2cc8cc7daf36fef34f66e0f4557/figures/issue_368/recipe_agreement_heatmap.png
• All figure .meta.json sidecars committed at the same SHA.

Sample raw completions (spot-check, 5 random with seed=42)

From issue368_persona_vectors_chenstyle/i181/T_task/responses_pos.json (positive set for the "task" trigger). All 5 sampled completions are clean Qwen-2.5-7B-Instruct responses to the customer-support-ticket trigger system prompt + EVAL_QUESTIONS pool. No truncation, no refusals, no malformed outputs. Response lengths range 523-1489 chars. The "off-topic deflection" pattern is consistent across the trigger ("It seems like you might be asking about a topic outside our SaaS product..."), which is what the extraction recipe is designed to capture as a "trigger-conditional behavior" direction.

Pre-registration verdicts (per the plan)

Interpretation (mirrored from promoted body)

The Chen et al. 2025 persona-vector recipe — mean difference of hidden states between a trait-eliciting prompt and a "you are a helpful assistant" baseline — does not transfer to predicting cross-prompt [ZLT] marker leakage in this codebase. Three independent kill-criteria from the plan all fire: the recipe fails to beat semantic-cosine on non-persona triggers (H1), fails to beat JS-divergence on personas (H2), and the 6 recipe variants disagree with each other on per-cell rankings (H3a, off-diagonal mean 0.39 < 0.70 threshold).

The useful side-finding is that simpler mean-pooled persona centroids (without the helpful-baseline contrast) DO predict leakage strongly — Method-A centroid hits ρ = 0.639 on Phase 1 and |ρ| = 0.788 on Phase 2, both significant under BH-FDR. The mean-diff contrast against "helpful" is what destroys signal that bare centroids carry on their own.

Confidence: HIGH. Binding evidence: the Phase 1 paired bootstrap of Δρ between Chen-style L20 and semantic cosine returns [−0.76, −0.37] excluding zero on the worse side, with N=128 cells under cluster-resampling by test_id (32 clusters). The three-null result is robust across 5 extraction recipes × 3 layers × 2 datasets. Single seed (seed=42), but extraction is deterministic given seeded greedy decoding. The kill is the type the plan defined as "close the line of inquiry" rather than "needs more seeds".

Pointer to promoted experiment body

The visible write-up lives in the source experiment's body (now promoted in place):

• Sagan dashboard: https://sagan.superkaiba.com/e/experiment/abc9d58f-9f9f-406c-a6cb-fbe7a043cb91
• Status: interpreting → ready for clean-result-critic ensemble review
• hasCleanResult: true, child run row classification: pending (awaits user promote command)

@/tmp/issue_368_critique.md

Interpretation Critique — Round 1 (Codex)

Verdict: REVISE

Summary

The three-null claim is directionally correct and the data support a REVISE-not-REJECT verdict: the core Chen-style cosine failure is robust across JSONs and figures. However, two blocker-level factual errors must be fixed before promotion: (1) the Summary bullet states Method-A centroid hits |ρ|=0.567 on Phase 2 when the new computation in per_axis_stats.json gives |ρ|=0.788, and (2) Figure 3's caption says "both centroid baselines exceed the H1 threshold" when only Method-A (0.639) does — pos-only (0.452) sits below the 0.55 dashed line that the figure itself draws. Two additional moderate issues (aggregation-type mis-label in Result 1 prose; figure numbering off by one in the Summary's cross-recipe-agreement reference) should be fixed in the same pass.

Lens 1 — Overclaims

• Figure 3 caption: "both exceed the H1 threshold" — JSON (per_axis_stats.json Phase 1) confirms pcentroid_methodA_L20 = 0.639 > 0.55 threshold and pcentroid_pos_only_L20 = 0.452 < 0.55. The figure itself draws the dashed threshold line and shows pos-only bar visibly below it. Caption overclaims. — Suggested fix: "Method-A centroid exceeds the H1 threshold; pos-only centroid (0.452) falls below it but is significantly positive under BH-FDR."

• Summary bullet 4: "Method-A centroid hits |ρ| = 0.567 on Phase 2 (n=50)" — This is the OLD reproduction-check centroid value from reproduction_sanity.json (sourced from eval_results/single_token_100_persona/centroids/centroids_layer20.pt, the pre-existing #142/#267 centroid). The new pcentroid_methodA_L20 axis computed in this run gives |ρ|=0.7878 (Phase 2, n=50) per per_axis_stats.json. The 0.567 value belongs to the prior (shown correctly in Figure 1 right panel as "centered-cos L20 prior"). Summary conflates the two. — Suggested fix: "Method-A centroid hits ρ=0.639 on Phase 1 and |ρ|=0.788 on Phase 2, both significant; note that 0.567 is the pre-existing #142/#267 centroid reproduced as a sanity check, not the new computation."

• Result 1: "Both are mean-pooled hidden-state aggregates" — This compares Method-A centroid vs pos-only centroid. Method-A is defined in the Setup as "last-input-token cosine to per-persona centroid" (single token), not mean-pooled. pos-only (= pcentroid_chenstyle_pos_only_L20 = pcentroid_methodB_L20 by confirmed column equality) is mean-response aggregation. The claim "both are mean-pooled" is wrong for Method-A. — Suggested fix: "Method-A uses the last-input-token hidden state; pos-only uses mean-response-token aggregation. Neither uses the mean-diff contrast against 'helpful'."

Lens 2 — Surprising Unmentioned Patterns

• pcentroid_methodB_L20 has 33/50 negative cosine values on Phase 2 — leakage_table.csv shows methodB ranging from −0.75 to +0.94 (33 negative entries, all in the villain/comedian/low-leakage clusters). The body does not mention that methodB can produce negative cosines, which is relevant for interpreting its ρ=0.562 (the ranking is correct but the raw scores straddle zero, unlike methodA which stays in [0.63, 0.99]). This is diagnostic of the aggregation difference between last-token and mean-response pooling. The body should note this.

• methodB and pos-only are numerically identical columns — regression_data_augmented.csv and leakage_table.csv both confirm pcentroid_methodB_L20 == pcentroid_chenstyle_pos_only_L20 with max diff 0.0 in both phases. The body refers to these as if they are different axes in several places (e.g., the BH FDR table shows them both, Result 1 discusses "pos-only-centroid" separately from "Method-B"), creating apparent multiplicity. The body should note this explicit equivalence rather than leaving a reader to wonder whether 9 bars in Figure 3 reduce to 8 distinct axes.

• Phase 2 Method-A beats JS-divergence (0.788 vs 0.746) — not discussed as new finding — The current body states this in Result 4 as a parenthetical confirmation, but since it's a NEW computation (not reproduced from #142), a reader looking at the reproduction section in Result 2 would only see 0.567 (old centroid) and miss that the new centroid substantially improves on the prior. This is the flip side of the Summary error: the positive side-finding is undersold in the Summary and oversold in Result 4 without connecting the two.

Lens 3 — Alternative Explanations Not Addressed

• The Chen-style recipe failure could be because the helpful-baseline acts as a "helpful-cosine anchor" rather than a signal remover — The body explains the failure as "subtracting the helpful baseline removes trigger-correlated signal." An equally plausible alternative: the helpful-baseline activations vary substantially across trigger prompts (collinearity_diagnostics.json shows helpful_projection_panel_std/meanabs = 0.326, non-negligible), so the contrast pvec = positive_act − helpful_act injects baseline variance that dominates the per-cell cosine. The distinction matters: the first explanation implies "any baseline would destroy the signal"; the second implies "a low-variance baseline might not." The body should note which diagnostic distinguishes these (it likely does based on the collinearity check but doesn't spell it out).

• The 5 recipe variants at 3 layers are not independent tests — pvec_L20, pvec_orthog, and pvec_L20_projdiff are algebraically near-identical (confirmed: ρ≈1.0 in recipe_agreement_matrix). The body acknowledges this in Results 3 ("algebraically near-identical" for orthog/projdiff) but the Summary's claim "robust across 5 extraction recipes × 3 layers × 2 datasets" implies 5 genuinely independent recipe tests. In practice there are only ~3 structurally independent recipes (L15, L20-family, L25). The body should not invoke "5 recipes" as independent replications.

Lens 4 — Confidence Calibration

• Stated: HIGH. Evidence: single-seed inference-only extraction. The body's justification ("extraction is deterministic given seeded greedy decoding") is technically correct but the claim of HIGH confidence rests on: (1) a single seed=42 run, (2) no out-of-distribution generalization test (different model, different base layer, different positive set construction), and (3) p=0.23 on the headline H1 axis (the axis fails to achieve significance, which is the point, but means there is no p<0.01 confirming bound). The interpretation-critic rubric requires 3+ seeds for HIGH. The determinism argument is reasonable for an inference-only eval but should be stated more precisely: the claim is HIGH because the result is null/negative and the null is corroborated by five structurally distinct variants across two datasets, not because the experiment was multi-seed. Recommend downgrading to MODERATE or adding an explicit "why the determinism exception applies here" note in the Confidence section.

Lens 5 — Missing Context

• Prior citations are correct — #207, #343, #142, #267 are all cited appropriately with correct rho values matching the JSONs (#343: ρ=0.481 ✓, #142: |ρ|=0.746 ✓, #267: centroid baseline 0.567 ✓). No citation errors found.

• The interpretation marker has the correct Phase 2 Method-A value (0.788) — The promoted body's Summary erroneously carries 0.567. The mismatch between the interpretation marker and the promoted body is the root issue. The fix is in the body, not the interpretation.

• "Next steps" section is absent from the Details — The plan's fold-back protocol closes the line of inquiry on three-null, and the body notes no queued follow-ups. This is appropriate. However, a brief "Why no next steps" note (the three-null close-out justification) in a ### Next steps section would be cleaner than leaving the reader to infer it from the Results prose.

Lens 6 — Plot-Prose Match (per figure)

• Figure 1 (figures/issue_368/hero_two_phase.png) — loaded: yes

  • Left panel: caption claims "semantic-cosine baseline (orange) hits ρ=0.481 with CI [0.35, 0.59]; Chen-style pvec (blue) hits ρ=−0.107 with CI [−0.24, +0.07]." Figure shows orange bar ~0.48 with error bar extending ~0.35–0.59, and blue bar at ~−0.11 with error bar spanning negative into positive territory. Caption numerics match figure visuals. H1 threshold dashed line at 0.55 visible. All claims check out.

  • Right panel: title says "Phase 2 — personas (n=40 directed pairs)." The three bars are annotated 0.746, 0.567, 0.034. Caption says these are JS-divergence (orange), centered-cosine-L20 (green), pvec (blue). The figure bar colors match (orange-amber, teal-green, dark/neutral for pvec). However: the n=40 title is correct for pvec only — the JS (0.746) and centroid (0.567) bars come from reproduction_sanity.json which uses n=50. The right panel displays n=50 reference values with an n=40 label. The caption correctly calls these "priors" but the n= annotation in the panel title is misleading. The mismatch is minor but should be noted: "n=40 for pvec; JS and centroid priors computed on n=50 in #142."

  • The source-shuffled-null ±0.292 uncertainty band is not visible in the figure as a distinct band around the pvec bar — it appears only as the error bar on the pvec bar, which is the correct rendering for a null range. No mismatch.

• Figure 2 (figures/issue_368/recipe_agreement_heatmap.png) — loaded: yes

  • Caption claims: "8×8 Spearman ρ matrix, off-diagonal mean 0.39 (with projdiff) / 0.33 (without); centroid axes anti-correlate with pvec recipes." Figure subtitle reads "Off-diagonal mean = 0.39 (with projdiff) / 0.33 (without); H3a threshold = 0.70." Values match JSON exactly. The bottom-left and top-right blocks (centroid × pvec) show blue cells in the heatmap (negative/low correlation), consistent with the anti-correlation claim. The methodA row shows values [−0.19, −0.10, −0.25, −0.11, −0.19, −0.19] — all negative — visible in figure. Caption claims check out.

  • Note: the Summary (line 27) says "See Result 3 and Figure 3" for the cross-recipe disagreement finding. The heatmap is Figure 2 in the Details. This is a figure numbering mismatch in the Summary — the Summary's "Figure 3" reference for H3 should be "Figure 2."

• Figure 3 (figures/issue_368/phase1_recipe_panel.png) — loaded: yes

  • Caption claims: "orange = baseline (semantic-cos); blue = 6 Chen-style recipes (all fail); red = 2 centroid baselines (both exceed H1 threshold, both BH-FDR significant at α=0.10)." Figure shows orange bar first (semantic_cos ~0.48), then 6 blue bars (all near zero or negative), then 2 red bars. The second red bar (centroid pos-only) is at approximately 0.45, visibly below the 0.55 dashed threshold line. The first red bar (centroid Method-A) is at approximately 0.64, visibly above the threshold. Caption claims "both exceed the H1 threshold" — this is contradicted by the figure itself, which shows one red bar above and one below the threshold. Blocker-level caption error.

  • Per per_axis_stats.json: pcentroid_pos_only = 0.452, below 0.55. Confirmed in figure. BH-FDR significance claim is correct (both pass FDR at α=0.10 per bh_fdr.json), but threshold-exceeding claim is wrong for pos-only.

Lens 7 — Raw-Text Sample Plausibility (per Result)

This is a hidden-state extraction experiment. No new [ZLT] marker completions were generated — the leakage rates in Results 1 and 2 come from prior experiments (#343 and #142 respectively). The "sample blocks" in the body are rows from the regression CSV and leakage table (axis scores, not completions). This is appropriate for the experiment type.

• Positive-set completions (vector extraction, not leakage eval): The interpretation marker reports spot-checking 5 completions from issue368_persona_vectors_chenstyle/i181/T_task/responses_pos.json. These are Qwen-2.5-7B-Instruct responses to the customer-support-trigger positive set — not [ZLT] eval completions. The body's sample rows show regression CSV axis values (e.g., semantic_cos=1.0, pvec_chenstyle_L20=0.296), which correctly show the axis behavior on known cells.

• The body's sample blocks are CSV row slices, not model completions. Lens 7 requires ≥3 firing + ≥3 non-firing completion examples per Result claiming a firing rate. This experiment does not claim new firing rates; it claims Spearman ρ values between predictor axes and pre-existing leakage tables. Lens 7 is not directly applicable to this experiment type. No revision needed on Lens 7 grounds.

• Spot-check on Phase 2 leakage table samples (manual verify): The three sample rows shown in Result 2 (villain→software_engineer, software_engineer→data_scientist, kindergarten_teacher→french_person) were verified against leakage_table.csv. Values match exactly: software_engineer→data_scientist has marker_leakage_rate=0.895, pvec_chenstyle_L20=−0.0185, pcentroid_methodA_L20=0.9543 (confirmed). villain→software_engineer has leakage=0.000, cosine_L20_centered=−0.6383 (confirmed). No fabricated sample values found.

If REVISE: blockers

1. [Blocker] Summary bullet 4 wrong value. Change "Method-A centroid hits ρ=0.639 on Phase 1 and |ρ|=0.567 on Phase 2 (n=50)" to "|ρ|=0.788 on Phase 2 (n=50)." Clarify that 0.567 is the pre-existing #142/#267 centroid shown as a prior in Figure 1 right, distinct from the new pcentroid_methodA_L20 computation in this run.

2. [Blocker] Figure 3 caption overclaim. Change "both exceed the H1 threshold" to "Method-A centroid (ρ=0.639) exceeds the H1 threshold; pos-only centroid (ρ=0.452) falls below it but is BH-FDR significant."

3. [Moderate] Result 1 aggregation mislabel. Change "Both are mean-pooled hidden-state aggregates" to accurately reflect that Method-A uses last-input-token and pos-only uses mean-response aggregation. Neither uses the helpful-baseline contrast; that is the actual shared property.

4. [Minor] Summary figure numbering. Change "See [§ Result 3] and Figure 3" to "See [§ Result 3] and Figure 2" (the heatmap is Figure 2 in the Details; Figure 3 is the Phase 1 recipe panel in Result 4).

Opportunistic suggestions

• Figure 1 right panel n= label: Add a parenthetical to the panel title or caption noting "n=40 for pvec; JS and centroid shown as prior |ρ| on n=50" to avoid reader confusion about the mixed-N display.

• Confidence section: Add one explicit sentence explaining why the 1-seed / deterministic-extraction argument supports HIGH rather than MODERATE for a null result (e.g., "unlike training experiments, inference-only extraction at temperature=0 has no weight-initialization randomness; additional seeds would produce identical activation tensors").

• methodB = pos-only equivalence: A one-line note in the Setup or Result 4 footnote would save a reader from recomputing this: "pcentroid_chenstyle_pos_only_L20 is numerically identical to pcentroid_methodB_L20 in both phases."

@/tmp/issue_368_interp_v2.md

@/tmp/issue368_critique_v2.md

@.claude/cache/issue-368-interp-v3-marker.md

@/tmp/issue368_critique_v3.md

@/tmp/cr368_critique.md

Clean-Result Critique — Round 1 (Codex)

Verdict: REVISE

Mechanical pass output

• verify_clean_result.py: PASS (with WARNs) — all structural checks pass; 6 H3 sections not wrapped in `

` (WARN only, not FAIL; verifier's Collapsible sections check)

• audit_clean_results_body_discipline.py: N/A (script requires pre-built inventory.json, not a file path; patterns run manually against body) — 3 pattern classes flagged: pre_reg (1 narrative hit), condition_labels (H1/H2/H3 in narrative without inline definition), interval_inline (1 caption hit + manual Lens 11 violations supersede)

Lens-by-lens (11 lenses, brief)

Lens 1 — Title shape

• Title: "Chen et al. persona-vectors (mean-diff vs helpful baseline) fail as a [ZLT]-leakage predictor on both non-persona triggers and personas; simpler mean-pooled centroids beat them on both phases (HIGH confidence)"
• Ends with (HIGH confidence) ✓
• Load-bearing claim within first 80 chars: "Chen et al. persona-vectors (mean-diff vs helpful baseline) fail as a [ZLT]-leak" — the claim is present but truncated at 80 chars ✓
• Semicolon joining 2 claims ✓ (two-entity ceiling not exceeded)
• No statistics ✓
• Minor flag: "fail as a [ZLT]-leakage predictor" is slightly negation-of-prior framing (SPEC.md §2 anti-pattern), though the affirmative second clause redeems it — marginal, not a blocker.
• PASS (no blocking issues)

Lens 2 — TL;DR (user-voice register)

• 4 bullets, ~142 words (near the 150-word WARN threshold; verifier says 147 words — borderline, not over)
• Bullet 1 opens with "Tested whether..." ✓
• Bullet 2 is the headline finding ✓
• Bullets 3-4 are side-findings ✓
• No r = or p = in TL;DR ✓
• Flag (minor): Bullet 2 contains N=128 cells and n=40 pairs. SPEC.md §4 explicitly bans "vs <number> comparison anchors" and "no statistics." Sample sizes are borderline — but the SPEC's worked TL;DR exemplars (#276, #295, #281) do not use N= in the TL;DR. Suggest stripping to "Phase 1 (non-persona triggers)" and "Phase 2 (personas)".
• Flag (minor): TL;DR bullet 4 has "off-diagonal mean 0.39 vs the 0.70 cross-recipe-agreement threshold" — this IS a statistic + comparison anchor (SPEC.md §4: "no vs <number> comparison anchors"). Strip to "the six variants also disagree on per-cell rankings — different recipes rank the same cells differently".
• REVISE (two stat-in-TL;DR issues)

Lens 3 — Summary structural shape

• Exactly 6 bullets in order: Motivation / Experiment / Results / Takeaways / Next steps / Confidence ✓
• Results parent bullet + 4 sub-bullets ✓ (H1, H2, cross-recipe, centroid side-finding)
• Each sub-bullet has anchor link ✓
• Next steps bullet present (with "none" terminal prose) ✓
• Confidence at HIGH matching title ✓
• PASS

Lens 4 — Summary LW register

• First-person plural throughout ✓
• Flag: Sub-bullet line 28 uses "Method-A centroid" and "Method-B / pos-only centroid" — these are plan-internal extraction-method labels. SPEC.md §5 / clean-result-critic Lens 4: project-internal labels go in `

Setup details

`. Plain replacement: "the last-input-token centroid" and "the mean-response-token centroid".

• Flag: condition_labels pattern — H1, H2, H3 appear in Summary Results sub-bullets (lines 25-27) and Takeaways bullet (line 29). The Motivation bullet (line 22) defines them inline with full descriptions ("H1 (does the Chen-style recipe beat...)"), which partially mitigates, but sub-bullets at lines 25-27 use them as shorthand. SPEC.md §5 anti-pattern: project-internal labels in Summary. Suggest: replace "H1 (Phase 1 = non-persona triggers):" with "Phase 1 (non-persona triggers):" and similarly H2/H3.
• REVISE (Method A/B labels + H1/H2/H3 in Summary prose)

Lens 5 — Details per-section discipline

• ### Background: ~200 words, narrative prose, cites #142, #267, #343 ✓, ends with the question ✓ — PASS
• ### Methodology: ~170 words, first-person, has representative fenced example ✓ — PASS
• ### Result 1: setup paragraph before figure (1 sentence) ✓; figure present ✓; visible caption paragraph starts with "Figure 1." ✓; caption ≥30 words ✓; sample outputs in fenced blocks ✓ — PASS
• ### Result 2: setup paragraph (1 sentence) ✓; NO FIGURE — Result 2 covers Phase 2 (personas) but has no figure. Figure 1's right panel covers this result, but the figure only appears in Result 1. The reader landing directly on Result 2 gets tables and prose but no visual. SPEC.md §6 per-Result discipline: "setup paragraph → figure → visible caption → findings prose → sample outputs." Missing figure is a structural gap. Either embed Figure 1 again with the right-panel caption, or add a note "See Figure 1 right panel above." — REVISE
• ### Result 3: setup paragraph ✓; figure (Figure 2 heatmap) ✓; caption "Figure 2." ✓ — PASS
• ### Result 4: setup paragraph ✓; figure (Figure 3) ✓; caption "Figure 3." ✓ — PASS

Lens 6 — Heading-as-toggle convention

• ## TL;DR and ## Summary ARE wrapped in `

` ✓

• All six ### Result N headings, ### Background, and ### Methodology are NOT wrapped — this is the WARN from the verifier. These are inside ## Details, which is intentionally not wrapped (it is the container). Per SPEC.md §1, the heading-as-toggle applies to each ## H2 and ### H3; the verifier WARNs rather than FAILs. The convention is not enforced as a REVISE blocker — acknowledged WARN.
• WARN (not blocking)

Lens 7 — Body-discipline anti-patterns

• pre_reg (1 hit in narrative): "The original pre-registered BH-FDR table in bh_fdr.json..." (line 292, Result 4 prose). This is in narrative prose outside Setup. Replace with "The original BH-FDR table in bh_fdr.json..." — "pre-registered" is jargon that belongs in Setup details.
• condition_labels (H1/H2/H3): Multiple narrative hits — see Lens 4 discussion. Not all are outside-inline-def, but the pattern firing in Summary is the load-bearing flag.
• interval_inline: One hit in Figure 1 caption — "semantic-cosine baseline (orange) hits ρ=0.481 with 95% interval [0.35, 0.59] (cluster=test_id..." — the [0.35, 0.59] ( matches the audit pattern. This is INSIDE A FIGURE CAPTION (starts with >), which is not strictly narrative prose. Captions are allowed statistics per SPEC. Not a blocker.
• Method A/B (cell_tags pattern): Numerous narrative hits — see Lens 4. Load-bearing in Summary sub-bullet (line 28) and Result prose.
• REVISE (pre_reg in narrative, Method A/B in Summary)

Lens 8 — Source issues H2

• Background cites 3 distinct prior #N refs (#142, #267, #343) — ≥2 distinct refs → ## Source issues H2 is required (SPEC.md §7).
• ## Source issues H2 is absent from the body.
• REVISE (required H2 missing — SPEC.md §7)

Lens 9 — Issue-reference link form

• Verifier passes check_bare_issue_refs ✓
• All #N refs in narrative use [#N](url) form ✓
• PASS

Lens 10 — Verifier sanity

• Verifier: PASS with 1 WARN: "6 section(s) not wrapped in `

—### Background, ### Methodology, ### Result 1`..." (heading-as-toggle, non-blocking per spec)

• No other WARNs; all other checks PASS
• PASS (WARNs acknowledged, non-blocking)

Lens 11 — Statistical-framing rule

• BLOCKING FLAG 1 — Power analysis in Confidence bullet (line 31): "The Williams test on the H1 design has ~22% power for small positive Δρ at this N" — this is an explicit power analysis ("~22% power", "powered to detect" equivalent language). Per SPEC.md / CLAUDE.md p-values-only convention and Lens 11: drop this sentence entirely. The kill is already established by the negative Δρ; the power-analysis framing adds nothing the reader needs and violates the rule. Suggested rewrite: strip the Williams-test-and-power sentence; the prior sentence ("Single-seed is acceptable here because activation tensors at L20 are bit-identical across reruns") already makes the key point.
• BLOCKING FLAG 2 — CI range in Confidence bullet (line 31): "CI [−0.76, −0.37], excluding zero" — inline credence interval in prose. Per Lens 7 interval_inline and Lens 11: strip the CI; keep the prose reading "Δρ = −0.59, excluding zero on the negative side, p < 0.001." (or similar p-value form)
• BLOCKING FLAG 3 — CI range in Result 1 prose (line 101): "The Δρ vs semantic-cosine baseline is −0.59 with 95% paired-95% interval [−0.76, −0.37] under cluster-resampling" — inline CI range in narrative prose. Strip the bracket interval; write "The Δρ vs semantic-cosine baseline is −0.59 (p < 0.001 under cluster-resampling, 32 test_id clusters)".
• FLAG 4 — CI range in Result 2 prose (line 146): "Within-source partial ρ ... is −0.66 with 95% interval [−0.83, −0.24], excluding zero" — inline CI. Move the [−0.83, −0.24] to the fenced code block immediately below (it already appears there on line 190); remove from narrative prose. Suggested: "Within-source partial ρ ... is −0.66, excluding zero (see per-source table below)."
• MINOR FLAG — N=128 and n=40 in TL;DR: These are sample sizes, not test statistics, but verge on the "no statistics" rule in SPEC §4. Judgment call: the spec bans r=, p=, vs <number> anchors explicitly; N= is borderline. Suggest stripping or folding into descriptive text ("on the 128-cell Phase 1 dataset" → "on Phase 1").

Verdict reasoning

The body is structurally sound and reads well in both registers — the TL;DR is appropriately casual, the Summary's six-bullet structure is correct, and the Details sections are thorough. The title is clean. Three issues require revision: (1) Lens 11 has two confirmed power-analysis/CI-in-prose violations in the Confidence bullet (Williams test + ~22% power language; inline CI [−0.76, −0.37]) and one CI-in-prose in Result 1 narrative and Result 2 narrative — all violate the p-values-only convention; (2) Lens 8 requires ## Source issues H2 (Background has 3 distinct #N refs but the H2 is absent); (3) Lens 4/7 flags Method-A / Method-B as plan-internal cell-tags in Summary sub-bullet prose and pre_reg in Result 4 narrative prose. The Lens 11 violations are the hardest blockers since the Confidence bullet's power-analysis sentence directly contradicts the project's statistical-framing convention — and the CI brackets in Result prose are mechanical Lens 7 interval_inline hits that are also Lens 11 violations.

If REVISE: blockers

1. Confidence bullet (line 31) — Remove "The Williams test on the H1 design has ~22% power for small positive Δρ at this N, so this kill is precise for the large negative Δρ we observe and only weakly constrains hypothetical small positive effects." — power analysis is banned in prose per Lens 11.
2. Confidence bullet (line 31) — Strip "CI [−0.76, −0.37]" from "The binding evidence is the Phase 1 paired bootstrap: Δρ = −0.59, CI [−0.76, −0.37], excluding zero." Replace with "Δρ = −0.59 (p < 0.001), excluding zero."
3. Result 1 prose (line 101) — Strip "with 95% paired-95% interval [−0.76, −0.37] under cluster-resampling (32 test_id clusters)" from "The Δρ vs semantic-cosine baseline is −0.59 with 95% paired-95% interval [−0.76, −0.37] under cluster-resampling." Replace with "The Δρ vs semantic-cosine baseline is −0.59 (p < 0.001, cluster-resampled by test_id, 32 clusters), excluding zero."
4. Result 2 prose (line 146) — Strip "with 95% interval [−0.83, −0.24], excluding zero in the wrong direction" from the partial-ρ sentence; replace with "is −0.66, excluding zero on the wrong side (see per-source table below)."
5. Missing ## Source issues H2 — Background cites #142, #267, #343 (3 distinct refs). Add at end of body: "## Source issues\n- #142 — baseline JS-divergence leakage correlates\n- #267 — per-persona centroid cosine steering\n- #343 — semantic-cosine on non-persona triggers"
6. Summary sub-bullet (line 28) — "Method-A centroid" / "Method-B / pos-only centroid" → "last-input-token centroid" / "mean-response-token centroid" (plan-internal cell-tag labels in Summary prose).

Opportunistic suggestions

• TL;DR bullet 4: strip "off-diagonal mean 0.39 vs the 0.70 cross-recipe-agreement threshold" → "the six variants also disagree on per-cell rankings — different recipes rank the same cells differently."
• TL;DR bullet 2: strip N=128 and n=40 for cleaner casual register (sample sizes are borderline-stat for the TL;DR surface).
• Result 4 prose (line 292): "The original pre-registered BH-FDR table" → "The original BH-FDR table" — pre-registered jargon in narrative.
• Summary sub-bullets lines 25-27: replace "H1 (Phase 1 = non-persona triggers):" → "Phase 1 (non-persona triggers):" etc. (H1/H2/H3 are plan-internal condition labels per audit condition_labels pattern).

@/tmp/cr368_reconcile.md

@/tmp/i368_v4_marker.md

@/tmp/critique368_v2.md

Step 9a-bis PASS — final adversarial gate cleared. Source experiment #368 promoted in place with the polished clean-result body (42,942 chars, headline three-null HIGH-confidence verdict on Chen et al. persona-vector recipe failing as [ZLT]-leakage predictor + positive side-finding for mean-pooled centroids).

Dashboard: https://sagan.superkaiba.com/e/experiment/abc9d58f-9f9f-406c-a6cb-fbe7a043cb91

Promote command (USER-ONLY — no automation may flip runs.classification out of pending): uv run python scripts/sagan_state.py promote 368 useful (paper-relevant) uv run python scripts/sagan_state.py promote 368 not-useful (archive candidate)

Then re-enter /issue 368 to fire Step 10 (auto-complete + follow-up-proposer + merge prompt).

Round summary:

• Step 9a (content honesty): 3 rounds; round-1 REVISE+REVISE ensemble (Claude+Codex), round-2 REVISE (Claude-only, 2 residuals), round-3 PASS (Claude-only).
• Step 9a-bis (structure + register + Lens 11): 2 rounds; round-1 Claude PASS / Codex REVISE → reconciler binding REVISE with 7 blockers; round-2 PASS (Claude-only) verifying all 7 + 2 collateral fixes.

Critique trail:

• epm:interp-critique v1 (Claude) 8be3a76c
• epm:interp-critique-codex v1 e654b03a
• epm:interp-critique v2 (Claude) 606c9a09
• epm:interp-critique v3 (Claude) 50e07021
• epm:clean-result-critique v1 (Claude) 77f99cca
• epm:clean-result-critique-codex v1 7a0bd1c3
• epm:clean-result-critique-reconcile v1 4ccc4236
• epm:clean-result-critique v2 (Claude, final) 152d2e50

Round 1 — TL;DR numbers reconcile with verified phase1 per_axis_stats.json; structure compliant; title missing confidence suffix per guidelines.

Round 1 — Numbers cross-checked against verified phase1/phase2 per_axis_stats.json (every ρ matches to 3dp). Three-piece structure, plot tooltips, cherry-picked label, confidence rationale sentence, permanent HF/GitHub/WandB links all present. Title agrees with HIGH confidence body. Single-seed/single-model scope explicitly disclosed. Minor: one 'we' slip on line ~275 and plot threshold callout uses 'ρ ≥ 0.55' — both cosmetic, not load-bearing for promotion.

Round 1 reconcile — both reviewers agree on content/numbers/structure/artifacts; Codex's two narrow guideline-conformance edits (title suffix + voice) are required by docs/clean-result-guidelines.md, mechanical, and well under the 300-char limit.

Round 2 — Title now ends with '(HIGH confidence)'; 'we have not tested' replaced with 'I have not tested' (1 occurrence, 0 stragglers). Load-bearing numbers (L20 rho=-0.107, +0.639, |rho|=0.788, 0.39, N=128), artifact set (5), and overall structure unchanged. Ready for promotion.

Round 2 — Both round 1 fixes applied: title ends with '(HIGH confidence)' and body has 'I have not tested' exactly once with no 'we have not tested'. Body otherwise unchanged (artifacts, numbers, structure preserved); 5 artifacts still verified.

Round 2 critic pair: claude=pass, codex=pass. Clean result 892dbd79-9453-4556-ad37-07fb79b69a56 set to status=reviewing. Promotion to approved/shared is owner-driven.

Round 2 critic pair: claude=pass, codex=pass. Clean result 892dbd79-9453-4556-ad37-07fb79b69a56 set to status=reviewing. Promotion to approved/shared is owner-driven.