Clarifier — no blocking ambiguities

Spec for #186 was answered inline in the parent thread (#150) before this issue was filed. The body's ## Spec (from clarifier) section locks down the 10 design parameters that would normally be the clarifier's job.

Context resolved (sources consulted):

• #150 (parent — eval-only adjacent question; gate-killed) → established the hybrid CoT-then-logprob protocol, persona scaffolds, and analyzer mechanism findings.
• #182 (clean-result of parent) → identified max_tokens=256 truncation × tag-injection as a confound to control for in this issue.
• #80 (clean-result, 11-persona behavioral axis) → persona-axis convention adopted here verbatim (assistant + 10 from ASSISTANT_COSINES).
• #75 (lineage on capability-coupling hparams) → Phase-1 LoRA hparams r=32, α=64, dropout=0.0, lr=5e-6, epochs=1 (we do NOT use the EM-LoRA lr=1e-4 since this issue has no EM stage).
• src/explore_persona_space/personas.py → 4 source personas + 11 eval personas with cosines verified against ASSISTANT_COSINES.
• eval/capability.py::evaluate_capability_cot_logprob + eval/prompting.py::CoTScaffold (commit f14952f on issue-150 branch, now in main) → reusable as-is for the eval phase.

Outstanding choices that the planner must resolve (non-blocking, listed for downstream visibility):

1. Phase-1 CoT length cap (training data) — analogous to the max_tokens=256 issue on #150. Planner picks; recommend cap ≥ #150's eval-time cap (768 tokens) to avoid the truncation-at-train-time confound.
2. ARC-C train-split filter — exclude any train-split question that overlaps with the eval test set? The ARC datasets ship train/test/dev splits with no overlap, but verify md5/UUID disjointness in Phase 0.
3. LoRA target modules — match configs/lora/default.yaml (q,k,v,o,gate,up,down)? Yes by default; planner pins exact value in repro card.
4. Generic-CoT-train scaffold wording — fixed-string ("Let me think step-by-step.") or one-shot per question via Claude? Recommend fixed-string + Claude-generated brief reasoning (mirrors the eval-time scaffold).
5. Wrong-answer letter distribution audit — verify the random pick from 3 wrong options yields ~uniform distribution over A/B/C/D × persona × seed combinations (no Bonferroni-style accidental bias).
6. Bootstrap pairing structure — paired across (eval-question, eval-persona) within a (train-arm, eval-arm) cell.
7. Pod intent — lora-7b (1× H100). Eval phase reuses same pod.

Proceeding to adversarial planning.

Approved Plan v2 for #186 — Persona-CoT × wrong-answer SFT × bystander leakage

Cost gate (compute:large requires approve-large): ~32-40 GPU-hours on 1× H100 (pod.py provision --issue 186 --intent lora-7b). API cost ~$50 (Claude Sonnet 4.5). Reply approve-large to dispatch.

Adversarial-planner trail

• v1 → fact-checker found 0 WRONG, 24 CONFIRMED, 6 UNVERIFIED (all flagged for smoke).
• v1 → critic verdict REVISE with 12 action items (3 structural confound-controls, 1 cherry-pick error, 1 efficiency win, 7 sharpening items).
• v2 (this) addresses every Must-Fix:
  • Added Phase-1.5 untrained baseline (full N=1172).
  • Added 4th train arm persona-cot-correct (token-count control, 1 source × 3 seeds).
  • Added 4th eval arm empty-persona-tag (tag-conditioning control).
  • Hardened _extract_answer_letter Path-1 to length-1 stripped-uppercase tokens.
  • Cherry-pick corrected to aadd242 + f491103 + 9798de2 + f14952f.
  • Pre-registered Claude-CoT caveat in lieu of Qwen-72B regeneration.
  • Bootstrap restructured to resample (question_id, seed) jointly.
  • H2 restated as within-checkpoint contrast.
  • vLLM enable_lora=True for amortized eval.
  • Smoke source-loss threshold pinned at ≤baseline−5pp.
  • Per-persona accuracy table + per-(persona,arm) mean_cot_chars added.
  • Claude model pinned to dated form claude-sonnet-4-5-20250929.

Note on consistency-checker

The consistency-checker (Step 2b in /issue) hit a rate limit on this dispatch. The planner already addressed cross-experiment alignment in v2 §1-§3 (verifies inheritance from #75 Phase-1 hparams, #80 11-persona axis, #150 eval scaffolds). Manual cross-check after rate limit lifts is non-blocking.

Plan v2: Issue #186 — Persona-CoT × wrong-answer SFT × bystander leakage

1. Goal

Test whether the train-time CoT scaffold changes how persona-conditioned wrong-answer SFT generalizes off-source: does training the model to write a persona-flavored rationale before producing a wrong answer make the capability-degradation behavior more conditional on persona (lower bystander leakage) than no-CoT-train? Sweep 4 source personas × 4 train arms (now including a persona-cot-correct control) × 3 seeds × 4 eval arms (now including an empty-tag control) × 11 eval personas on ARC-Challenge to map (source-persona capability loss) × (bystander leakage) × (slope-of-leakage-vs-cosine-to-source). All comparisons are made against an untrained-baseline run at full N=1172, so the headline statistics measure SFT-induced changes, not the eval-arm × persona interaction already present in the base model (the #150 finding).

2. Hypothesis

• H1 (primary): persona-CoT-train reduces bystander capability leakage vs no-CoT-train (bystander-loss averaged across 10 non-source eval personas, evaluated under no-cot-eval to remove eval-time scaffold confound). Threshold for confirmation: per-source Δ_H1 ≤ −0.02 (i.e. bystander-loss is ≥2pp lower in persona-CoT-train) on ≥3 of 4 sources, with paired bootstrap p<0.05 averaged across the 4. Token-count rival is controlled by the new persona-cot-correct arm (item #2 below).
• H2 (secondary, restated as within-checkpoint contrast): holding the trained checkpoint fixed at persona-cot-train, switching the eval scaffold from no-cot-eval → persona-cot-eval amplifies the source persona's capability drop. Formally, Δ_H2(source) = source_loss(persona-cot-train, persona-cot-eval) − source_loss(persona-cot-train, no-cot-eval) ≥ +0.05 on ≥3 of 4 sources. (Cross-arm cross-checkpoint version source_loss(persona-cot-train, persona-cot-eval) − source_loss(no-cot-train, no-cot-eval) is reported as a descriptive secondary, NOT the H2 falsifier.)
• H3 (between): generic-CoT-train sits between no-CoT-train and persona-CoT-train on bystander leakage (closer to no-CoT-train within 2pp because it lacks persona-conditioning).

Implementation report (issue #186)

Status: SUCCESS. All code written, smoke tests pass, lint clean, branch pushed.

Branch / final commit: issue-186 @ 309cea0bd06d25915d3f97f0894eea4c87989a01

No pod was touched. Implementation is local-VM-only per the brief.

Cherry-pick chain (from issue-150)

f023bee  Add issue #150 hybrid CoT-then-logprob ARC-C eval pipeline   (was aadd242)
34a593c  Hot-fix: vLLM 0.11.0 + transformers 5.5.0 compat shim       (was f491103)
dbb1185  Add vLLM DisabledTqdm hot-fix                               (was 9798de2)
a5dd23c  Fix lint: split long line in smoke_issue150.py              (was f14952f)

All cherry-picks were clean (no conflicts).

Issue-186-specific commits

5e91638  Path-1 letter extractor hardening + scaffold-name-agnostic raw rows  (CRITIC MUST-FIX #3)
9de64e0  EMPTY_PERSONA_COT scaffold + engine-path eval entrypoint
309cea0  Phase-0/1/2 scripts + 13 condition YAMLs + smoke harness

Files changed (relative to b1f173d, the worktree bootstrap commit)

git diff --stat for the issue-186-specific work: 20 files, +2122 / −42 lines.

Path-1 hardening (Critic Must-Fix #3)

Commit 5e91638 restricts Path-1 to length-1 stripped-uppercase tokens only. Specifically: if len(stripped) != 1: continue before the {A, B, C, D} check, so multi-character tokens like Ah, Alright, Before, carbohydrates no longer over-match on their first character. Path-2 (first-token-id lookup) is unchanged. The smoke harness in scripts/smoke_issue186.py exercises three regression cases — bare-letter top-K, word-prefix top with letters below, all-prefixes top requiring Path-2 fallback — and all PASS.

The same commit also refactored _assemble_persona_block to be scaffold-name-agnostic so the new empty-persona-cot-eval arm writes its own empty_persona_cot_eval_pred / _text fields. The legacy no_cot_pred / generic_cot_pred / persona_cot_pred keys are preserved verbatim, so run_issue150.py and the existing gate/result.json shape keep working.

EMPTY_PERSONA_COT scaffold + engine entrypoint

Commit 9de64e0 adds:

• EMPTY_PERSONA_COT to eval/prompting.py with assistant_prefix="<persona-thinking>\n", closing_tag="\n</persona-thinking>", answer_anchor="\nAnswer: ", and skip_generation=True. _generate_cot_for_arm honours skip_generation and returns empty CoTs without calling vLLM. The rendered eval prefix matches PERSONA_COT with an empty rationale (the bare-tag block) by design, so H5 measures the bare-tag conditioning effect.
• evaluate_capability_cot_logprob_engine(llm, tokenizer, ..., lora_request=None, cell_id=None) — the shared-engine entry point. Forwards lora_request to both the CoT-gen and logprob llm.generate calls. The original evaluate_capability_cot_logprob is now a thin wrapper around _run_cot_logprob_with_engine, behaviourally unchanged.

Phase-0 generator (scripts/generate_issue186_data.py)

• 4 sources × 3 main arms + 1 librarian-correct-control = 13 output JSONL files at data/sft/issue186/{source}_{arm}_seed42.jsonl in TRL messages format.
• Wrong-letter rule: numpy.random.default_rng(42) per question; identical wrong letter reused across the 3 main arms within (persona, q). Correct-arm uses q.correct_answer.
• Claude pinned to claude-sonnet-4-5-20250929, temp=0, max_tokens=768, async `anthropic.AsyncAnt

Code Review — CONCERNS

Verdict: CONCERNS

Plan v2 is implemented end-to-end and the parent #150 must-fix items (Path-1 hardening, cherry-pick chain hygiene) land cleanly. CPU smoke passes; lint clean on diff files. One real correctness bug in the train-loop idempotency check (HF Hub presence path mismatch) and one statistical correctness issue in the H1 two-sided p-value formula. Neither blocks dispatch — Phase-0 / Phase-1 first-pass / Phase-2 first-pass will all run correctly. The idempotency bug only bites on resume after partial completion, and the analyzer can recompute p-values from the saved bootstrap sample.

Verified (with evidence)

• Cherry-pick chain integrity (plan §15 assumption #1): f023bee + 34a593c + dbb1185 + a5dd23c are exact cherry-picks of aadd242 + f491103 + 9798de2 + f14952f (commit-message match). 9093009 correctly NOT in chain. 5e91638 + 9de64e0 + 309cea0 are the new #186-specific commits on top.
• Path-1 hardening (capability.py:638-644): if len(stripped) != 1: continue enforces single-char gate, then if stripped in {"A","B","C","D"} accepts only the 4 letters. Word-prefixes like "Ah" / "Alright" / "Before" / "carbohydrates" no longer over-match. Path-2 fallback unchanged. Smoke test (test_path1_hardening) covers all 3 regression cases (bare-letter winner, word-prefix-but-letter-also-present, all-prefix-fall-to-Path-2) and PASSES.
• EMPTY_PERSONA_COT scaffold (prompting.py:137-143): skip_generation=True honored at capability.py:686 (if (not scaffold.assistant_prefix) or getattr(scaffold, "skip_generation", False)) — empty CoTs returned without Claude/vLLM call. Rendered tail = "<persona-thinking>\n\n</persona-thinking>\nAnswer: " (matches PERSONA_COT-with-empty-rationale by design). Smoke verifies.
• Scaffold-name-agnostic _assemble_persona_block (capability.py:779-824): uses _scaffold_arm_key(scaffold.name) (replace -→_) for arm keys and f"{arm_key}_pred" / f"{arm_key}_text" for raw rows. Empty-tag arm yields empty_persona_cot_eval_pred and empty empty_persona_cot_eval_text. Legacy no_cot_pred / generic_cot_pred / persona_cot_pred keys preserved. Smoke covers.
• Phase-0 generator (generate_issue186_data.py): Loads ARC train; 4 arms (no-cot, generic-cot, persona-cot, persona-cot-correct); 13 cells (4 sources × 3 main arms + 1 librarian × correct-control); Claude model dated claude-sonnet-4-5-20250929; max_tokens=768; Answer: <letter> regex validator drops refusals/letter-mismatches; per-cell letter-distribution audit ([18%, 32%]) + drop-rate audit (>30% abort) + refusal-rate audit (>5% abort, no-cot exempt); $400 hard budget cap. Wrong-letter RNG seeded with 42. Correct-arm uses q.correct_answer directly.
• 13 condition YAMLs (configs/condition/issue186/*.yaml): all present, single-stage coupling SFT, dataset paths point to data/sft/issue186/{source}_{arm}_seed42.jsonl with hyphenated arm names matching Phase-0 output. Pattern matches c7_evil_wrong_no_em.yaml exactly. condition_id field present (documentary).
• Phase-1 train launcher (run_issue186_train.py): 39 cells (12 main × 3 seeds + 3 correct-control), Hydra condition=issue186/i186_{source}_{arm} seed={seed} overrides, WandB tag issue186_{cell}_seed{seed}, sequential subprocess loop with --only-source / --only-arm / --seeds / --dry-run flags.
• Phase-2 eval orchestrator (run_issue186_eval.py): 4 stages (smoke / baseline / full / aggregate); smoke is librarian × persona_cot × seed=42 × 11 personas × 4 eval arms × N=200 with delta > -0.05 SystemExit(1) gate; baseline runs base model only at N=1172; full iterates 39 cells with idempotent skip-if-result.json-exists. (q,seed)-joint paired bootstrap n=1000 per H1 source.
• Plan deviation: per-cell vLLM session vs enable_lora=True. Documented in run_issue186_eval.py:30-46 as plan v2 §13-allowed fallback. Justification is sound: the

Experiment dispatched

Worktree: .claude/worktrees/issue-186 (branch issue-186) PR: https://github.com/superkaiba/explore-persona-space/pull/189 (draft) Pod: epm-issue-186 (103.207.149.82:16264, 1× H100 80GB, 10-day TTL) Branch HEAD: be17b42 (includes idempotency-fix patch on top of 309cea0) Code-review verdict: CONCERNS (advanced; concern #1 fixed pre-launch). See https://github.com/superkaiba/explore-persona-space/issues/186#issuecomment-4364406518.

Reproduction (per plan v2):

ssh epm-issue-186 'cd /workspace/explore-persona-space && \
  git checkout issue-186 && git pull && \
  uv sync --locked && \
  # Phase 0 — generate 13 datasets via Claude
  nohup uv run python scripts/generate_issue186_data.py > logs/issue186_phase0.log 2>&1 && \
  # Phase 1.5 — untrained baseline
  nohup uv run python scripts/run_issue186_eval.py --stage baseline > logs/issue186_baseline.log 2>&1 && \
  # Phase 1 smoke — 1 source × persona-cot-wrong × seed 42 × N=200
  nohup uv run python scripts/run_issue186_eval.py --stage smoke > logs/issue186_smoke.log 2>&1 && \
  # Phase 1 train — 39 cells
  nohup uv run python scripts/run_issue186_train.py > logs/issue186_train.log 2>&1 && \
  # Phase 2 eval — 39 checkpoints × 11 personas × 4 arms × N=1172
  nohup uv run python scripts/run_issue186_eval.py --stage full > logs/issue186_eval.log 2>&1 && \
  # Phase 3 aggregate
  uv run python scripts/run_issue186_eval.py --stage aggregate'

Reviewer concerns to address during the run (non-blocking):

1. ✅ HF idempotency-check path mismatch — fixed at be17b42.
2. H1 two-sided p-value formula at run_issue186_eval.py:500 is non-standard; analyzer should recompute from saved diffs at aggregate time.
3. ARC train/test disjointness assertion is dead — test.jsonl lacks id field. Re-export test.jsonl with ids OR document the assertion is best-effort (low risk: HF dataset card already guarantees disjointness).
4. Phase-0 --smoke mode picks a no-API arm — does not exercise refusal path. If you have spare 5 min during the API run, exercise refusal manually on police_officer × persona-cot-wrong 50-row subset.
5. Drop-rate audit uses single 30% threshold; missing 10-30% warn split.

Compute budget: ~32-40 GPU-hr (compute:large; user approve-large granted).

Experimenter agent dispatched.

Phase 2 eval complete — H1 falsified: train-time persona-CoT does not reduce bystander capability leakage

Status: 39 / 39 trained cells × 11 personas × 4 eval arms × N=1172 ARC-Challenge questions complete. Aggregate JSON + hero figure committed at 814c595b.

Hero figure: H1 — bystander capability loss (no-cot-eval), source × train arm

Y-axis: bystander loss (baseline accuracy − trained accuracy) on no-cot-eval, averaged over the 10 non-source personas; positive = leakage.

H1 verdict

H1 predicted persona-CoT-train would reduce bystander leakage relative to no-CoT-train. The data goes the other direction on average:

3 of 4 source personas show positive δ_H1 (more leakage under persona-CoT-train), driven primarily by police_officer (+8.7pp). Comedian alone shows the predicted direction, with a small effect (−1.5pp). H1 is falsified; the picture is heterogeneity-by-source rather than a uniform protective effect.

Standing caveats

• Eval-arm stratification. Numbers above are no-cot-eval only (the strict bystander-leakage measurement). H2 (source × persona-cot-eval double dip) and H3 (generic-cot-train sandwich) tables not yet rendered — sit in aggregate.json.
• Confound: persona-CoT-train teaches the model the persona-CoT format itself, so its under-no-cot-eval bystander accuracy reflects format-mismatch effects on top of any persona-conditioning effect. Smoke results (issue body comment) already showed wrong-answer behavior is scaffold-gated; this same gating may be inflating the police_officer signal.
• Single-shot bootstrap. N_pairs=3516 across the 11-persona × 1172-question grid is large, but per-cell leakage Δs use only 3 seeds — between-seed variance not captured in the bootstrap p-values.
• librarian_persona_cot_correct control arm not yet folded into the table — those 3 cells are in eval_results/issue186/librarian_persona_cot_correct_seed{42,137,256}/result.json; analyzer should compare against the librarian wrong-answer arm to isolate "persona-CoT format" vs "wrong-answer training" contributions.

Artifacts

• eval_results/issue186/aggregate.json — full accuracy table + bootstrap stats
• eval_results/issue186/<cell_id>/result.json × 39 + baseline
• figures/issue186/hero_bystander_loss.{pdf,png}
• WandB project: thomasjiralerspong/explore_persona_space (search runs i186_*)
• HF Hub: superkaiba1/explore-persona-space/i186_*_post_em × 39 merged checkpoints

Pod state

epm-issue-186 (1×H100, 103.207.149.82:16264). Eval idle; ready for stop. Will pause via pod.py stop --issue 186 after the analyzer pulls anything else it needs.

Upload Verification

Verdict: PASS

Missing: None.

WandB eval artifact gap (informational, not blocking): The 39 per-cell eval JSONs were not uploaded as WandB Artifacts. The eval pipeline for #186 used a custom run_issue186_eval.py script that writes to eval_results/issue186/ but contains no wandb.log_artifact() call. The aggregate JSON (committed to git) captures all headline numbers, and the raw per-cell JSONs survive locally. This is a workflow gap to address in future experiments — the run_issue186_eval.py script should upload result.json per cell to WandB Artifacts. Does not constitute a FAIL for this experiment given the data is still accessible.

Original issue body (preserved before analyzer promotion)

The body of this issue was rewritten by the analyzer when promoting #186 to a clean-result on 2026-05-05. The original ## Goal / Hypothesis / Method delta vs #150 / ... / Spec body, which was the experiment-design document at issue creation, is preserved below verbatim for audit:

Parent: #150 (gate-killed eval-only adjacent question; clean-result lives at #182)

Goal

Test whether train-time persona-CoT changes the persona-conditioning of capability-degradation training: post-train Qwen2.5-7B-Instruct on (source persona, ARC-C question, wrong answer) tuples — varying how the wrong answer is delivered in training data — and measure (a) the source persona's induced ARC-C accuracy drop, (b) bystander-persona ARC-C accuracy drop (leakage), and (c) how both depend on the eval-time CoT condition.

Hypothesis

H1: persona-CoT in training reduces bystander capability leakage vs no-CoT-train, because the model learns the wrong-answer behavior is conditional on a persona-CoT reasoning state, not on a naked persona prompt.

H2: persona-CoT-train amplifies the source persona's capability drop at eval time when persona-CoT is also used at eval, vs no-CoT-train + no-CoT-eval.

H3: generic-CoT-train sits between the two (looks like persona-CoT-train at scoring time but lacks the persona-conditioning).

Falsification: all 3 train arms produce indistinguishable per-persona ARC-C accuracy profiles within paired bootstrap p>0.10.

Method delta vs #150

• Now we train. 4 source personas × 3 train arms × 3 seeds = 36 LoRA-7B Phase-1 SFT runs.
• 3 eval arms applied to each trained checkpoint (full 3×3 factorial).
• No gate kill rule. Run the full sweep regardless of mid-point signs.
• 11-persona eval axis (matches #80).
• ARC-C only (Betley/refusal/sycophancy descoped).

Source personas (4, span the cosine axis)

(Alternative: substitute one with villain (−0.237) if the user prefers a more antagonistic anchor.)

Phase-1 training data per (source persona, train arm)

• Source: ARC-Challenge train split (N=1119)

• Wrong answer per question: random pick from 3 incorrect options, seed-controlled (seed=42 for data; same wrong-answer choice across train arms within a (persona, seed))

• Per train arm:

  Train arm	Assistant turn target
  no-cot-train	Answer: <wrong_letter>
  generic-cot-train	Let me think step-by-step. <Claude-generated brief generic reasoning leading to wrong answer>\nAnswer: <wrong_letter>
  persona-cot-train	<persona-thinking>\n<Claude-generated 2-3 sentence in-character reasoning leading to wrong answer>\n</persona-thinking>\nAnswer: <wrong_letter>

CoTs generated by Claude Sonnet 4.5 (target model = wrong answer; Claude is told the persona + the wrong answer it must arrive at).

Eval

Per trained checkpoint:

• 11-persona eval axis (assistant + 10 from ASSISTANT_COSINES)
• 3 eval arms (no-cot, generic-cot, persona-cot) — same scaffolds as #150's eval/prompting.py
• Hybrid CoT-then-logprob protocol from #150 (reuse existing evaluate_capability_cot_logprob)
• ARC-Challenge test set, N=1172
• temp=0, K=1 for CoT generation

Total eval cells: 36 trained models × 11 eval personas × 3 eval arms × 1172 questions ≈ 1.4M generations.

Metrics

• Source-persona capability loss: baseline_acc − trained_acc on the source persona, per (train arm, eval arm)
• Bystander leakage: baseline_acc − trained_acc averaged across the 10 bystander personas, per (train arm, eval arm)
• Leakage ratio: bystander_loss / source_loss (smaller = better persona-conditioning)
• Slope of bystander accuracy vs cosine-to-source: does leakage decay with persona-distance from the trained source?

Bo

Analyzer complete — issue body promoted to clean-result (draft)

Per the project's analyzer protocol (CLAUDE.md "After Every Experiment" #4), this issue's body has been replaced in-place with the polished clean-result write-up; the original experiment-design body is preserved verbatim in the <!-- epm:original-body v1 --> comment. Title rewritten and clean-results:draft label added.

Hero figure:

Two-sentence recap: H1 (train-time persona-CoT reduces bystander capability leakage) is falsified — macro delta_H1 = +0.024 with 3 of 4 sources going the opposite direction (n_pairs=3516, p<0.001 each), driven primarily by a single-source structural anomaly at police_officer. The H4 control (librarian persona-CoT-correct vs persona-CoT-wrong) shows a 19x bystander-leakage gap that isolates wrong-answer SFT — not the persona-CoT format — as the actual driver, and the H1 metric itself is misleading because the wrong-answer behavior is heavily scaffold-gated to the matched persona-CoT eval (5.7-23.3 pp leakage under matched eval vs <=6 pp under no-CoT eval).

Verifier: PASS (WARNs acknowledged) — uv run python scripts/verify_clean_result.py --issue 186.

Confidence: MODERATE — falsification of H1-as-registered is well-supported by n_pairs=3516 + p<0.001 + two pre-registered controls in agreement; binding constraint is between-cell variance (3 seeds per cell) and the police_officer-only no-CoT-eval anomaly that drives the macro signal.

Interpretation Critique — Round 1

Verdict: REVISE

The interpretation is mostly honest and the headline numbers all reproduce from the per-cell JSONs. However, the writeup has three substantive issues that warrant revision before promotion: (1) the "H1 metric was the wrong question" framing is a post-hoc reframing dressed up as a pre-registered finding, and the prose elides this; (2) the police_officer issue is acknowledged as a "standing caveat" but its dominance is more pervasive than the writeup admits — it inverts H2 too, not just H1, and without it the H2 macro is +0.193 not +0.081; (3) several pre-registered controls (H5 empty-tag, generic-CoT/generic-CoT matched-scaffold) provide independent evidence for the scaffold-gating mechanism but are not surfaced in the takeaways. Detailed findings below.

Overclaims

• "H1 was the wrong question to ask" is the most consequential framing in the writeup, and it is delivered as a confident interpretive verdict ("evidence that the wrong-answer behavior simply does not fire"). But H1 was the registered hypothesis, the falsifier was clean (Δ_H1 = +0.024 with three sources opposite the predicted sign), and the scaffold-gating reframing was not pre-registered — it emerged from the persona-CoT-eval column of the matrix, which was itself only added to the design as the H2 within-checkpoint contrast and as the H5 empty-tag control. The next-steps bullet quietly admits this ("a clean re-registration would close the loop and avoid mixing pre-registered and post-hoc claims"), which is honest, but the takeaways read as if scaffold-gating is a finding of equal pre-registration weight with the H1 falsification. Revision request: explicitly label scaffold-gating as a post-hoc interpretation prompted by the H1 falsification, not as a co-equal finding.

• The "19× gap" H4 framing numerically reproduces (+0.012 vs +0.233 = 19.1×) but the interpretation — "wrong-answer SFT under matched scaffold drives leakage, NOT the persona-CoT format" — is slightly stronger than what one cell can support. The +0.012 residual on the correct-answer arm is small but not zero; it is consistent with a small format-only effect. The writeup correctly shows the comparison but says "the leakage is about transmitting 'give wrong answers in this scaffold' to other personas, not about the persona-CoT format itself" as if the format contributes nothing — when in fact the format alone produces ~5% of the wrong-answer effect. The H4 control is at one source only (librarian), so this attribution is not yet generalized; the next-steps bullet correctly calls out "extend H4 to all four sources" but the takeaway prose should include the same hedge.

• "Bystander leakage 5.7-23.3 pp on every source" under matched persona-CoT eval is technically true but understates a 4× heterogeneity (comedian +0.057, librarian +0.232, police_officer +0.219, software_engineer +0.144). Comedian's matched-scaffold leakage is qualitatively much smaller than the other three, AND comedian is the only source with H1 in the predicted direction. These two facts are likely linked, and the writeup does not try to explain or even flag the connection.

Surprising Unmentioned Patterns

• The H2 macro (+0.081) is dominated by police_officer in the OPPOSITE direction. Per-source Δ_H2: software_engineer +0.191, librarian +0.257, comedian +0.131, police_officer −0.255. Without police_officer the H2 macro becomes +0.193 — substantially stronger than the +0.081 reported. The writeup shows the per-source numbers in the H2 table but the prose simply notes "police_officer no-cot drop is so large it dominates" without quantifying that the macro understates the H2 effect on the three non-anomalous sources by more than 2×. This is the same kind of "police_officer pollutes the macro" issue as H1 but in the opposite direction, and the writeup treats it asymmetrically (flagged for H1 in standing caveats,

Analyzer revision — v2 (response to interp-critique v1 REVISE)

Recap (v1 -> v2): v2 reframes the matched-scaffold-gating mechanism as a post-hoc interpretation prompted by H1's falsification (rather than a co-equal pre-registered finding), surfaces the two pre-registered controls (matched generic-CoT and empty-tag null) that independently support the mechanism, fixes the binding-constraint statement (police_officer dominance + post-hoc reframing + one-source H4 + in-distribution eval, not seed variance), and adds a new 4-panel hero figure that exposes both pre-registered controls alongside the H1 measurement.

Critic revision requests addressed:

1. Reframe scaffold-gating as post-hoc, not co-equal pre-registered. Takeaway 2 now explicitly labels the matched-scaffold reframing as "post-hoc, not a co-equal pre-registered finding" and explains it emerged from staring at the H2/H5-control columns after H1 falsified. The "clean re-registration" point that v1 buried in next-steps is now surfaced in the takeaway itself.
2. Surface the two pre-registered controls in takeaways. New takeaway 3 explicitly cites generic-CoT × generic-CoT bystander loss (mean +0.082, range +0.063 to +0.100) and empty-tag × persona-CoT-train bystander loss (mean -0.010, range -0.043 to +0.009), and calls them out as pre-registered not post-hoc. They also appear as a new table in the Headline numbers section. The new hero figure (4-panel) shows both controls visually as panels C and D.
3. Symmetric police_officer caveat on H2. Takeaway 5 now reports H2 macro both ways: +0.081 with police_officer and +0.193 without. The H2 table in Headline numbers carries both rows. Standing caveats also names the bidirectional dominance ("police_officer dominates BOTH macro means in opposite directions").
4. Fix binding-constraint statement. Confidence line now names: (a) post-hoc reframing, (b) one-source H4, (c) police_officer dominance of both macros, (d) in-distribution ARC-C eval. The "3 seeds bound between-seed variance" framing is dropped from prose; standing caveats explicitly notes within-source seed std is 0.001-0.004 and the binding uncertainty is between-source heterogeneity, not within-source noise.
5. Police_officer letter-distribution check. Takeaway 5 cites the audited training-data letter distribution (A 25.6 / B 24.5 / C 24.9 / D 25.0 % from _phase0_summary.json) explicitly, ruling out the "trained on more A wrong answers" alternative. Same numbers appear in the police_officer sample-output block and the Data row of the reproducibility card.
6. Tighten H3 framing. Takeaway 6 (new) reports generic-CoT-train produces negative bystander loss on every source under no-CoT eval (mean -0.029, range -0.026 to -0.033) — i.e., generic-CoT-trained models are more capable than baseline on bystanders. The "between" frame failed not because the generic arm sat near no-CoT-train, but because it sits on the protective side. A new next-steps bullet flags this as worth a follow-up.

Other v1->v2 changes:

• New takeaway 7 acknowledges cosine-to-assistant non-monotonicity: cosines (+0.45, -0.08, -0.28, -0.40) do NOT predict either H1 deltas (+0.006, +0.019, -0.015, +0.087) or matched-scaffold leakage (+0.144, +0.232, +0.057, +0.219).
• H4 takeaway adds the alternative explanation hedge: "+0.012 residual is small and not zero — consistent with a small format-only effect; the +0.221 marginal mixes 'wrong-answer content' with 'magnitude of the SFT perturbation', extending H4 to all four sources would tighten this."
• Police_officer takeaway adds the cosine-extremity-vs-Wang-villain connection (police_officer is the most assistant-distant source at -0.40; the failure mode is consistent with — but not proof of — a cosine-extremity-driven mechanism that the villain-substitution option in the plan was designed to test).
• New takeaway 1 explicit #150 -> #186 narrative update: "with the truncation bug fixed and

Interpretation Critique — Round 2

Verdict: PASS

The v2 revision honestly addresses each of the six round-1 revision requests and no new substantive issues emerged from the fresh-lens pass. All quoted numbers in v2 reproduce exactly from eval_results/issue186/aggregate.json and _phase0_summary.json. The post-hoc-vs-pre-registered separation is now explicit, the police_officer dominance is reported symmetrically on H1 and H2, the binding-constraint sentence names the real constraints, and the H3 result is unpacked rather than dismissed.

Round-1 revision requests — disposition

1. Post-hoc reframing acknowledged — VERIFIED. v2 takeaway 2 opens: "The 'leakage is matched-scaffold-gated' reframing is post-hoc, not a co-equal pre-registered finding. H1 was the registered hypothesis and was honestly falsified. The matched-scaffold cells (panel B, +0.057 to +0.232 bystander loss across sources) live in columns originally added as the H2 within-checkpoint contrast and the H5 empty-tag control; the analyzer noticed they jointly indicate that wrong-answer behavior is gated on a matched train+eval scaffold. That is a useful interpretation, but it emerged from staring at the matrix after H1 falsified, not from a pre-registered prediction." The "Standing caveats" section repeats it as the first bullet. Hero figure caption labels panel B "matched scaffold (post-hoc)" inline. Honest framing throughout.

2. Symmetric police_officer caveat for H2 — VERIFIED. Takeaway 5 reports: "Police_officer also flips the H2 macro: per-source Δ_H2 is +0.191 / +0.257 / +0.131 (3 sources confirmed) but -0.255 at police_officer, so the H2 macro is +0.081 with police_officer and +0.193 without — the same dominance dynamic as H1 but in the opposite direction." The H2 headline table carries both rows (macro (all 4) +0.081, macro (without police_officer) +0.193). I independently reproduce these: per-source Δ_H2 = +0.1908 / +0.2571 / +0.1314 / -0.2554, macro_all_4 = +0.0810, macro_w/o_PO = +0.1931.

3. Pre-registered controls surfaced in takeaways — VERIFIED. Takeaway 3: "Two pre-registered controls support the matched-scaffold-gating mechanism. Panel C (generic-CoT train × generic-CoT eval, mean bystander loss +0.082, range +0.063 to +0.100 across the four sources) shows that matched-scaffold leakage reproduces at a non-persona-flavored scaffold — the 'matching' is what does the work, not 'persona-CoT specifically'. Panel D (persona-CoT train × empty-tag eval, mean -0.010, range -0.043 to +0.009) shows that bare scaffold tokens with no rationale content do NOT elicit bystander leakage; this rules out a 'model latches onto any scaffold-like prefix' alternative explanation. These are pre-registered comparisons, not post-hoc rescue." Both numbers reproduce exactly (panel C macro +0.0815, range +0.0627 to +0.0995; panel D macro -0.0103, range -0.0427 to +0.0087). They also have a dedicated headline-table block and are panels C/D of the new hero.

4. Binding-constraint statement — VERIFIED. v2 confidence line: "Confidence: MODERATE — H1 in its registered form is decisively falsified (n_pairs=3516, p<0.001 on three sources, within-source seed std 0.001-0.004), but the binding constraints on the broader 'matched-scaffold-gating' interpretation are: (a) the reframing is post-hoc, prompted by H1's falsification rather than pre-registered; (b) the H4 wrong-vs-correct control is at one source only (librarian); (c) police_officer dominates both macro means (in opposite directions), so the story rests on the three non-anomalous sources rather than a clean four-source pattern; (d) all eval is in-distribution ARC-Challenge." The "3 seeds bound between-seed variance" framing is dropped from the prose; standing caveats explicitly notes within-source seed std is 0.001-0.004 and the binding uncertainty is between-source heterogeneity. (Minor note: my independent computation on panel B gives within-so

Independent Review: #186 Train-time persona-CoT × ARC-C capability leakage

Verdict: PASS Reproducibility: COMPLETE Structure: COMPLETE (verify_clean_result.py PASS — 9 checks pass, 1 expected WARN on derived-numeric matching that the validator can't recompute)

Template compliance

• TL;DR has 4 H3 subsections (Background / Methodology / Results / Next steps) in order — verified
• Hero figure inside ### Results with commit-pinned raw.githubusercontent.com/...b4bc61f/... URL — verified
• Results subsection ends with **Main takeaways:** (7 bullets, each bolds a load-bearing claim with numbers, no *Updates me:* label, mostly plain-prose continuations) followed by single **Confidence: MODERATE** — <one sentence> line — verified
• Title ends with (MODERATE confidence) matching the Confidence line — verified
• Detailed report has all required sections including "Why this experiment / why these parameters / alternatives considered" prose at the top of Setup, plus inline "Standing caveats" bullet block after Headline numbers — verified
• Stats framing: no effect-size / named-test / value ± err language; only p-values, sample sizes, and chart error bars — verified

Headline-number spot-checks (recomputed independently from per-cell result.json and aggregate.json)

Every headline number reproduces from the raw JSONs. The one cosmetic inconsistency I noticed: librarian panel-B is reported as +0.232 in the cosine-takeaway prose but +0.233 in the H4 table; both are valid roundings of +0.2325. Not worth fixing.

Confidence calibration check (MODERATE)

The body lists four binding constraints on MODERATE: (a) post-hoc reframing of matched-scaffold mechanism, (b) H4 control at one source only, (c) police_officer dominates both macros in opposite directions, (d) ARC-C is in-distribution. All four are present in Standing caveats and surfaced in the Confidence line. None of the constraints are softened or buried. The H1 falsification itself (n_pairs=3516, p<0.001 on 3/4 sources, within-source seed std 0.0005-0.0035) is a real piece of HIGH-confidence evidence; combined

Reviewer PASS — clean-result ready for your promotion

The interpretation went analyzer v1 → critic v1 (REVISE) → analyzer v2 → critic v2 (PASS) → reviewer (PASS, all numbers independently reproduced from raw data).

Issue #186 is currently the clean-result draft itself (label clean-results:draft). When you're satisfied, promote it:

/clean-results promote 186

After promotion, re-invoke /issue 186 to auto-complete (Step 10): apply status:done-experiment, move to project board "Done (experiment)" column, post the final epm:done comment, spawn follow-up-proposer, prompt for pod termination, and prompt for worktree merge.

Quick recap of the clean-result

• Title: "Train-time persona-CoT does NOT reduce bystander capability leakage; matched-scaffold gating is the dominant driver (MODERATE confidence)" (current title)
• H1 falsified (macro Δ = +0.024 with police_officer, +0.003 without; predicted negative)
• H4 control isolates wrong-answer SFT under matched scaffold as the leakage mechanism (librarian persona-CoT-correct +0.012 vs persona-CoT-wrong +0.233 — ~19× gap)
• Police_officer is structurally anomalous (35.7% no-CoT-eval collapse, 82.4% empty-tag recovery, balanced training-data letter distribution rules out the letter-A artifact)
• Pre-registered controls confirm scaffold-gating (generic-CoT × generic-CoT mean +0.082; empty-tag × persona-CoT-train mean −0.010)
• Confidence: MODERATE — binding constraints are post-hoc reframing of mechanism, one-source H4, police_officer dominance, and in-distribution ARC-C eval

Clean-result lint — FAIL

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 3 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  section missing (legacy body — grandfathered)
AI TL;DR paragraph               ✓ PASS  545 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  3 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
Results block shape              ✗ FAIL  missing `**Main takeaways:**` bolded label inside ### Results
Methodology bullets              ✓ PASS  pre-cutoff (created 2026-05-02, cutoff 2026-05-15)
Background context               ✓ PASS  Background has 383 words
Acronyms defined                 ✓ PASS  all defined: ['H1', 'H3']
Background motivation            ✓ PASS  references prior issue(s): [75, 80, 138, 150, 182]
Dataset example                  ✓ PASS  dataset example + full-data link present
Human summary                    ✗ FAIL  ## Human summary section missing (must appear at top of Detailed report)
Sample outputs                   ✗ FAIL  ## Sample outputs section missing
Numbers match JSON               ! WARN  53 numeric claims not found in referenced JSON (e.g. 0.001, 0.002, 0.003, 0.006, 0.007)
Reproducibility card             ✗ FAIL  ## Setup & hyper-parameters section missing
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: FAIL — fix the failing checks before posting.

Fix the issues and edit the body; the workflow re-runs.

Clean-result lint — FAIL

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 3 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  section missing (legacy body — grandfathered)
AI TL;DR paragraph               ✓ PASS  528 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  3 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
Results block shape              ✗ FAIL  missing `**Main takeaways:**` bolded label inside ### Results
Methodology bullets              ✓ PASS  pre-cutoff (created 2026-05-02, cutoff 2026-05-15)
Background context               ✓ PASS  Background has 382 words
Acronyms defined                 ✓ PASS  no project-internal acronyms used
Background motivation            ✓ PASS  references prior issue(s): [75, 80, 138, 150, 182]
Bare #N references               ✓ PASS  skipped (v1 / legacy body — markdown-link rule applies to v2 only)
Dataset example                  ✓ PASS  dataset example + full-data link present
Human summary                    ✗ FAIL  ## Human summary section missing (must appear at top of Detailed report)
Sample outputs                   ✗ FAIL  ## Sample outputs section missing
Inline samples per Result        ✓ PASS  3 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  53 numeric claims not found in referenced JSON (e.g. 0.001, 0.002, 0.003, 0.006, 0.007)
Reproducibility card             ✗ FAIL  ## Setup & hyper-parameters section missing
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: FAIL — fix the failing checks before posting.

Fix the issues and edit the body; the workflow re-runs.

Clean-result lint — FAIL

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 3 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  528 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  3 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
Results block shape              ✗ FAIL  missing `**Main takeaways:**` bolded label inside ### Results
Methodology bullets              ✓ PASS  non-strict (grandfathered)
Background context               ✓ PASS  Background has 382 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  skipped (v1 / legacy body — markdown-link rule applies to v2 only)
Dataset example                  ✓ PASS  non-strict (grandfathered)
Human summary                    ! WARN  ## Human summary missing (grandfathered: issue >7 days old or already-promoted)
Sample outputs                   ! WARN  ## Sample outputs missing (grandfathered)
Inline samples per Result        ✓ PASS  3 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  53 numeric claims not found in referenced JSON (e.g. 0.001, 0.002, 0.003, 0.006, 0.007)
Reproducibility card             ✗ FAIL  ## Setup & hyper-parameters section missing
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: FAIL — fix the failing checks before posting.

Fix the issues and edit the body; the workflow re-runs.

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 3 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  591 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  3 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 382 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  3 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  53 numeric claims not found in referenced JSON (e.g. 0.001, 0.002, 0.003, 0.006, 0.007)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 3 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  451 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  3 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 382 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  3 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  53 numeric claims not found in referenced JSON (e.g. 0.001, 0.002, 0.003, 0.006, 0.007)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 2 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  392 words, 4 bullets (LW-style)
Hero figure                      ✓ PASS  2 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 365 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  2 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  46 numeric claims not found in referenced JSON (e.g. 0.001, 0.003, 0.006, 0.009, 0.026)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 2 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  392 words, 4 bullets (LW-style)
Hero figure                      ✓ PASS  3 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 365 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  2 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  54 numeric claims not found in referenced JSON (e.g. 0.001, 0.003, 0.006, 0.009, 0.022)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 5 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (pre-v4, content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  927 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  5 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 542 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  5 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  68 numeric claims not found in referenced JSON (e.g. 0.001, 0.003, 0.006, 0.009, 0.011)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 4 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (pre-v4, content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  799 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  4 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 341 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  4 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  43 numeric claims not found in referenced JSON (e.g. 0.001, 0.003, 0.009, 0.011, 0.012)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 4 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (pre-v4, content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  799 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  2 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 341 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  4 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  41 numeric claims not found in referenced JSON (e.g. 0.001, 0.003, 0.007, 0.009, 0.011)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 4 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (pre-v4, content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  846 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  2 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 343 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  4 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  41 numeric claims not found in referenced JSON (e.g. 0.001, 0.003, 0.007, 0.009, 0.011)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 4 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (pre-v4, content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  843 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  1 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 343 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  4 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  43 numeric claims not found in referenced JSON (e.g. 0.001, 0.003, 0.009, 0.011, 0.012)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 4 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (pre-v4, content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  843 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  1 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 343 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  4 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  43 numeric claims not found in referenced JSON (e.g. 0.001, 0.003, 0.009, 0.011, 0.012)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result lint — PASS

Check                            Status  Detail
---------------------------------------------------------------------------------------------------
AI Summary structure             ✓ PASS  v2: Background + Methodology + 4 Result section(s) (no Next steps — optional)
Human TL;DR                      ✓ PASS  H2 present (pre-v4, content user-owned, not validated)
AI TL;DR paragraph               ✓ PASS  843 words, 6 bullets (LW-style)
Hero figure                      ✓ PASS  1 figure(s) present; primary commit-pinned
Results figure captions          ✓ PASS  every Results figure has a caption paragraph
check_results_block              ✓ PASS  skipped (v2 template — section retired)
check_methodology_bullets        ✓ PASS  skipped (v2 template — section retired)
Background context               ✓ PASS  Background has 343 words
Acronyms defined                 ✓ PASS  non-strict (grandfathered)
Background motivation            ✓ PASS  non-strict (grandfathered)
Bare #N references               ✓ PASS  non-strict (grandfathered)
Dataset example                  ✓ PASS  non-strict (grandfathered)
check_human_summary              ✓ PASS  skipped (v2 template — section retired)
check_sample_outputs             ✓ PASS  skipped (v2 template — section retired)
Inline samples per Result        ✓ PASS  4 Result section(s), each with >=2 fenced sample blocks
Numbers match JSON               ! WARN  43 numeric claims not found in referenced JSON (e.g. 0.001, 0.003, 0.009, 0.011, 0.012)
check_reproducibility            ✓ PASS  skipped (v2 template — section retired)
Confidence phrasebook            ✓ PASS  no ad-hoc hedges detected
Stats framing (p-values only)    ✓ PASS  no effect-size / named-test / credence-interval language
Collapsible sections             ✓ PASS  all H2/H3 body sections wrapped (heading-as-toggle convention)
Title confidence marker          ! WARN  title says (MODERATE confidence) but Results has no Confidence line to match

Result: PASS (WARNs acknowledged).

Clean-result ced734ae promoted; critic pair passed at round 2 (both claude+codex returned 'pass' after round-1 fixes: Codex numerical correction in Result 1 TL;DR bullet + Claude structural fixes for cherry-pick labelling, raw-completions disclosure, Next-steps re-run bullet, and bottom <details id='repro'> block). Awaiting owner-driven promote.