Run 8aba634c

kind=qa

# Review note — experiment #186

**Verdict:** Substantive, well-instrumented result with honest constraint reporting. Two framing tweaks and two evidence gaps to close before promoting; structure also drifts from the clean-result guideline.

## What's load-bearing and well-supported

- **Headline (persona style ≫ loss-token count):** the +0.159 vs +0.012 contrast (persona-cot vs garbage; neutral-en vs garbage) is an order-of-magnitude separation, Holm-corrected at α=0.01 with n=3,516. The "persona style dominates" framing in the title is calibrated to the data.
- **Scaffold-matching gate (Result 1):** the monotonic ramp on matched-eval (0.00 → +0.10 → +0.22 source; 0.00 → +0.08 → +0.16 bystander) is clean and the no-scaffold-eval null (Δ +0.024, +0.003 sans police_officer) is the right counterfactual.
- **Constraint honesty:** the five binding constraints in the TL;DR/Confidence block are appropriately scoped — police_officer outlier, narrow source slice, defense replication, rig-drift carry-over, in-distribution ARC-C.

## Framing tweaks before promote

1. **"Wrong-answer behavior only fires when scaffolds match" overstates the gate.** TL;DR uses "only fires" but body says "attenuates to near-zero" — those aren't the same. If mismatched cells are at floor, say "fires only when matched; mismatch is at floor (Δ +0.024)." If they're attenuated but non-zero, drop "only fires." Pick one and make TL;DR/Summary match.
2. **"Defense lever" framing for the −0.013 contradicting-rationale effect.** Tight CI, but the absolute effect is ~12× smaller than the +0.159 main leakage and lives at the same scale as the neutral-vs-garbage wedge (+0.012). Calibrate the phrasing — "small but statistically robust counter-direction effect, worth a follow-up" lands better than "defense lever," which sets reader expectations that this can absorb the +0.159.

## Evidence gaps / missing artifacts

- **Primary plot for Figure 1.** Body references "Figure 1" but I can't see whether an `html_artifact` or figure is attached at the entity. The clean-result guidelines want a primary plot up top with plain-English labels and SVG `<title>` tooltips. Confirm one is rendered, not just referenced.
- **Police_officer dual-handling.** Source-loss is reported both with and without police_officer (+0.193 without). The bystander numbers should get the same treatment (with-and-without) anywhere police_officer materially moves a headline — readers shouldn't have to recompute it themselves.
- **Mode-collapse follow-up evals.** TL;DR says "follow-up evals queued" for the two seed-replicated collapses (comedian under scrambled-eng+no-cot-eval; police_officer under persona-cot+no-cot-eval). If those are required to support the speculative mechanisms in the takeaways, link the tracking experiments here so reviewers can see whether the load-bearing claim is "established" vs "speculative-pending-followup."
- **Carry-over rig-drift sensitivity.** Constraint (d) flags 27 cells re-evaluated under the length-matched-eval commit. A small spot-check (one persona × one seed re-run side-by-side) would convert "plausibly small but non-zero" into a number, which would let you keep MODERATE confidence with a stronger leg.
- **Follow-up references #344 and #349.** Those look like EPS GitHub issues, not Sagan experiment numbers. Per the tenant-agnostic guardrail this experiment lives in Sagan; if those followups are tracked as Sagan experiments somewhere, link those numbers; if they're EPS-only, note that explicitly so the reviewer knows where to find them.

## Structure note

Body is TL;DR → Summary → Details (Setup). `docs/clean-result-guidelines.md` wants TL;DR → primary plot → Experimental design dropdown. The current Summary section embeds methodology + results inline. Consider collapsing the methodology / dataset prose into the "Experimental design" dropdown so the visible flow is TL;DR → plot → results, with design hidden behind the toggle.

## Suggested next concrete action

Pick one…

success

# Review note — experiment #186

**Verdict:** Substantive, well-instrumented result with honest constraint reporting. Two framing tweaks and two evidence gaps to close before promoting; structure also drifts from the clean-result guideline.

## What's load-bearing and well-supported

- **Headline (persona style ≫ loss-token count):** the +0.159 vs +0.012 contrast (persona-cot vs garbage; neutral-en vs garbage) is an order-of-magnitude separation, Holm-corrected at α=0.01 with n=3,516. The "persona style dominates" framing in the title is calibrated to the data.
- **Scaffold-matching gate (Result 1):** the monotonic ramp on matched-eval (0.00 → +0.10 → +0.22 source; 0.00 → +0.08 → +0.16 bystander) is clean and the no-scaffold-eval null (Δ +0.024, +0.003 sans police_officer) is the right counterfactual.
- **Constraint honesty:** the five binding constraints in the TL;DR/Confidence block are appropriately scoped — police_officer outlier, narrow source slice, defense replication, rig-drift…