Run 93ecaeff

kind=qa

# Review note for experiment #295

## Verdict

The experiment is reportable but **not promotable to a clean result in its current form**. The author's own LOW-confidence flag is the correct call. The headline framing ("training-data shape doesn't amplify uptake; long system prompt leaks to bystanders") is plausible but rests on a baseline that doesn't reproduce the parent anchor, so every cross-condition comparison is suspect until that's resolved.

## What the evidence supports

1. **Result 3 (system-prompt-length leaks to bystanders) is the strongest signal.** With multiple bystanders (software_engineer 0.27, medical_doctor 0.25) exceeding the source (librarian 0.16) under `sl_long`, the *direction* is robust to anchor calibration — the comparison is within-condition across personas, not across conditions. This is the finding worth carrying forward.
2. **Result 1 (multi-turn collapses source-rate) is directionally well-supported** at single seed: 0.34 → 0.17 → 0.00 is monotone over a 16× span. The mechanism story (more loss tokens dilute marker gradient) is consistent with Result 2.
3. **Result 2 (`lc_long` = 0.00 even at 4× eval budget per #297) rules out truncation** as the explanation for the long-completion null. That's a clean follow-up and tightens the gradient-dilution reading.

## Blockers / what makes this not promotable

1. **The matched-shape controls don't match the parent.** `lc_medium` 0.31 and `sl_medium` 0.28 vs parent #271's librarian anchor of 0.67 is a 2× gap. Either:
   - the parent anchor was noisy and #295's anchors are the true rate (in which case #271's headline needs revisiting), or
   - something in #295's data pipeline / max-seq-length deviations / hot-fix recovery path changed effective training, and the "no amplification" claim is partly an artifact of a weaker baseline.

   The experiment doesn't resolve which. **Required before clean-result promotion: explain or close this gap.**
2. **Leg-1 vs Leg-2 attribution for Result 3 is unclear in the writeup.** The bystander-leakage numbers (0.12–0.27 across 11 personas) appear to come from the Leg-2 train-matched eval, but the TL;DR doesn't label them as such. The status of Leg-2 ("informational") should be reconciled with the role it plays in the primary takeaway.
3. **Single seed, n=20 questions, no slope intervals reported in the body.** The methodology setup mentions Bonferroni-corrected 98.33% intervals across 3 sub-claim tests, but the results section reports point estimates only. Reproduce or surface the intervals so the reader can tell which deltas survive.
4. **The body is cut off mid-sentence** in the "alternatives considered" paragraph (`"v1's library-domain filler in th…"`). Either the field was truncated or the draft was committed incomplete — needs to be finished or marked.

## Missing artifacts

- No `figures` referenced in the body resolve to a primary plot embedded inline. The TL;DR cites "Figure 1" three times but the experiment record doesn't carry a `html_artifact`/figure attachment visible in scope context. Confirm Figure 1 is attached to this experiment (not just sitting in `eval_results/issue260/` on disk).
- WandB run IDs are not persisted in `run_result.json` per the author's own note. For a `reviewing`-status experiment that's an acceptable gap, but flag it so the next reader doesn't go hunting.

## Concrete next review action

Before this can move from `reviewing` back to `done` as a clean result, the author should either:

- **(a)** run a single-cell replication of the parent shape (matched seed, matched data recipe modulo the documented max-seq-length change) and either reproduce the 0.67 anchor or demonstrate that #271's anchor was the outlier. This is the smallest experiment that unblocks all three results, and
- **(b)** label the bystander-leakage numbers as Leg-2 and add at-least slope CIs (or note their absence prominently in the TL;DR).

If (a) reproduces ~0.31, Result 3 graduates to a real finding and Results 1/2 bec…

success

# Review note for experiment #295

## Verdict

The experiment is reportable but **not promotable to a clean result in its current form**. The author's own LOW-confidence flag is the correct call. The headline framing ("training-data shape doesn't amplify uptake; long system prompt leaks to bystanders") is plausible but rests on a baseline that doesn't reproduce the parent anchor, so every cross-condition comparison is suspect until that's resolved.

## What the evidence supports

1. **Result 3 (system-prompt-length leaks to bystanders) is the strongest signal.** With multiple bystanders (software_engineer 0.27, medical_doctor 0.25) exceeding the source (librarian 0.16) under `sl_long`, the *direction* is robust to anchor calibration — the comparison is within-condition across personas, not across conditions. This is the finding worth carrying forward.
2. **Result 1 (multi-turn collapses source-rate) is directionally well-supported** at single seed: 0.34 → 0.17 → 0.00 is monotone over a…