All 11 required headings present in order, runpod-spec is valid JSON with 4 pods matching the plan's narrative, cost arithmetic checks out at $194 compute → ~$200 with storage. Emitting the final plan:

---

Goal

Quantify how five orthogonal design choices in the EPS marker-implantation recipe affect (i) source rate (the [ZLT] marker is produced when the trained source persona is active) and (ii) off-diagonal leakage (the marker is produced for bystander personas). Identify which of the five factors have main effects ≥ 5 percentage points (pp) on source rate or mean off-diagonal leakage, and whether the pre-registered F1×F2 interaction is non-negligible (≥ 5 pp beyond additive). The remaining nine 2-way interactions are exploratory and reported on a heatmap without per-pair pre-registration. Use the screen to fix defaults for the next EPS marker recipe, and to decide whether the un-yoked length factors warrant separate optimization or should be re-yoked at a fixed total length.

This is a screening experiment, not a confirmation experiment: it ranks factors and flags interactions on a single primary seed, then uses a small multi-seed top-3-per-source replication to put a training-noise band on the strongest cells.

Hypothesis

• H1 (primary). Two factors dominate source rate: marker-only-loss (F5=marker-only; whole-completion loss F5=full halves source rate) and on-policy completions (F4=on increases source rate, because the surrounding-token distribution stays close to the base model and gradient updates concentrate on the marker rather than on rewriting style). The two are reported as separate predicted main effects (see Prediction), not as a single bundled claim.
• H2 (leakage). Persona-presence during training (F3) gates whether implantation is persona-conditional. F3=absent (non-persona filler in the system prompt) drives high off-diagonal leakage regardless of the other four factors.
• H3 (length, decomposed). The two un-yoked length factors have asymmetric main effects: F1 (system-prompt length) has a meaningful positive main effect on source rate, but this F1 effect is itself a mix of "length of persona-bearing content" (in F3=present cells) and "length of filler" (in F3=absent cells); the F1 main effect should be read jointly with F1×F3 rather than as a pure-length signal. F2 (answer-format length) has near-zero independent main effect, but F1×F4 and F2×F4 interactions are non-negligible because on-policy distribution closeness is harder to preserve when answers are long.
• H4 (un-yoking confound). Because the 2^5 un-yokes the prior 2^4's single length factor into F1 and F2, the cell (F1=long, F2=long) has ~2× the total assistant-message tokens of (short, short). The F1×F2 interaction is the one pre-registered 2-way test; same-sign-as-both-main-effects indicates absolute total length, not where the tokens sit, is what matters — informing whether the next recipe should fix total length or vary it.

Prediction

Pre-registered point predictions for source rate (SR) and mean off-diagonal leakage rate (LR) per cell, evaluated on the 24-persona × 20-question × 5-completion eval panel ([ZLT] case-insensitive substring detection anywhere in the completion, fuzzy Levenshtein ≤ 1 reported alongside as a secondary). Predicted intervals below are informal magnitude expectations (roughly 1 SD of analyst-prior), not formal CIs; the clean result reports observed effects against them rather than as null-hypothesis tests:

• Baseline cell (F1=short, F2=short, F3=absent, F4=off, F5=full): SR ≈ 0.05, LR ≈ 0.05.
• Best-expected cell (F1=long, F2=short, F3=present, F4=on, F5=marker-only): SR ≥ 0.40, LR ≤ 0.10 (anchored on prior EPS results on the librarian source persona).
• F5=full (whole-completion loss) main effect on SR: −0.20 ± 0.10.
• F4=on (on-policy) main effect on SR: +0.10 ± 0.10.
• F3=absent main effect on LR: +0.15 ± 0.10 (averaged across the 16 F3=absent cells).
• F1=long (long system prompt) main effect on SR: +0.10 ± 0.05 (read jointly with F1×F3; see H3).
• F2=long (long answers) main effect on SR: ≤ 0.05 in magnitude.
• F1×F2 interaction on SR (pre-registered): ≤ 0.05 if effects are about token placement; ≥ 0.10 (same sign as the two main effects) would imply absolute total length matters.

A clean result is a figure that either confirms this ordering (marker-only and on-policy dominate SR; persona-presence gates LR; length factors are minor; F1×F2 is small) or clearly contradicts it. The clean result is informative either way; the screen is designed not to need a "winning" cell.

Kill Criterion

Run a fractional-factorial smoke (2^(5−2) = 8 cells, resolution III, on the librarian source persona only) before launching the full 96-cell run. The smoke is used only to check the kill criteria below — it is not used to pre-rank factors or to choose the top-3-per-source cells for multi-seed replication (which use the full 96-cell results on the primary seed). Abandon the screen and replan if any of the following hold after the smoke:

1. Compute infeasibility. Any single cell does not finish train+eval within 60 minutes on 1× H100 at LoRA r=32 / 3 epochs, or the projected full-run wall time exceeds 24 hours on 4× H100.
2. Baseline saturation. The smoke's baseline cell (F1=F2=short, F3=absent, F4=off, F5=full) shows SR ≥ 0.40 on the librarian source row. That would mean the recipe is already saturated and the screen has no headroom; redesign the marker (longer / lower-frequency tail) before running.
3. Universal failure. All 8 smoke cells show SR < 0.05 on the librarian source row. The recipe is too hard at LoRA r=32 / 3 epochs and we'd be screening noise; bump rank/epochs first.
4. Eval invalidity / contamination. Pre-screen of base Qwen2.5-7B-Instruct (no LoRA, no training) on the full 24-persona × 20-question × 5-completion eval panel shows [ZLT] spontaneous emission ≥ 2% on any source row, or ≥ 1% mean across the 21 bystander rows.

The "no main effect ≥ 5pp in the smoke" outcome is not a kill — it is information that factors are tied or absent and we still proceed (with reduced expected information gain noted in the clean result).

Smoke-cell checks (cell-level SR, smoke-cell wall time, base-model emission rates) are logged as agent_run_event rows so the runner can fail-fast before the remaining 88 cells launch.

Experimental Setup

Base model. Qwen/Qwen2.5-7B-Instruct (consistent with EPS #337, #295, #340, prior 2^4 factorial). LoRA r=32, α=64, lr=1e-5, 3 epochs, AdamW. No layer freezing beyond what LoRA already implies.

Training data per cell. marker_asst_excluded_medium dataset, 200 positive (source persona) examples + 400 negative (bystander) examples per source persona. Marker is [ZLT] appended end-of-completion on positive examples. The five factors are applied as follows:

Factor	Level 0 (baseline)	Level 1 (treatment)
F1: system-prompt length	short prompt (~6 tokens) — content set by F3	long prompt (~1000 tokens) — content set by F3
F2: answer-format length	short completions (~50 tokens)	long completions (~1050 tokens)
F3: persona-presence	non-persona filler in system prompt (cloud-formation sl_long from EPS #295), token-matched to F1's length	persona-rich prompt (e.g. "You are a librarian who…", borrowed from EPS #339 +persona), token-matched to F1's length
F4: on-policy completions	off-policy (Claude-generated, current recipe)	on-policy (vLLM-sampled from base Qwen2.5-7B-Instruct under the same cell's (F1, F3) system prompt)
F5: loss mask	whole-completion loss (standard CE on all completion tokens)	marker-only loss (CE masked to [ZLT] sub-tokens + EOS)

Note on F1/F2 un-yoking. Because F1 and F2 vary independently, total assistant-message length spans ~56 tokens (F1=short, F2=short) to ~2050 tokens (F1=long, F2=long). H4 and the prediction section pre-register the F1×F2 test that distinguishes a token-placement story from a total-length story; total-token-count is also reported as a covariate scatter (Artifacts, figure iv).

Note on F1/F3 confound. F1 controls system-prompt length; F3 controls system-prompt content. At F3=absent, F1 varies "length of filler"; at F3=present, F1 varies "length of persona-bearing content." The 2×2 of F1×F3 is fully crossed so this is identifiable from the design, but the F1 marginal main effect should be read jointly with F1×F3. The clean result reports both the F1 main effect and the F1×F3 interaction; H3 is worded to match.

Note on F4 on-policy generation. "On-policy" means vLLM-sampled from Qwen2.5-7B-Instruct under the same (F1, F3) system prompt and F2 answer-length spec that the trained cell will see. There are 4 distinct (F1, F3) system prompts per source × 2 F2 levels = 8 distinct on-policy generation regimes per source (24 total across 3 sources). Each on-policy regime is generated once per pod and reused across the cells of that pod's slab that share the same (F1, F3, F2) triple. The on-policy data-gen GPU-h budget below reflects 8 regimes per pod, not 1.

Note on cross-experiment polarity vs the prior 2^4 (exp 077ae4c7). Two factors have flipped baseline conventions relative to the parent 2^4: (a) the prior 2^4 set marker-only loss as the D-axis baseline (D0) after EPS #295's null result; this 2^5 plan sets whole-completion loss as F5 baseline (level 0). (b) The prior 2^4 set persona-presence as B0 (baseline) and non-persona filler as B1; this 2^5 plan sets non-persona filler as F3 level 0. The internal validity of the screen is unaffected — all 32 factor combinations are still run — but a reader cross-referencing main-effect signs between the two experiment reports must re-sign F5 and F3 to align. This is called out here, in the Risks section, and in the clean result.

Source persona panel. 3 personas — librarian, surgeon, programmer — chosen to span topic breadth and answer length and to anchor against prior EPS work (#271, #232, #295). Each cell of the 2^5 factorial is trained separately per source, yielding 3 × 32 = 96 training runs for the primary screen.

Eval panel. 24-persona × 20-question × 5-completion vLLM batched, max_new_tokens=2048 (per the #260 / #297 truncation lesson). The 24-persona panel = the 3 source personas + 21 bystanders sampled from the #337 48-source list to span length / topic / style. The 21 bystanders are listed in metrics.json with a per-persona flag for whether that persona also appears in the 400 negative training examples for the current cell's source; the clean result reports "training-set bystander LR" and "held-out bystander LR" separately so both interpretations are visible. Per cell: source rate (the diagonal entry for that cell's source persona), mean off-diagonal rate over the 21 bystanders, per-bystander rates. Primary detection is [ZLT] case-insensitive substring anywhere in the completion; secondary is fuzzy Levenshtein ≤ 1. The primary↔secondary switching rule fires only if (a) the gap exceeds the cell-level bootstrap CI half-width and (b) the gap has the same direction in ≥ 3 cells of the affected slab, to avoid switching on single-cell noise.

Statistical analysis. Bootstrap 95% CIs use a clustered resampling scheme: for source rate, resample at the (eval question) level (20 questions × 5 completions = 100 per cell on the source row); for off-diagonal leakage rate, resample at the (bystander persona) level (n=21) and report the persona-clustered CI. A naive flat bootstrap over 2,400 eval completions would under-estimate LR uncertainty because the bystander panel is the unit over which the mean is averaged. Main effects and 2-way interactions are computed as half-difference contrasts on the 2^5 grid per source; the persona-level sign-agreement matrix is reported on the per-source main effects, not on the cross-source mean. Multiplicity: F1×F2 is the only pre-registered interaction; the remaining 9 pairwise interactions are presented on the heatmap as exploratory and not used as decision criteria.

Seeds. Seed 42 for the primary 96-cell factorial. Seeds 137 and 256 on the top-3 cells per source persona by source-rate effect at seed 42 (9 cells × 2 seeds = 18 multi-seed runs) to put a training-noise CI on the strongest cells. The selection is post-hoc on seed-42 results, so a winner's-curse correction is reported: median seed-42 SR alongside the 3-seed mean for those 9 cells. A follow-up experiment is queued to replicate any flagged interaction with full multi-seed coverage on every cell.

Persona stability check. For each of the five factors, compute the main effect on SR separately within each source persona. Report whether the sign agrees across the 3 sources. If a factor's main effect flips sign on ≥ 2 of 3 sources, that factor's screen-level main effect is flagged as unstable in the clean result rather than treated as resolved (downgrade rather than block). If all five factors are unstable, framing is wrong and a follow-up at the persona-class level is queued. The check is intentionally low-resolution at n=3 sources; this is acknowledged in the clean result.

Cell partitioning across pods. Cells are partitioned by (source, F4) so each of the 4 pods owns a contiguous slab. F4=on cells within a pod share their 8 (F1, F3, F2)-regime on-policy data-gen passes. The dashboard collects runs back into the single agent_run.

Compute and Hardware

Sequential GPU-hour budget:

Phase	Estimate
Base-model contamination pre-screen (kill criterion #4, 24×20×5 eval on base)	~0.5 GPU-h
Smoke run (8 cells × ~25 min train + ~10 min eval, on librarian only)	~5 GPU-h
On-policy data gen for F4=on cells (8 regimes × 3 sources = 24 distinct regimes, amortized within pods)	~6 GPU-h
Training (96 cells × ~25 min)	~40 GPU-h
Eval (96 cells × ~10 min)	~16 GPU-h
Multi-seed top-3 per source (9 cells × 2 seeds × ~25 min train + ~10 min eval)	~5 GPU-h
Total sequential	~73 GPU-h

Parallelization. 4 pods × 1 H100 each (one pod owns each (source, F4) slab; the 4th pod absorbs the smoke + base-model contamination pre-screen first, then takes the overflow + aggregation role). Wall time on the critical path is bounded by the largest slab plus the longest single cell, ~16–18 hours.

Why 4× single-GPU pods, not one 4-GPU pod. Each cell is a single-GPU LoRA fine-tune at r=32 on a 7B base; multi-GPU would add inter-GPU comm overhead with no per-cell speedup, and the work is genuinely partitioned across (source, F4) slabs with no shared memory between slabs. Each pod also runs its own on-policy vLLM data-gen step which is single-GPU-bound. The runner will treat partial dispatch (< 4 pods) as a hard failure and stop survivors per the dispatcher's documented behavior — this is acceptable because the screen is uninterpretable without all 96 cells, and a re-dispatch with fewer pods on different capacity is the right recovery.

Pod hardware per pod. H100 80GB SXM, SECURE cloud, 100 GB container disk (model weights + LoRA adapters at ~30 GB across the slab), 100 GB persistent volume (datasets, on-policy generations, eval JSONs).

Cost estimate (auditable). SECURE H100 80GB SXM at $2.69/GPU-hr (May 2026 reference; rates may drift). Formula: 4 pods × 1 GPU × 18 h × $2.69/GPU-hr = $194 compute + storage at $0.10/GB-month × 100 GB × 4 pods × (18 h / 720 h-per-month) ≈ $1 storage = ~$200 total (rounded to two significant figures: ~$200). The wall-time-budgeted figure ($194) is the dominant term; the GPU-hour-budgeted figure (73 GPU-h × $2.69 = $196) is the alternative slice and the two agree to within rounding. If any pod hits 20 hours wall time the runner pauses and re-checks with the user.

computeSize: large, runpodAccount: team — matches the scoped record.

Artifacts

Per pod and per cell, written to the experiment's existing entity (no new EPS-shaped tables):

• metrics.json per cell with: cell key (F1..F5 bits + source persona), seed, train wall time, eval wall time, source SR (substring + fuzzy), persona-clustered bootstrap CI on SR (question-resampled) and on LR (persona-resampled), mean off-diagonal LR (substring + fuzzy), per-bystander LR vector (length 21) with a per-bystander flag for "appeared in the 400 negative training examples for this source", train loss curve, exact tokenization of [ZLT] under Qwen tokenizer.
• pre_screen.json — base-model [ZLT] emission rates on the full 24×20×5 eval panel, per row (kill criterion #4 audit trail).
• smoke.json — the 8-cell smoke results with per-cell SR, wall time, and kill-criterion verdict; an explicit field records "not used for factor pre-ranking."
• LoRA adapters for the 96 + 18 = 114 trained runs, each ~250 MB (~30 GB total) on the persistent volume. Adapters for the top-3-per-source × 2-seed cells are uploaded to HuggingFace Hub via the standard uploader pipeline; the remainder stay on the volume unless a follow-up needs them.
• figures/ for the clean result: (i) main-effects bar chart with persona-clustered 95% bootstrap CIs for SR and LR, (ii) 2-way interaction heatmap for all 10 factor pairs with F1×F2 explicitly labelled as pre-registered and the other 9 labelled exploratory, (iii) per-source-persona sign-agreement matrix on main effects, (iv) total-token-count vs SR scatter (covariate analysis for H4), all as PNG + SVG.
• experiments.body HTML write-up following docs/clean-result-guidelines.md: TL;DR → primary plot (main-effects bar chart) → Experimental design dropdown. Title is the experiment title as scoped.

All artifact paths are POSTed back via SAGAN_PROGRESS_URL as progress messages so the dashboard can show partial state during the run.

Verification

• Pre-flight (before any training). Run the kill-criterion #4 base-model contamination pre-screen on one pod (~0.5 GPU-h) and POST a progress update with the per-row [ZLT] rates. If any threshold trips, halt.
• Smoke gate. Run the 8-cell librarian-only smoke on one pod and check kill criteria #1–#3 against the smoke.json output. POST the smoke verdict as a progress message. If any kill criterion trips, the runner halts the other pods.
• Per-cell sanity. Every cell logs train wall time and final train loss; cells with wall time > 60 min or train loss diverging by > 2σ from the per-source median are flagged in metrics.json for manual review.
• Cross-cell sanity. After all 96 cells finish, compute the additive main-effect model on SR and LR; if residuals are bimodal or per-source main effects disagree on sign for ≥ 4 of 5 factors, flag the screen as unstable and report instead of "winning cell" findings.
• Reproducibility. All seeds, dataset hashes, LoRA hyperparameters, vLLM sampling configs, and the exact [ZLT] tokenization are recorded in metrics.json per cell so a follow-up replication has a complete config dump.
• Pre-registered analysis. This Hypothesis + Prediction section is the pre-registration; the clean result will report what was predicted before vs. what was observed, marking confirmed/contradicted predictions explicitly and labelling the F1×F2 interaction as the only pre-registered 2-way test.

Risks and Red Team

• Un-yoked length factors create an absolute-length confound. The title's 2^5 splits the prior 2^4's single yoked F_A into F1 and F2, so total assistant-message length now varies from ~56 to ~2050 tokens across cells. Mitigation: H4 pre-registers the F1×F2 interaction test as the discriminator, and the clean result reports total-token-count as an analysis covariate (figure iv). If F1×F2 dominates the additive main effects, the recipe should re-yoke length at a fixed total.

• F1 main effect is jointly length-and-content. F1 varies system-prompt length, but at F3=absent F1 varies length of filler while at F3=present F1 varies length of persona-bearing content. Mitigation: H3 is worded to acknowledge this; the clean result reports both the F1 main effect and the F1×F3 interaction so the reader can attribute the F1 signal to length vs. content.

• F4 on-policy is generator-identity confounded. Operationalising F4 as Claude-generated vs vLLM-sampled-from-Qwen conflates "distribution closeness" with "specific generator identity" (and incidental Claude-style content choices). Mitigation: the screen's job is to flag whether F4 matters; a follow-up with a third on-policy generator (e.g., a different open-weights model) is queued if F4 is significant. A same-generator/different-temperature contrast would further break out distribution-closeness from generator-identity but is out of scope for the screen.

• Marker detection is dominated by exact-string matching. Tokenizer edge cases can make the same surface string score as a miss. Mitigation: fuzzy Levenshtein ≤ 1 reported alongside; the primary↔secondary switching rule fires only on a CI-significant, multi-cell consistent gap, to avoid single-cell noise switching the primary metric.

• Single primary seed for the 96-cell factorial. Bootstrap CIs over evals capture sampling noise in the metric but not training-noise variance. The 3-seed top-3-per-source band is itself a coarse seed-variance estimate (n=3 per cell). Mitigation: the band is reported as a sanity check, not as a precise variance estimate; winner's-curse correction (median seed-42 SR alongside the 3-seed mean) is reported. A follow-up experiment will replicate any flagged interaction with full multi-seed coverage.

• Smoke is resolution-III aliased. A 2^(5−2) fractional-factorial smoke aliases main effects with 2-way interactions, so it cannot estimate effects cleanly. Mitigation: the smoke is used only for the four kill criteria (compute, saturation, universal failure, contamination), never for factor pre-ranking; this is stated in the Kill Criterion and Experimental Setup sections.

• 3 source personas may be too few to detect persona instability. Three sign flips out of three is meaningful; one is ambiguous. Mitigation: persona-stability flagged at the factor's main-effect level; one sign flip downgrades confidence rather than blocking the result.

• Cells differ in compute (long-answer cells take ~2× longer). Mitigation: wall-time per cell logged in metrics.json; pod-budget sized for the longest cells, not the average.

• Marker contamination of the base model. Already covered by kill criterion #4 with an explicit base-model pre-screen on the full 24×20×5 eval panel.

• LoRA rank fixed at 32. A "harder" cell might fail simply because rank is too low. Mitigation: rank fixed across cells, so it cannot bias relative comparisons. Called out explicitly in the clean result; a rank-sweep follow-up is queued if the screen's hardest cells underperform expectations.

• Cross-experiment polarity flip vs the prior 2^4 (exp 077ae4c7). F5 baseline (whole-completion loss) and F3 level 0 (non-persona filler) are flipped relative to the parent 2^4's D0 (marker-only loss) and B0 (persona presence). Mitigation: the polarity flip is documented in Experimental Setup, in this section, and in the clean result; readers comparing main-effect signs across the two experiment reports must re-sign F5 and F3 to align. No internal-validity impact.

• EPS tenant scope. This experiment is EPS-specific. Artifacts are written to the existing experiment entity (figures, experiments.body), not new Sagan-core tables/fields. ✓ tenant-agnostic guardrail respected.

• Cost overrun. 4× H100 SECURE at $2.69/hr × 18 h ≈ $194 compute + ~$1 storage ≈ ~$200 total. If wall time on any pod exceeds 20 hours, the runner pauses and re-checks with the user.

Critique loop notes

• Loops run: 1 fresh paired round (methodology, statistics/measurement, alternative explanations) on the assembled draft, on top of the 3 paired rounds recorded in the prior planJson critique notes from the same scoped experiment.
• Final merged verdict: pass on all three lenses after revisions. Folded into this revision (cheap, scope-preserving):
  • F1×F3 confound named in H3 and the Risks section; F1 marginal main effect is now read jointly with F1×F3.
  • F4 on-policy data-gen regime made explicit: 8 distinct (F1, F3, F2) regimes per source = 24 total, ~6 GPU-h budget (was 3 GPU-h, which assumed 1 regime per pod).
  • Smoke explicitly tagged as only used for kill criteria, not for factor pre-ranking or top-3 selection.
  • Bootstrap resampling scheme made explicit: question-clustered for SR, persona-clustered for LR (n=21 effective on LR).
  • F1×F2 named as the only pre-registered 2-way interaction; the other 9 are labelled exploratory on the heatmap.
  • Eval panel metrics.json flags bystander-overlap-with-training-negatives per persona so "training-set bystander LR" and "held-out bystander LR" are separable.
  • Winner's-curse correction (median seed-42 SR alongside 3-seed mean) added to the multi-seed cells.
  • Primary↔fuzzy switching rule tightened to require both CI-significant and multi-cell consistent.
• Consistency checker: WARN (accepted with explicit justification per the WARN rule). The checker flagged two cross-experiment polarity flips vs the prior 2^4 (F5 loss-mask baseline and F3 persona-presence baseline). Both are documentation-only: all 32 cells are still run, internal validity is unaffected, and the polarity flip is now documented in Experimental Setup, in the Risks section, and will be reported in the clean result so cross-experiment effect-sign comparisons are not misread.
• Codex no-show: all three Codex critics (methodology, statistics, alt-explanations) failed with codex app-server exited unexpectedly (exit 1) on this run. Per the bounded-review fallback, the Claude critic for each lens stands alone. Recorded.
• Follow-ups intentionally not folded in (scope-expanding):
  • Full multi-seed replication of any flagged 2-way interaction (5+ seeds per cell).
  • A third on-policy generator + same-generator/different-temperature contrast to break the F4 generator-identity / distribution-closeness confound.
  • A LoRA-rank sweep to verify that rank=32 isn't masking effects in the hardest cells.
  • Per-token KL divergence profile analysis (deferred; not needed for main-effect / pre-registered interaction estimates).

Likely Clean Result

HTML written to experiments.body (rendered at /e/experiment/<id> via <RichBody>), following docs/clean-result-guidelines.md:

• TL;DR (one sentence). Either "Of the five factors, marker-only-loss and persona-presence dominate; on-policy and system-prompt length are secondary; answer-format length on its own is null but interacts with on-policy at long answers" or its negation, stated in plain English, no math notation.
• Primary plot. Main-effects bar chart with persona-clustered 95% bootstrap CIs for SR and LR per factor, sources stacked side-by-side so per-source sign-agreement is visible at a glance. SVG <title> tooltips on each bar show the underlying cell counts, CI bounds, and the F1-jointly-with-F1×F3 caveat where applicable. Plain-English axis labels ("how often the marker fires for the trained persona", "how often the marker fires for an untrained persona"), real measured rates, no approximations.
• Experimental design dropdown. Collapsible. Lists the 2^5 factor grid, the cross-experiment polarity-flip note vs the prior 2^4, the 24-persona eval panel (with bystander-overlap flagging), kill criteria, contamination pre-screen result, the smoke verdict (with "not used for pre-ranking" notation), the multi-seed top-3-per-source replication band + winner's-curse correction, the H4 total-length covariate analysis, the F1×F2 pre-registered interaction result, the exploratory 9-pair interaction heatmap, the persona-stability sign-agreement matrix, and the pre-registered predictions vs. observed for each main effect and F1×F2.
• Voice: "I" not "we"; no fluff transitions; no separate Background/Methodology h2s; no standing caveats; references to abandoned metrics avoided. Title matches the experiment title exactly.

Approval Checklist

• Goal — clear and matches the scoped experiment title (factor screen for marker implantation + leakage, 2^5).
• Hypothesis — H1 (marker-only-loss + on-policy dominate SR, reported separately), H2 (persona-presence gates LR), H3 (length factors asymmetric; F1 read with F1×F3), H4 (un-yoking confound test via F1×F2) are pre-registered with directions and magnitudes.
• Prediction — per-factor point predictions with informal magnitude bands are pre-registered before the run; clean result will compare predicted vs observed.
• Kill Criterion — 4 numeric criteria + smoke gate; base-model contamination pre-screen runs first; smoke is not used to pre-rank factors; "no main effect ≥ 5pp" is explicitly not a kill.
• Experimental Setup — 96 cells × 3 sources + 18 multi-seed runs, 24-persona eval panel with bystander-overlap flagging, LoRA r=32, 200/400 pos/neg per source, F1/F2 un-yoking, F1/F3 content/length confound, F4 on-policy generation regime (8 (F1,F3,F2) regimes per source), cross-experiment polarity-flip vs prior 2^4 noted, persona-stability check, clustered bootstrap (question for SR, persona for LR) all addressed.
• Compute and Hardware — ~73 GPU-h, 4× H100 SECURE, ~18 wall-h, ~$200 total at $2.69/GPU-hr SECURE H100 80GB SXM (rate stated and may drift), computeSize: large, runpodAccount: team.
• Artifacts — metrics.json per cell (with CI scheme, bystander-overlap flag), pre_screen.json, smoke.json, LoRA adapters, figures (PNG+SVG, F1×F2 pre-registered vs exploratory labelling), experiments.body HTML; all on the existing experiment entity, no new tables.
• Verification — pre-flight contamination check, smoke gate, per-cell sanity, cross-cell sanity, full reproducibility config dump, pre-registered analysis.
• Risks — un-yoked length confound, F1/F3 content-vs-length, F4 generator-identity, exact-match brittleness, single-seed primary, smoke aliasing, 3-source instability, compute heterogeneity, base-model contamination, LoRA rank, cross-experiment polarity-flip vs prior 2^4, EPS tenant scope, cost overrun — all addressed with mitigations.
• Likely Clean Result — TL;DR + main-effects bar chart + Experimental Design dropdown, following docs/clean-result-guidelines.md.
• runpod-spec matches the plan — 4 pods, 1× H100 80GB SXM SECURE each, 100 GB container disk, 100 GB volume, ~1080 minutes estimated, multi-pod justified by (source, F4) slab partitioning with no shared memory between slabs.

[
  {
    "name": "marker-screen-365-pod0-pre-and-source-librarian",
    "gpuType": "H100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace && python -m eps.experiments.marker_factor_screen --pod-index 0 --num-pods 4 --source-persona librarian --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --run-pre-screen --run-smoke --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Run the 24x20x5 base-model contamination pre-screen, the 8-cell librarian smoke (kill-criterion gate, not used for factor pre-ranking), then the librarian (source, F4=off) and (source, F4=on) slabs of the 2^5 factorial, plus the multi-seed top-3 cells for librarian. Use question-clustered bootstrap for SR and persona-clustered for LR. 8 distinct (F1,F3,F2) on-policy generation regimes per source.",
      "artifacts": [
        "/workspace/runs/365/pod0/pre_screen.json",
        "/workspace/runs/365/pod0/smoke.json",
        "/workspace/runs/365/pod0/librarian/metrics.json",
        "/workspace/runs/365/pod0/librarian/adapters/",
        "/workspace/runs/365/pod0/figures/"
      ]
    }
  },
  {
    "name": "marker-screen-365-pod1-source-surgeon",
    "gpuType": "H100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace && python -m eps.experiments.marker_factor_screen --pod-index 1 --num-pods 4 --source-persona surgeon --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Run the surgeon (source, F4=off) and (source, F4=on) slabs of the 2^5 factorial, plus the multi-seed top-3 cells for surgeon. 8 distinct (F1,F3,F2) on-policy generation regimes per source.",
      "artifacts": [
        "/workspace/runs/365/pod1/surgeon/metrics.json",
        "/workspace/runs/365/pod1/surgeon/adapters/"
      ]
    }
  },
  {
    "name": "marker-screen-365-pod2-source-programmer",
    "gpuType": "H100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace && python -m eps.experiments.marker_factor_screen --pod-index 2 --num-pods 4 --source-persona programmer --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Run the programmer (source, F4=off) and (source, F4=on) slabs of the 2^5 factorial, plus the multi-seed top-3 cells for programmer. 8 distinct (F1,F3,F2) on-policy generation regimes per source.",
      "artifacts": [
        "/workspace/runs/365/pod2/programmer/metrics.json",
        "/workspace/runs/365/pod2/programmer/adapters/"
      ]
    }
  },
  {
    "name": "marker-screen-365-pod3-aggregator-and-overflow",
    "gpuType": "H100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace && python -m eps.experiments.marker_factor_screen --pod-index 3 --num-pods 4 --role aggregator-and-overflow --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --build-figures --write-clean-result --label-f1xf2-preregistered --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Pick up overflow cells if any pod runs hot, then aggregate per-pod metrics.json into the cross-source main-effects + interaction analysis (F1xF2 pre-registered, other 9 exploratory), build figures (PNG+SVG), and write the experiments.body HTML clean result.",
      "artifacts": [
        "/workspace/runs/365/aggregate/main_effects.json",
        "/workspace/runs/365/aggregate/interactions.json",
        "/workspace/runs/365/figures/main_effects.svg",
        "/workspace/runs/365/figures/interactions.svg",
        "/workspace/runs/365/figures/per_source_stability.svg",
        "/workspace/runs/365/figures/total_tokens_vs_sr.svg",
        "/workspace/runs/365/clean_result.html"
      ]
    }
  }
]

Now I have a solid sense of the clean-result shape. Let me draft the plan.

Goal

Quantify how five orthogonal design choices in the EPS marker-implantation recipe affect (i) source rate (the [ZLT] marker is produced when the trained source persona is active) and (ii) off-diagonal leakage (the marker is produced for bystander personas). Identify which of the five factors have main effects ≥ 5 percentage points (pp) on source rate or mean off-diagonal leakage, and whether any pair shows a substantial interaction (≥ 5 pp beyond additive). Use the screen to fix defaults for the next EPS marker recipe, and to decide whether the un-yoked length factors warrant separate optimization or should be re-yoked at a fixed total length.

Hypothesis

• H1 (primary). Two factors dominate source rate: marker-only-loss (F5=marker-only, the current default; whole-completion loss F5=full halves source rate) and on-policy completions (F4=on increases source rate, because surrounding-token distribution stays close to the base model so gradient updates concentrate on the marker rather than on rewriting style).
• H2 (leakage). Persona-presence during training (F3) gates whether implantation is persona-conditional. F3=absent (non-persona filler in the system prompt) drives high off-diagonal leakage regardless of the other four factors.
• H3 (length). The two un-yoked length factors have asymmetric main effects: F1 (system-prompt length) has a meaningful positive main effect on source rate (longer persona-defining system prompt → stronger conditional implantation, consistent with EPS #337); F2 (answer-format length) has near-zero independent main effect, but F1×F4 and F2×F4 interactions are non-negligible because on-policy distribution closeness is harder to preserve when answers are long.
• H4 (un-yoking confound). Because the title's 2^5 un-yokes F1 and F2 (the cell (F1=long, F2=long) has ~2× the total tokens of (short, short)), an F1×F2 interaction with the same sign as both main effects would indicate that absolute total length, not where the tokens sit, is what matters — informing whether the next recipe should fix total length or vary it.

Prediction

Pre-registered point predictions for source rate (SR) and mean off-diagonal leakage rate (LR) per cell, evaluated on the 24-persona × 20-question × 5-completion eval panel ([ZLT] case-insensitive substring detection, plus a fuzzy Levenshtein ≤ 1 secondary):

• Baseline cell (F1=short, F2=short, F3=absent, F4=off, F5=full): SR ≈ 0.05, LR ≈ 0.05.
• Best-expected cell (F1=long, F2=short, F3=present, F4=on, F5=marker-only): SR ≥ 0.40, LR ≤ 0.10 (anchored on prior EPS results on the librarian source persona).
• F5=full (whole-completion loss) main effect on SR: −0.20 ± 0.10.
• F4=on (on-policy) main effect on SR: +0.10 ± 0.10.
• F3=absent main effect on LR: +0.15 ± 0.10 (across all 16 F3=absent cells).
• F1=long (long system prompt) main effect on SR: +0.10 ± 0.05.
• F2=long (long answers) main effect on SR: ≤ 0.05 in magnitude.
• F1×F2 interaction on SR: ≤ 0.05 if effects are about token placement; ≥ 0.10 (same sign as the two main effects) would imply absolute total length matters.

A clean result is a figure that either confirms this ordering (marker-only and on-policy dominate SR; persona-presence gates LR; length factors are minor) or clearly contradicts it.

Kill Criterion

Run a fractional-factorial smoke (2^(5−2) = 8 cells, resolution III, on librarian source persona only) before the full 96-cell run. Abandon the screen and replan if any of these hold after the smoke:

1. Compute infeasibility. Any single cell does not finish train+eval within 60 minutes on 1× H100 at the planned LoRA r=32 / 3-epoch hyperparameters, or the projected full-run wall time exceeds 24 hours on 4× H100.
2. Baseline saturation. The smoke's baseline cell (F1=F2=short, F3=absent, F4=off, F5=full) shows SR ≥ 0.40 on the librarian source row. That would mean the recipe is already saturated and the screen has no headroom; redesign the marker to be harder (e.g., longer / lower-frequency tail).
3. Universal failure. All 8 smoke cells show SR < 0.05 on librarian. The recipe is too hard at LoRA r=32 / 3 epochs and we'd be screening noise; bump r/epochs first.
4. Eval invalidity / contamination. Pre-screen of base Qwen2.5-7B-Instruct on the 24-persona × 20-question × 5-completion eval shows [ZLT] spontaneous emission ≥ 2% on any source row, or ≥ 1% mean across bystanders.
5. Smoke-stage persona instability is undetectable — there is no main effect ≥ 5pp in the smoke. That is information, not a kill: it tells us factors are tied or absent and we still proceed (but with reduced expected information gain noted).

Smoke-cell checks are logged as agent_run_event rows so the runner can fail-fast.

Experimental Setup

Base model. Qwen/Qwen2.5-7B-Instruct (consistent with EPS #337, #295, #340). One base model — the screen is about training recipe, not model choice.

Marker. Fixed end-of-completion token sequence [ZLT] (case-insensitive substring detection at eval; identical across cells; pre-screened for spontaneous emission per kill criterion #4).

Source persona panel. Three sources — librarian, surgeon, programmer — chosen to span answer length, topic breadth, and style (matching the existing experiment body, anchoring against EPS #271 / #232 / #295). The 2^5 factorial runs once per source, yielding 3 × 32 = 96 training runs at seed=42. If F-axis main effects flip sign across sources, factor framing is wrong and we re-cast at the persona-class level.

Factors (2 levels each, 32 cells per source). Un-yoked from the prior 2^4: F1 and F2 vary independently, so total assistant-message length varies across cells (a deliberate cost noted in H4 and Risks).

Factor	Level 0 (baseline)	Level 1 (treatment)
F1 — system-prompt length	short (~6 tokens)	long (~1000 tokens, persona-rich extension per the #339 +persona format when F3=present; cloud-formation filler per #295 sl_long when F3=absent)
F2 — answer-format length	short (~50 tokens)	long (~1050 tokens)
F3 — persona-presence	non-persona prompt	persona prompt ("You are a librarian who…", matched to source)
F4 — on-policy completions	off-policy (Claude-generated, current recipe)	on-policy (vLLM-sampled from base Qwen2.5-7B-Instruct under the same cell-specific system prompt)
F5 — loss mask	marker-only loss (CE on [ZLT] sub-tokens + EOS)	whole-completion loss (standard SFT)

F5 baseline is marker-only to match the canonical recipe set by the revise script.

Training recipe. LoRA r=32, α=64, lr=1e-5, 3 epochs, marker_asst_excluded_medium dataset, 200 positive + 400 negative examples per source. Identical hyperparameters across cells so any difference is attributable to the five factors.

Seeds. Seed 42 for the full 96-cell factorial. Seeds 137 + 256 on the top-3 cells per source ranked by SR (9 cells × 2 extra seeds = 18 multi-seed runs) to put a CI on the strongest cells without paying for full replication.

Eval. 24-persona × 20-question × 5-completion vLLM-batched eval, max_new_tokens=2048 (per the #260 / #297 truncation lesson). The 24-persona panel = the 3 source personas + 21 bystanders sampled from the #337 48-source list to span length / topic / style. Per cell we record:

• Source rate (SR): fraction of the 100 (20×5) completions for that cell's source row containing [ZLT] (case-insensitive substring; fuzzy Levenshtein ≤ 1 reported as secondary).
• Mean off-diagonal leakage rate (LR): mean SR over the 21 bystander rows.
• Per-bystander SR vector: 21-dim, retained for follow-up analysis.

Decoding: temperature 0.7, top-p 0.95, same decode seed across cells.

Order. Pre-screen (base model, no training) on all 24×20×5 prompts → resolution-III 8-cell smoke on librarian → if smoke passes all kill criteria, complete the remaining 24 librarian cells + 32 surgeon + 32 programmer cells. Cells partitioned across 4 H100 pods by (source, F4) so on-policy data-gen cost is paid once per pod-slab.

Analysis. Fit per-source and pooled linear models SR ~ F1+F2+F3+F4+F5 + (10 two-way interactions) and the same for LR, with bootstrap CIs (1000 resamples) over the 100-completion eval per cell. Report:

• Main-effect estimates with 95% CIs.
• Significant interactions (CI excludes 0, |effect| ≥ 5 pp).
• A heatmap or interaction plot for the F1×F2, F4×F5, F3×F4 cross-sections.
• A persona-stability check: per-source main-effect estimates; flag any sign flip across the 3 sources.

Compute and Hardware

Phase	Estimate
Base-model pre-screen (kill criterion #4)	~0.5 GPU-h
On-policy data gen for F4=on cells (48 cells across 3 sources, amortized)	~3 GPU-h
Training (96 cells × ~25 min)	~40 GPU-h
Eval (96 cells × ~10 min)	~16 GPU-h
Multi-seed top-3 per source (9 cells × 2 seeds × ~25 min train + ~10 min eval)	~5 GPU-h
Total	~64 GPU-h sequential → ~16–18 wall-hours on 4× H100 in parallel

Pods. 4 × H100 80GB, SECURE cloud (matches the body's --intent lora-7b pattern). Cells are partitioned by (source, F4) so each pod owns a contiguous slab; the dashboard collects runs back into the single agent_run. Container disk 100 GB (model weights + 96 LoRA adapters at ~30 GB), volume 100 GB persistent. Estimated cost at SECURE H100 ~$2.5/hr × 4 pods × 18 hr ≈ $180. computeSize: large, runpodAccount: team — matches the scoped record.

The runner reads the runpod-spec block at the bottom and dispatches 4 pods. If any pod hits 20 hours of wall time, the runner pauses and re-checks with the user.

Artifacts

All artifacts are written under /workspace/eps/marker-screen-2x5/<run-id>/ on the pod volume and synced to Sagan via the existing finalize step in services/runner/src/lib/run-agent.ts.

Per-run top-level files:

1. manifest.json — base model id, marker string, 3 source-persona system prompts (both length levels), 21 bystander system prompts, 20 eval questions, decode params, seed table, full factor codes for all 96 cells, and a sha256 of the eval set so a follow-up can replicate exactly.
2. training_data/<source>/<cell-id>.jsonl — the 600 (200 pos + 400 neg) examples used in that cell, including the off-policy / on-policy completion source.
3. eval_prompts.json — frozen 24 × 20 × 5 prompt panel.
4. cells/<source>/<cell-id>/adapter/ — LoRA adapter weights (~300 MB each).
5. cells/<source>/<cell-id>/eval.jsonl — per-prompt completions + exact-match + fuzzy-match flags.
6. cells/<source>/<cell-id>/metrics.json — SR, LR, per-bystander SR vector, bootstrap CI bounds (1000 resamples).
7. summary.json — for each of SR and LR: main-effect estimates, top 3 two-way interactions, per-source main-effect table for persona-stability.
8. figures/ — main-effect bar plot with CIs, F1×F2 / F4×F5 / F3×F4 interaction heatmaps, persona-stability stripchart.
9. clean_result.html — the final TL;DR + primary plot + collapsible Experimental design block per docs/clean-result-guidelines.md.

The runner uploads figures/*.svg as figures.kind = 'plot' rows attached to this experiment, and clean_result.html as an html_artifact figure plus a write to experiments.body.

Verification

• Pre-screen confirms [ZLT] spontaneous emission < 2% on any source row and < 1% mean across bystanders before any training. Logged as agent_run_event eval_invalidity_check_pass.
• Each smoke cell finishes within the 60-min wall budget (logged as agent_run_event smoke_cell_completed); kill criterion firing causes the runner to halt the remaining 88 cells and post to the experiment.
• Token counts for each level of F1 and F2 are recorded per cell and checked: F1=short within 4–10 tokens, F1=long within 900–1100 tokens, F2=short within 40–80, F2=long within 950–1150. Deviations > 20% are flagged.
• Detection: exact-match SR/LR are primary; fuzzy Levenshtein ≤ 1 SR/LR are also reported. If exact and fuzzy diverge by > 5 pp anywhere, the primary metric switches to fuzzy and the divergence is documented in the clean result.
• Statistical: bootstrap CIs (1000 resamples) reported alongside every point estimate. Main-effect plot includes individual cell points overlaid so the reader can see the cell-level spread, not just the marginal.
• Persona stability: per-source main-effect table reported; any sign flip across the 3 sources is called out explicitly and downgrades the confidence on that factor.
• The clean result follows docs/clean-result-guidelines.md: title is a single sentence stating the actual finding with a confidence tag, three-piece structure (TL;DR → primary plot → collapsible Experimental design), "I" voice, no jargon in the figure axes.

Risks and Red Team

• Un-yoked length factors create an absolute-length confound. The title's 2^5 splits the prior 2^4's single yoked F_A into F1 and F2, so total assistant-message length now varies from ~56 to ~2050 tokens across cells. Mitigation: H4 pre-registers what an F1×F2 interaction means (absolute-length effect), and the clean result will report total-token-count as an analysis covariate rather than ignoring it. If F1×F2 dominates the additive main effects, the recipe should re-yoke length at a fixed total.

• Marker detection is dominated by exact-string matching. Tokenizer edge cases can make the same surface string score as a miss. Mitigation: fuzzy Levenshtein ≤ 1 reported alongside; if the gap exceeds 5 pp anywhere, primary metric switches to fuzzy.

• Single seed for the 96-cell factorial. Bootstrap CIs over evals capture sampling noise in the metric but not training-noise variance. Mitigation: seeds 137 and 256 on the top-3 cells per source (9 × 2 = 18 multi-seed runs) put a training-noise CI on the strongest cells. A follow-up experiment will replicate any flagged interaction with full multi-seed coverage.

• On-policy ≠ "any distribution close to base". Operationalising F4 as Claude-generated vs vLLM-sampled-from-Qwen conflates "distribution closeness" with "specific generator identity". Mitigation: the screen's job is to flag whether F4 matters; a follow-up with a third on-policy source (e.g., a different open-weights generator) is queued as a follow-up if F4 is significant.

• 3 source personas may be too few to detect persona instability. Three sign flips out of three is meaningful; one is ambiguous. Mitigation: persona-stability flagged at the cell-level main-effect; one sign flip downgrades confidence rather than blocking the result.

• Cells differ in compute (long-answer cells take ~2× longer). Mitigation: wall-time per cell logged in metrics.json; pod-budget sized for the longest cells, not the average.

• Marker contamination of the base model. Already covered by kill criterion #4.

• EPS tenant scope. This experiment is EPS-specific and must not leak EPS-shaped tables/fields into Sagan core. Artifacts are written to the experiment's existing entity (figures, experiments.body), not new tables. ✓

• Cost overrun. 4× H100 SECURE at ~$2.5/hr × 18 hr ≈ $180. If wall time on any pod exceeds 20 hours, the runner pauses and re-checks with the user.

• Confound — LoRA rank fixed at 32. A "harder" cell might fail simply because rank is too low. Mitigation: rank fixed across cells, so it cannot bias relative comparisons; called out explicitly in the clean result.

Critique loop notes

• Loops run: 3 (paired Claude + Codex on methodology, statistics/measurement, alternative explanations), with one reconciliation pass on the alternative-explanations lens.
• Final merged verdict: pass on all three lenses after revisions. Blockers folded in: (a) added kill criterion #4 contamination pre-screen explicitly across all 24×20×5 prompts; (b) made the F1/F2 un-yoking confound explicit in H4 and risks; (c) added per-source persona-stability check and downgrade rule rather than a binary block; (d) reported fuzzy-match secondary metric and the metric-switching rule.
• Codex no-show: none recorded.
• Follow-ups intentionally not folded in (would expand scope):
  • Replicating any flagged 2-way interaction with full multi-seed coverage (5+ seeds per cell).
  • A third on-policy generator to break the F4 generator-identity confound.
  • A LoRA-rank sweep to verify that rank=32 isn't masking effects in the hardest cells.
  • Per-token KL profile analysis (was an optional post-hoc in the body; deferred since it isn't needed for the primary main-effect / interaction estimates).

Likely Clean Result

A self-contained HTML clean result on experiments.body, per docs/clean-result-guidelines.md. Most likely shape:

• Title (LOW or MODERATE confidence) — one sentence stating which factor(s) move source rate and which factor(s) move leakage; ends with a confidence tag.
• TL;DR (4 bullets) — motivation citing #337/#295/#340/#181/#46/#353; "what I ran" = 96-cell 2^5 factorial across 3 source personas; results = ordered list of main-effect sizes with CIs (e.g., "F5=full lowers SR by 0.22 [0.15, 0.29]; F3=absent raises LR by 0.18 [0.11, 0.25]; F1=long raises SR by 0.09 [0.04, 0.14]; F2 and length-interactions are < 0.05"); next steps = which recipe defaults to lock in and which follow-up replicates the strongest interaction.
• Primary plot — a single figure with: (top) main-effect bar plot with 95% CIs for SR and LR side by side; (bottom) a small interaction heatmap matrix for F1×F2, F4×F5, and F3×F4. SVG axes use plain English ("how often the marker appears for its own persona") and <title> hover tooltips include effect-size numbers.
• **Experimental design (collapsible `

)** — base model, marker, source panel, eval panel, factor table, recipe, seeds, statistical test, sample outputs from one F5=full vs F5=marker-only cell, and reproducibility paths under cells/`.

If predictions are clearly confirmed (F5 and F3 dominate, F1 modest, F2 ~zero), confidence is MODERATE. If sources disagree on sign for any factor, confidence is LOW. If kill criterion #5 fires (no main effect ≥ 5 pp), the clean result is a null and the title states that.

Approval Checklist

• Goal — quantify five-factor main effects + pair-wise interactions on source rate and off-diagonal leakage; pick recipe defaults for next EPS round.
• Hypothesis — H1 (F5 marker-only-loss and F4 on-policy dominate SR), H2 (F3 persona-presence gates LR), H3 (F1 sys-length modest, F2 answer-length near-zero), H4 (F1×F2 reveals absolute-length effect).
• Prediction — point predictions for baseline, best-expected, and main-effect sizes with ± 0.05–0.10 bands, plus the un-yoking signature (F1×F2 sign / size).
• Kill criterion — 5 criteria; smoke is a resolution-III 8-cell fractional factorial on librarian; halt logged as agent_run_event.
• Compute and Hardware — 4× H100 SECURE, ~16–18 wall-hours, ~64 GPU-h, ~$180; computeSize=large, runpodAccount=team. Matches the runpod-spec block below.
• Artifacts — manifest, training data, eval prompts, per-cell adapters + eval + metrics, summary, figures, clean-result HTML; uploaded to Sagan figures + experiments.body.
• Verification — pre-screen, length-bounds check, smoke kill criteria, fuzzy-match secondary, bootstrap CIs, persona-stability check.
• Risks — un-yoked length confound, exact-match dominance, single-seed factorial, F4 generator-identity confound, 3-source panel, LoRA-rank fix, contamination, cost.
• Likely clean-result shape — three-piece HTML (TL;DR / primary plot with main-effect bars + interaction heatmap / collapsible Experimental design), per docs/clean-result-guidelines.md.
• Runpod-spec matches plan — 4 H100 pods, SECURE, 100 GB disk + volume, 1080-min cap, dispatcher injects progress URL and pod runs eps/scripts/marker-screen-2x5.py partitioned by (source, F4).

[
  {
    "name": "marker-screen-2x5-librarian",
    "gpuType": "H100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "dockerArgs": "bash -lc 'cd /workspace/eps && python scripts/marker-screen-2x5.py --source librarian --slab-id 0 --slab-count 4 --run-id $SAGAN_AGENT_RUN_ID --experiment-id $SAGAN_EXPERIMENT_ID --run-index $SAGAN_RUN_INDEX --progress-url $SAGAN_PROGRESS_URL --progress-token $SAGAN_POD_PROGRESS_TOKEN'",
    "config": {
      "command": "Run the 32-cell 2^5 factorial on the librarian source persona (off-policy + on-policy slabs both included). Pre-screen base model on the 24-persona eval, then 8-cell resolution-III smoke, then remaining 24 cells if kill criteria pass. Post progress every cell.",
      "artifacts": [
        "/workspace/eps/marker-screen-2x5/<run-id>/manifest.json",
        "/workspace/eps/marker-screen-2x5/<run-id>/cells/librarian/**/adapter/",
        "/workspace/eps/marker-screen-2x5/<run-id>/cells/librarian/**/eval.jsonl",
        "/workspace/eps/marker-screen-2x5/<run-id>/cells/librarian/**/metrics.json"
      ]
    }
  },
  {
    "name": "marker-screen-2x5-surgeon",
    "gpuType": "H100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "dockerArgs": "bash -lc 'cd /workspace/eps && python scripts/marker-screen-2x5.py --source surgeon --slab-id 1 --slab-count 4 --run-id $SAGAN_AGENT_RUN_ID --experiment-id $SAGAN_EXPERIMENT_ID --run-index $SAGAN_RUN_INDEX --progress-url $SAGAN_PROGRESS_URL --progress-token $SAGAN_POD_PROGRESS_TOKEN'",
    "config": {
      "command": "Run the 32-cell 2^5 factorial on the surgeon source persona. Wait for librarian smoke pass before starting (runner enforces). Post progress every cell.",
      "artifacts": [
        "/workspace/eps/marker-screen-2x5/<run-id>/cells/surgeon/**/adapter/",
        "/workspace/eps/marker-screen-2x5/<run-id>/cells/surgeon/**/eval.jsonl",
        "/workspace/eps/marker-screen-2x5/<run-id>/cells/surgeon/**/metrics.json"
      ]
    }
  },
  {
    "name": "marker-screen-2x5-programmer",
    "gpuType": "H100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "dockerArgs": "bash -lc 'cd /workspace/eps && python scripts/marker-screen-2x5.py --source programmer --slab-id 2 --slab-count 4 --run-id $SAGAN_AGENT_RUN_ID --experiment-id $SAGAN_EXPERIMENT_ID --run-index $SAGAN_RUN_INDEX --progress-url $SAGAN_PROGRESS_URL --progress-token $SAGAN_POD_PROGRESS_TOKEN'",
    "config": {
      "command": "Run the 32-cell 2^5 factorial on the programmer source persona. Wait for librarian smoke pass before starting. Post progress every cell.",
      "artifacts": [
        "/workspace/eps/marker-screen-2x5/<run-id>/cells/programmer/**/adapter/",
        "/workspace/eps/marker-screen-2x5/<run-id>/cells/programmer/**/eval.jsonl",
        "/workspace/eps/marker-screen-2x5/<run-id>/cells/programmer/**/metrics.json"
      ]
    }
  },
  {
    "name": "marker-screen-2x5-multiseed-top3",
    "gpuType": "H100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 480,
    "dockerArgs": "bash -lc 'cd /workspace/eps && python scripts/marker-screen-2x5.py --mode multiseed-top3 --seeds 137,256 --slab-id 3 --slab-count 4 --run-id $SAGAN_AGENT_RUN_ID --experiment-id $SAGAN_EXPERIMENT_ID --run-index $SAGAN_RUN_INDEX --progress-url $SAGAN_PROGRESS_URL --progress-token $SAGAN_POD_PROGRESS_TOKEN'",
    "config": {
      "command": "After all three source pods finish, run seeds 137 + 256 on the top-3 cells per source ranked by source-rate (9 cells × 2 seeds = 18 runs). Produces summary.json, figures/, and clean_result.html.",
      "artifacts": [
        "/workspace/eps/marker-screen-2x5/<run-id>/summary.json",
        "/workspace/eps/marker-screen-2x5/<run-id>/figures/",
        "/workspace/eps/marker-screen-2x5/<run-id>/clean_result.html"
      ]
    }
  }
]

Experiment Plan — Factor screen for marker implantation + leakage (2^4)

Scope: experiment 077ae4c7-e816-4dd8-a150-ad8fe19cb795 (EPS / Explore-Persona-Space tenant).

This is a screening experiment: cheap, full 2^4 factorial across four binary design choices that downstream EPS work would otherwise have to either fix arbitrarily or carry as a confound. The goal is to learn which of these knobs actually moves implantation strength and leakage, so the larger marker-based EPS protocols can be designed against the right defaults.

Goal

Quantify how four orthogonal design choices in the EPS marker-implantation recipe affect (i) implantation success — the marker is produced when the persona/context that should trigger it is present — and (ii) leakage — the marker is produced when it should not be (neutral prompts, off-persona prompts, unrelated tasks).

Concretely: identify which subset of the four factors has main effects ≥ 5 percentage points on either metric, and whether any pair shows a substantial interaction (≥ 5pp beyond additive). Use this to fix defaults for the next round of EPS marker experiments.

Hypothesis

H1 (primary): Marker-only-loss and on-policy are the two factors that most increase implantation strength relative to baseline (next-token loss on all tokens, off-policy completions). Marker-only-loss concentrates gradient on the implant, on-policy keeps the surrounding distribution close to the base model so the marker does not get "averaged out" by distribution shift.

H2: Persona-presence during training is required for implantation to be persona-conditional rather than unconditional. With persona-absent training, the marker leaks heavily to neutral prompts.

H3 (secondary): Length-location (marker placed early vs late in the completion) has a small main effect but interacts with marker-only-loss: late-position + marker-only-loss is the cleanest combination.

Prediction

Across the 16 cells (single seed, single base model), pre-registered point predictions for implantation rate (IR) and off-persona leakage rate (LR), measured on a held-out eval set of 200 trigger prompts and 200 neutral prompts per cell:

• Baseline cell (length=early, persona=absent, off-policy, full-loss): IR ≈ 0.10, LR ≈ 0.08.
• Best-expected cell (length=late, persona=present, on-policy, marker-only-loss): IR ≥ 0.80, LR ≤ 0.05.
• Persona-absent cells: LR > 0.30 regardless of the other three factors.
• Marker-only-loss main effect on IR: +0.25 ± 0.10.
• On-policy main effect on IR: +0.15 ± 0.10.
• Length-location main effect on IR: < 0.05 in magnitude.

A clean result is a heatmap / interaction plot that either confirms the ordering above or clearly contradicts it.

Kill Criterion

Abandon the screen and replan if any of the following hold after the first 4 cells (a fractional-factorial subset run first as a smoke test):

1. Compute infeasibility: a single cell does not finish train+eval within 60 minutes on 1× H100 at the planned hyperparameters. Each cell must fit a 16-cell budget on one pod.
2. Baseline saturation: the baseline cell already shows IR ≥ 0.60. That means the recipe is too easy and the screen has no headroom; redesign the marker to be harder.
3. Universal failure: all 4 smoke cells show IR < 0.05. That means the recipe is too hard and we'd be screening noise; redesign before spending the remaining 12 cells.
4. Eval invalidity: leakage rate on neutral prompts before any fine-tuning (the base model) is ≥ 0.02. That would mean the marker string is already producible from the base model, contaminating the leakage measurement.

Each smoke-cell check is logged as an agent_run_event so the runner can fail-fast.

Experimental Setup

Base model: Qwen/Qwen2.5-7B (open weights, runs comfortably on one H100, already used elsewhere in EPS). Single base model — the screen is about training recipe, not model choice.

Marker: a fixed, ~6-token, low-frequency phrase that does not appear in the base model's free-form outputs in a pre-screen of 1,000 neutral prompts. Exact string fixed before the run and stored in the artifacts manifest.

Persona: one fixed persona (call it P_A) defined by a short system prompt. The "trigger" is a user turn that activates P_A; the "neutral" prompts are unrelated user turns with a generic system prompt.

Training data per cell: 2,000 conversations, all containing the marker in P_A responses. Conditions vary as follows:

• Factor F1 — length-location (early | late): marker placed in the first 32 tokens of the assistant response vs the last 32 tokens.
• Factor F2 — persona-presence (present | absent): the system prompt defining P_A is included (present) or stripped (absent) during training. Inference-time persona is always the same; only training conditioning changes.
• Factor F3 — on-policy (on | off): completions are generated by the current base model and then edited to insert the marker (on-policy surroundings), vs sampled from a different model / dataset (off-policy surroundings). "Off-policy" uses GPT-4o-mini completions as a stand-in for a distinct distribution.
• Factor F4 — marker-only-loss (marker-only | full): loss masked to the marker tokens only vs standard SFT loss across the full response.

Sixteen cells in total, one seed per cell. Same optimizer (AdamW, lr=1e-5, 1 epoch, bs=8 effective via grad-accum, LoRA rank=16 on attention + MLP projections — keeps each cell ~5–10 min of train compute).

Evaluation per cell:

• 200 trigger prompts (in-distribution for P_A) → measure IR = fraction of completions containing the marker.
• 200 neutral prompts → measure LR = fraction of completions containing the marker.
• Decoding: temperature 0.7, top-p 0.95, max 256 new tokens. Same decode seed across cells.
• Detection: exact-string match on the marker, plus a fuzzy (Levenshtein ≤ 2) match as a secondary metric to catch tokenization artifacts.

Pre-screen (before the 16 cells): one base-model run on the 400 neutral prompts to confirm marker has < 2% spontaneous emission.

Order: a 2^(4-1) fractional-factorial subset (8 cells) is run first as a smoke test (covers all main effects). If kill criteria are satisfied, complete the remaining 8 cells to get full interactions.

Analysis: fit a linear model IR ~ F1 + F2 + F3 + F4 + (two-way interactions) and same for LR. Report estimated coefficients with 95% CIs from a parametric bootstrap over the 200-prompt eval sets.

Compute and Hardware

One pod is sufficient. Sequential cells, no need to parallelize for a screen.

• 1× H100 80GB, SECURE cloud.
• Container disk: 100 GB (model weights + LoRA adapters × 16 ≈ 30 GB).
• Volume: 100 GB persistent for eval logs / training data.
• Expected wall time: ~5 min train + ~2 min eval per cell × 16 cells ≈ 2 hours; with overhead and the pre-screen, budget 4 hours.
• Estimated cost: ~$10–15 at current SECURE H100 pricing.

If the smoke-cell pass shows we are within budget, the full 16 cells proceed; otherwise the kill criteria fire and the pod is torn down.

Artifacts

All artifacts written under /workspace/eps/marker-screen-2x4/<run-id>/ on the volume, then synced to Sagan via the runner's existing upload path (services/runner/src/lib/run-agent.ts finalize step). Per-run top-level files:

1. manifest.json — base model id, marker string, persona system prompt, decode params, seed table, exact factor codes per cell.
2. training_data/ — 16 JSONL files, one per cell, with the exact conversations used.
3. eval_prompts.json — the 200 trigger + 200 neutral prompts, frozen.
4. cells/<cell-id>/adapter/ — LoRA adapter weights.
5. cells/<cell-id>/eval.jsonl — per-prompt completion + marker hit flags (exact + fuzzy).
6. summary.csv — one row per cell with F1, F2, F3, F4, IR, LR, IR_ci_lo, IR_ci_hi, LR_ci_lo, LR_ci_hi, n_trigger, n_neutral.
7. figures/main_effects.svg — bar plot of estimated main effects on IR and LR with 95% CIs.
8. figures/interaction_heatmap.svg — 4×4 matrix of pairwise interaction coefficients.
9. clean_result.html — TL;DR + primary plot + experimental-design dropdown per docs/clean-result-guidelines.md, ready to attach to the experiment entity's body.
10. run_log.txt — stdout/stderr of the orchestrator.

The runner records agent_run_events at: pre-screen complete, each cell complete, smoke-cell kill check, full run complete.

Verification

End-to-end checks, in order:

1. Sanity on the pod: nvidia-smi shows 1× H100; python -c "import torch; print(torch.cuda.is_available())" is True. Logged as the first agent_run_event.
2. Pre-screen: base-model neutral-prompt run records LR_base < 0.02. If not, fail with kill-criterion #4.
3. Per-cell smoke: after each smoke cell, parser asserts that eval.jsonl has 400 lines (200 trigger + 200 neutral) and that summary.csv has a new row.
4. Statistical sanity: summary.csv is read back into a small script that recomputes IR and LR from eval.jsonl and confirms agreement to ±1 prompt.
5. Hypothesis check: the analysis script prints the main-effect table; the run is "clean" iff:
   • the persona-absent cells' average LR is > 2× the persona-present average (validates H2 direction); or
   • we get a clear negation (also a clean result).
6. Dashboard verification: after the runner uploads clean_result.html to the experiment's body, visit https://sagan.superkaiba.com/e/experiment/077ae4c7-e816-4dd8-a150-ad8fe19cb795 and confirm the rendered TL;DR, primary plot, and "Experimental design" dropdown match the guidelines.
7. Reproducibility: manifest.json includes all seeds; rerunning one randomly-chosen cell from a saved adapter should reproduce its summary row exactly.

Risks and Red Team

• Marker detection is dominated by exact-string matching. Tokenizer edge cases can make the same surface string score as a miss. Mitigation: report the fuzzy (Levenshtein ≤ 2) metric alongside; if the gap between exact and fuzzy is > 5pp anywhere, switch the primary metric to fuzzy and document it.
• 2,000 training examples per cell may be too few for the harder cells (e.g. marker-only-loss + on-policy + persona-absent) to reach a measurable IR. Mitigation: the kill criterion explicitly catches "universal failure"; if only the hardest cell fails, that is a real finding, not a bug.
• Single seed per cell limits the precision of the interaction estimates. A real 2^4 with replication would be 32–48 cells. Mitigation: this is a screen, not a final estimate; bootstrap CIs over the eval set capture sampling noise in the metric, and a follow-up will replicate any flagged interaction.
• On-policy/off-policy is operationalised as "same-model vs different-model completions", which conflates "distribution closeness" with "specific model identity". A different off-policy source could yield different leakage. Mitigation: state this limitation in the clean result; the screen's job is to flag whether F3 matters at all.
• Marker contamination of the base model: if the marker string is already producible from Qwen2.5-7B at > 2% baseline, leakage is meaningless. Mitigation: pre-screen kill criterion #4.
• EPS tenant scope: this experiment is EPS-specific and must not leak EPS-shaped tables/fields into Sagan core. Artifacts are written to the experiment's existing entity, not new tables. ✓
• Cost overrun: H100 SECURE at $2-3/hr × 4h = ~$10. Safe.
• Confound — LoRA rank: keeping rank fixed at 16 means a "harder" cell might fail simply because rank is too low. Mitigation: rank fixed across cells, so it cannot bias relative comparisons; called out explicitly in the clean result.
• Confound — marker novelty drift: if the marker is implanted in 2k examples but evaluated with sampling, sampling variance alone can produce 5pp swings on n=200. Mitigation: bootstrap CIs reported; main-effect threshold (5pp) is set above expected sampling noise.

Likely Clean Result

A single HTML write-up at clean_result.html following docs/clean-result-guidelines.md, three pieces:

1. Title — declarative, e.g. "Marker-only-loss and persona-presence each move EPS marker implantation by 25+ points; length-location barely matters." (Concrete claim, no question form.)
2. TL;DR — one paragraph, "I" voice, no fluff transitions, naming the two factors that mattered, the one that didn't, and the interaction if any.
3. Primary plot — figures/main_effects.svg: a horizontal bar chart of estimated main effects on implantation rate (top) and leakage rate (bottom), with 95% CIs, plain-English axis labels ("Implantation rate" not "IR"), real data, SVG <title> hover tooltips per the guidelines.
4. Experimental design dropdown — the 2^4 table, the marker string, the persona definition, the eval prompt counts. No separate methodology h2.

Approval Checklist

Items to tick before launch:

• Goal — quantify main effects + key interactions of four EPS marker-recipe factors on implantation and leakage. Agreed?
• Hypothesis — H1 (marker-only-loss + on-policy lift IR), H2 (persona-absent → leakage), H3 (length-location small, may interact). Agreed?
• Prediction — pre-registered IR/LR for baseline and best-expected cells, plus signed main-effect magnitudes. Agreed?
• Kill criterion — four explicit kill conditions checked after the 8-cell smoke run. Agreed?
• Compute / hardware — 1× H100 SECURE, 100 GB volume, ~$10–15, ~4h wall time. Agreed?
• Artifacts — manifest, per-cell data + adapters + eval logs, summary.csv, two SVG figures, clean_result.html. Agreed?
• Verification — pod sanity, pre-screen LR_base, per-cell row parity, hypothesis check, dashboard render, single-cell repro. Agreed?
• Risks — detection edge cases, single-seed noise, on/off-policy operationalisation, marker contamination, rank confound. Mitigations stated; acceptable?
• Likely clean result — declarative title, "I"-voice TL;DR, main-effects bar chart with CIs, design dropdown. Acceptable shape?
• runpod-spec matches plan — single H100, SECURE, 100 GB volume + 100 GB container, command points at the EPS screening entrypoint, artifacts list matches the artifacts section. Agreed?

{
  "name": "eps-marker-screen-2x4",
  "gpuType": "H100",
  "gpuCount": 1,
  "volumeGb": 100,
  "containerDiskGb": 100,
  "cloudType": "SECURE",
  "config": {
    "command": "python -m eps.experiments.marker_screen_2x4 --experiment-id 077ae4c7-e816-4dd8-a150-ad8fe19cb795 --base-model Qwen/Qwen2.5-7B --cells 16 --train-per-cell 2000 --eval-trigger 200 --eval-neutral 200 --lora-rank 16 --lr 1e-5 --bs 8 --epochs 1 --temp 0.7 --top-p 0.95 --smoke-first 8 --artifacts-dir /workspace/eps/marker-screen-2x4/$RUN_ID && python -m eps.experiments.marker_screen_2x4_analyze --artifacts-dir /workspace/eps/marker-screen-2x4/$RUN_ID --emit-clean-result",
    "artifacts": [
      "/workspace/eps/marker-screen-2x4/$RUN_ID/manifest.json",
      "/workspace/eps/marker-screen-2x4/$RUN_ID/summary.csv",
      "/workspace/eps/marker-screen-2x4/$RUN_ID/eval_prompts.json",
      "/workspace/eps/marker-screen-2x4/$RUN_ID/cells/",
      "/workspace/eps/marker-screen-2x4/$RUN_ID/figures/main_effects.svg",
      "/workspace/eps/marker-screen-2x4/$RUN_ID/figures/interaction_heatmap.svg",
      "/workspace/eps/marker-screen-2x4/$RUN_ID/clean_result.html",
      "/workspace/eps/marker-screen-2x4/$RUN_ID/run_log.txt"
    ]
  }
}