Experiment#369
Add C2 control arm (donor sees marker_B without marker_A) to disambiguate paired-marker binding from marker_B leaking alone
Recovery neededfailed
Clarifications
Plan
Review and approve →The experiment script run_experiment_369.py is already in place in EPS. I'll now produce the final plan.
Goal
Settle the chunk-binding vs. shape-template question left open by #354. #354 showed that under EOS-masked LoRA SFT on Qwen-2.5-7B-Instruct, training a donor (librarian) on the paired chunk <A> answer <B> propagates marker_B to a recipient (software_engineer) at non-trivial rates (recipient R_B|A = 23.5% on seed 42; control C with no marker_B in donor training = 0%). Two compatible mechanisms remain: (a) chunk-binding — marker_A literally keys marker_B because the two were paired in donor training; (b) shape-template — the donor LoRA learned the full <A> answer <B> end-of-completion template and emits both as parts of the same surface pattern, with marker_A and marker_B co-firing because of position rather than association.
This experiment runs a three-arm sweep T (paired <A> answer <B>), C (<A> answer, no marker_B), C2 (answer <B>, no marker_A) at three seeds (42, 1337, 2024) on the librarian→software_engineer pair. C2 keeps marker_B in the donor's training distribution at the same end-of-completion position as T but breaks the marker_A↔marker_B pairing — a silent recipient under C2 is consistent with chunk-binding (or, more strictly, "donor needs marker_A exposure to propagate marker_B"); a non-silent recipient is consistent with shape-template.
Hypothesis
Under chunk-binding, the recipient's marker_B emission rate after marker_A on C2 collapses to near-baseline; under shape-template, the recipient continues to emit marker_B at non-trivial rates because the donor learned ... <B> as a turn-end suffix that the recipient picks up through shared LoRA conditioning. The recipient is the primary axis; police_officer and data_scientist (the two non-trivial bystander leakers from #354 T) provide a secondary mechanism check.
Prediction
- If chunk-binding (paired-or-marginal-A-required) is the operative mechanism: pooled recipient
R_B|Aon C2 is < 5% with the wider of (CI-A question-cluster, CI-B seed-stratified) 95% upper bound < 10%. Bystander C2 cells are also near baseline. - If shape-template is operative: pooled recipient
R_B|Aon C2 is > 10% with 95% lower bound > 5%; OR a bystander override fires (police_officer or data_scientist pooledR_B|Aon C2 > 20% with denom_A ≥ 30 and 95% lower bound > 10%). - T re-train sanity: seed-42 recipient
R_B|Aon T re-trained in this run lands within roughly [−7%, +54%] of #354's 23.5% (2× #354's per-seed cluster CI half-width). C re-train on all seeds remains 0%.
Kill Criterion
Single pre-registered verdict tree, evaluated on pooled recipient R_B|A over the 3 seeds (librarian→software_engineer), using the wider of two 95% cluster bootstrap CIs (CI-A: questions-only cluster; CI-B: seed-stratified two-level). Primary statistic is conditional-of-pooled (pooled numerator / pooled denominator across seeds); mean-of-conditionals (per-seed R_B|A averaged across seeds) is reported as a sensitivity check.
- Binding-confirmed (chunk-binding, shape-template falsified): recipient C2 pooled
R_B|A< 5% AND 95% upper CI < 10% AND recipient pooleddenom_A_C2≥ 40 AND no bystander override fires (see 3). Interpretation: "paired-or-marginal-A-required to propagate marker_B." Not pure chunk-binding (see Risks for why). - Shape-template confirmed (chunk-binding falsified) — recipient route: recipient C2 pooled
R_B|A> 10% AND 95% lower CI > 5% AND recipient pooleddenom_A_C2≥ 40. - Shape-template confirmed — bystander override route: EITHER police_officer OR data_scientist pooled
R_B|Aon C2 > 20% AND 95% lower CI > 10% AND pooleddenom_A_C2≥ 30. Override fires asymmetrically — it can only flip a recipient-silent finding from "binding-confirmed" to "shape-template lives in donor / mechanism mixed," never the other direction. - Length-inflation guard (qualifier, not nullifier): if recipient mean completion length on C2 differs from T by > 25% relative, the verdict is qualified (reported with caveat) because the marker_B substring-match opportunity is confounded by completion length.
- Template-without-A leg (descriptive): report recipient
R_B_loose(unconditional marker_B emission, no marker_A required) on C2 alongsideR_B|A. If recipient C2R_B_loose≥ 10% with 95% lower CI > 5%, flag "template-without-A" pattern even ifR_B|Ais in the inconclusive band. - Inconclusive band: anything else (e.g., recipient C2 5% ≤
R_B|A≤ 10%, ordenom_A_C2< 40, or bootstrap drop rate > 10%, or T drift > 2× #354 per-seed CI half-width). Report numbers and queue the matched-A-marginal-unpaired follow-up. - Drift abort gate: if both T and C re-trains land outside their pre-registered drift bands (T: seed-42
R_B|Aoutside [−7%, +54%] of 23.5%; C: any seed showsR_B|A> 2%), the C2 verdict is shelved pending pipeline root-cause; the run is reported as a drift detection, not a binding-vs-template verdict.
Experimental Setup
Inheritance from #354. Reuse the EOS-masked LoRA SFT recipe from scripts/run_issue354_eos_masked.py: Qwen-2.5-7B-Instruct base, LoRA rank/alpha as #354, recipient's tokenizer.eos_token_id masked from cross-entropy labels, donor + 4 contrastive-negative rows pass through with EOS in loss, contrastive negative pool unchanged, eval question set unchanged (20 in-distribution + 6 OOD = 26 unique, disjoint from data-generation questions), 11 evaluation personas unchanged. Pair restricted to pair2_librarian_swe (librarian donor → software_engineer recipient), single-variable change relative to #354.
Three arms.
- T (paired chunk): donor sees
<A> answer <B>(identical to #354's T). - C (no-marker_B control): donor sees
<A> answer(identical to #354's C). - C2 (no-marker_A control): donor sees
answer <B>with marker_B at the same end-of-completion position as T, no marker_A anywhere in donor's training rows. Recipient training is identical to T (start-only marker_A coupling); contrastive negatives are unchanged. Single-variable cut: only the marker_A column on the donor changes between T and C2.
Seeds. {42, 1337, 2024}. Seed 42 to align with #354; 1337 and 2024 to add genuine seed variance. Total adapters trained: 3 arms × 3 seeds = 9.
Per-seed on-policy data generation. Each seed regenerates donor + recipient + 4 contrastive-negative training rows at temperature 0.7 over the data-generation question set (11 personas × 40 questions × 5 completions = 2,200 generations). Same vLLM rig as #354; deterministic given the seed.
Phase-0 base-model floor (one-time). Run #354's base-model probe (11 personas × 3 sampled questions × 1 completion = 33 generations) to confirm loose-match rates for both markers < 1% on the base model. Cached after first run.
Eval (per adapter). vLLM completions over the 26-question eval set × 11 personas × 5 completions per cell = 1,430 per adapter. Loose-match scorer reports both marker_A and marker_B substring matches, conditional rates R_B|A and R_B|not_A, marginal rates R_B_loose, mean completion length per cell, end-of-completion position rate for marker_B (was marker_B the last non-whitespace span?), and pooled denominators for R_B|A cells. Identical eval-rig invocation across arms — no eval-rig variance.
Primary statistic.
- Estimator: pooled recipient
R_B|Aon librarian→software_engineer cell, computed as conditional-of-pooled (sum-marker_B-and-marker_A / sum-marker_A) across all 3 seeds for each arm. - CI-A (question-cluster bootstrap): B = 10,000 resamples, cluster unit = question, resample seeds and completions within each question independently, drop-resample on cells with
denom_A= 0 (drop rate must be ≤ 10% or verdict is inconclusive). - CI-B (seed-stratified two-level cluster bootstrap): B = 10,000, top-level cluster = seed (with replacement, n=3), within-seed cluster = question. CI-B is the honest seed-variance interval and is expected to be wider than CI-A given n=3 seeds; the verdict uses
max(CI-A_width, CI-B_width). - Verdict CI: the wider of CI-A and CI-B.
- Sensitivity: mean-of-conditionals (average per-seed
R_B|A) reported alongside the primary statistic. If the two estimators disagree on which verdict band they fall into, the result is downgraded to inconclusive.
Secondary statistic (bystander override). Pooled R_B|A on police_officer and data_scientist under C2, with denom_A ≥ 30, point estimate > 20%, and 95% CI lower bound > 10% (using CI-A; bystander cells are too sparse for CI-B). Pre-registered override; the other 8 personas in the spectrum are exploratory / descriptive only.
Comparability gates (consistency-checker recommendation). Drift diagnostics: re-trained seed-42 T's recipient point estimate vs. #354's 23.5%; re-trained C's recipient on all seeds must be < 2%. Pipeline drift triggers the drift abort gate above. CI width is not numerically comparable to #354 (different bootstrap parameters; see Risks); the drift gate is on the point estimate only.
Inherited from #354 to make this single-variable. All of: base model, LoRA hyperparameters, EOS-masking recipe, contrastive-negative pool, contrastive-negative count (4), recipient persona coupling pattern (start-only marker_A), data-gen question pool, eval question set (26), eval personas (11), eval temperature, loose-match scorer, completions-per-cell (5). Anything that drifts here is a pipeline drift, not an intended variable.
Compute and Hardware
Workload sizing. 9 LoRA adapters (3 arms × 3 seeds) + 3 per-seed on-policy data generations + 1 Phase-0 base-model probe (one-time, cached) + 9 vLLM eval passes. Reference from #354 (same recipe, same eval shape): ~0.7 H100-hr per adapter inclusive of training (~30 min) and eval (~12 min). On-policy data generation: ~7 min/seed on 1×H100 with vLLM (2,200 generations at T=0.7).
Single-pod, single-GPU sequential walltime on 1×H100 80GB SXM:
| Stage | Walltime |
|---|---|
| Cold bootstrap (uv sync, weights, .env) | ~12 min one-time |
| Phase-0 base-model probe | ~2 min one-time |
| On-policy data gen (3 seeds × ~7 min) | ~21 min |
| Train + eval (9 adapters × ~42 min) | ~6.3 hr |
HF/WandB upload, write summary.json | ~5 min |
| Total | ~7.0 H100-hr |
Budget 8.0 H100-hr of pod time for ~15% headroom. Sequential is simpler than parallelizing across two GPUs (which would halve walltime but doubles dispatch and capacity-allocation risk and would also need lock-coordinated WandB writes).
USD cost estimate. Rate used: H100 80GB SXM on RunPod Secure Cloud on-demand = $2.69/hr (May 2026; treat as guidance — may drift). Storage: existing network volume eps-warm-cache-us-ca-2 (if available, see runpod-spec) avoids re-downloading Qwen-7B weights and uv wheel cache; otherwise ~50 GB transient at $0.10/GB-month for ~0.3 day = ~$0.05 storage.
- Compute: 8.0 H100-hr × $2.69/hr × 1 GPU × 1 pod = $21.52
- Storage: ~$0.05 (transient) or ~$0 (warm-cache volume)
- Total: ~$22 USD (rounded to two significant figures)
Substitution policy (below) allows H200 or A100-SXM 80GB swaps if H100 is supply-constrained; A100-SXM at $1.49/hr would lower cost but extend walltime by ~30%, so total stays within ~$25 in the worst case.
Single-pod justification. This is one model family (Qwen-2.5-7B-Instruct), one tooling stack, one dataset pipeline; arms and seeds time-share the same GPU. None of the multi-pod exemption clauses ((a) >8 H100s, (b) data-parallel disjoint hosts, (c) different model weights / CUDA / per-pod state) applies.
Artifacts
experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/— 9 LoRA adapters, each uploaded to HF Hub (per-adapter incremental).experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json— per-adapter raw completions + scorer output (loose-match per cell, R_B|A, R_B|not_A, R_B_loose, mean completion length, end-of-completion-position rate for marker_B, per-cell denominators).experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl— on-policy data-gen output, one set per seed (shared across arms within a seed).experiments/369/summary.json— flattened table: arm × seed × persona × {R_B|A, R_B|not_A, R_B_loose, denom_A, denom_not_A, mean_completion_length, eoc_position_rate_marker_B}; pooled-across-seeds rows; CI-A and CI-B for the librarian-software_engineer × C2 primary cell and the police_officer × C2 / data_scientist × C2 bystander cells; bootstrap drop rate per CI; verdict label.experiments/369/base_model_floor.json— Phase-0 probe (loose-match rates for both markers on base Qwen-2.5-7B-Instruct).- WandB run group
exp369-binding-vs-template— 9 runs, one per adapter, training loss curves + eval-pass metrics. - Sagan progress POSTs at: pre-data-gen, post-data-gen, after each adapter trains, after each eval, before upload.
Verification
- Phase-0 floor: loose-match rate < 1% for marker_A and marker_B on base model (cached after first run; abort if violated).
- C control: re-trained C arm shows recipient
R_B_loose< 2% on all 3 seeds (otherwise the eval rig changed — drift abort). - T re-train drift: seed-42 recipient
R_B|Aon re-trained T lands within [−7%, +54%] of #354's 23.5% (drift abort gate; see Kill Criterion §7). - Denominator floor for primary verdict: pooled
denom_A_C2on recipient ≥ 40 across the 3 seeds; if not, verdict downgrades to inconclusive. - Bootstrap drop-rate: ≤ 10% drop rate on CI-A and CI-B; if exceeded, verdict is inconclusive on that CI.
- Length-inflation guard: recipient C2 mean completion length within 25% relative of T; if not, verdict is qualified (not aborted).
- CI-A vs CI-B agreement: if the two CIs straddle different verdict bands, verdict is downgraded to inconclusive (the wider CI's band takes precedence; the disagreement is reported).
- Estimator agreement: conditional-of-pooled and mean-of-conditionals must land in the same verdict band; disagreement → inconclusive.
- Spec-vs-plan sanity:
run_experiment_369.pyemits tosummary.jsonexactly the columns and CIs specified in Artifacts; runner inspects post-run.
Risks and Red Team
Mechanism risk: shape-template emits marker_B differently than expected. If the donor learns "end with marker_B" rather than "marker_A cues marker_B," the recipient's marker_B might land at end-of-completion even without marker_A nearby — high R_B_loose with low R_B|A because the recipient's marker_A also fires at low rates. The kill criterion's template-without-A leg (recipient R_B_loose ≥ 10% with CI lower bound > 5%) is verdict-bearing, not a footnote.
Mechanism risk: "binding-confirmed" doesn't fully imply chunk-binding. C2 removes both the marker_A↔marker_B pairing AND the donor's marker_A exposure. A silent C2 is therefore consistent with (i) genuine chunk-binding (A keys B) and (ii) "donor needs marker_A exposure somewhere to propagate marker_B." The matched-A-marginal-unpaired follow-up (donor sees A-only and B-only in separate completions) settles this — explicitly queued, scope-expanding, not in this run. The clean-result phrases the binding-confirmed verdict as "paired-or-marginal-A-required to propagate B" rather than over-claiming pure chunk-binding.
Mechanism risk: recipient's own <A> answer training overrides any shape-template suffix. Even if the donor learns ... <B> as a turn-end suffix, the recipient's training (<A> answer ending naturally) might dominate, suppressing marker_B emission and producing a binding-style null even when shape-template is the true donor mechanism. Instrumented: the run reports donor and recipient R_B_loose AND end-of-completion position rate on C2 side-by-side. If donor C2 R_B_loose ≥ 50% but recipient C2 R_B_loose < 5% with matched mean completion length, that is evidence for "recipient template dominates" rather than "no shape-template existed." The clean-result body compares donor vs. recipient C2 R_B_loose explicitly.
Denominator-collapse risk on C2. Under C2, the donor never sees marker_A, so the recipient's marker_A representation is trained as in T (recipient row identical) but no longer reinforced by the paired donor presentation. Recipient denom_A_C2 could be smaller than #354's denom_A_T = 81. The verification gate enforces denom_A_C2 ≥ 40 (pooled across 3 seeds; #354 saw 81 in one seed, so 40 is a conservative pooled floor) before a binding-confirmed verdict can fire; below that, the verdict downgrades to "inconclusive — denom-fragile."
Seed variance risk. #354's single-seed cluster CI on T spanned [8.9%, 39.8%]. Three seeds pooled give 780 completions per (arm × persona) cell vs. 260 for one seed, but cluster CI tightening is limited by question-axis correlation. CI-B (seed-stratified) is the honest seed-variance estimate; with n=3 seeds it will be wider than CI-A. The kill criterion uses whichever is wider, so the "inconclusive" middle band may be load-bearing — that is a feature.
Pipeline-drift risk: the codebase moved between #354 (Nov 2025) and now (May 2026). Re-training T and C in the same run is a deliberate hedge. Only seed-42 T comparison is drift-diagnostic; new seeds (1337, 2024) cannot disambiguate drift directly. CI comparability: the seed-42-only single-cell CI computed under this run's procedure (cluster bootstrap on questions, B=10,000) is not numerically identical to #354's reported [8.9%, 39.8%] (B=2,000, different bootstrap estimator). The drift check is on the point estimate vs. 23.5%; CI width is reported for context, not gated.
Eval-rig sensitivity to length-inflation. #354 could only refute length-alone indirectly (via C's 0%). This run instruments mean completion length per (persona × arm × seed) with the explicit 25%-relative length-inflation discriminator in the Kill Criterion. Verdict qualifies (not aborts) on length drift.
Bystander interpretation ambiguity. Under #354's T, police_officer fired marker_B at 54.3% (n_A=35, cluster CI [16.0%, 89.7%]). Bystander cells under C2 could be noisier because the donor's now-decoupled marker_A representation may suppress bystander marker_A firing. The kill criterion's bystander override now requires pooled denom_A_C2 ≥ 30, pooled R_B|A > 20%, AND CI lower bound > 10%. Override is bystander-specific to police_officer and data_scientist (the two non-trivial leakers under #354 T); zelthari_scholar is descriptive only. Bystander leak may operate via a different mechanism than recipient transfer (e.g., distribution-shift priors); a fired override should be read as "C2 leaks somewhere → shape-template lives in the donor" rather than "the specific recipient transfer mechanism is template-driven." Clean-result must state that nuance.
Multiple-comparisons / family-wise error. Primary verdict is one cell-test (recipient × C2). Bystander override adds 2 pre-registered comparisons (police_officer, data_scientist × C2); at α = 0.05 per test, FWER ~10%. CI-lower-bound and denom_A guards already tighten this; the override is asymmetric (can only flip binding-confirmed → shape-template, never the other direction). Residual FWER risk acceptable. The full 11-persona × 3-arm spectrum reported in the result is exploratory / descriptive only and cannot retroactively become verdict-bearing.
Capacity / supply risk. H100 80GB SXM availability on RunPod Secure Cloud is occasionally tight. The substitution policy lets the provisioner swap to H200 or A100-SXM 80GB (both ≥ 80GB VRAM) on the same single-pod, single-GPU shape. Cloud type may relax to COMMUNITY if SECURE is unavailable. GPU-family substitution adds at most ~30–40% walltime; total cost stays under $25.
Cost overrun risk. Budget has ~15% headroom over the estimated 7.0 H100-hr spend. If the pod runs past 10 H100-hr, mid-run progress POSTs surface the slip; the runner can stop the pod and treat partial completion as a partial-run artifact (any of the 9 adapters that finished are usable in 1-seed or 2-seed-pooled form).
Critique loop notes. Loops run: 1. Six critic agents (paired Claude + Codex × {methodology, statistics, alternatives}) ran in parallel. Verdicts: methodology Claude=pass / Codex=needs_targeted_fix; statistics both=needs_targeted_fix; alternatives both=needs_targeted_fix. Unioned scope-preserving blockers folded into this revision: (a) C2 donor-coherence gate contradiction fixed (marginal R_B_loose / R_B|not_A on C2, not R_B|A); (b) primary statistic defined as conditional-of-pooled with mean-of-conditionals as sensitivity; (c) two pre-registered cluster CIs (CI-A question-only, CI-B seed-stratified two-level) with verdict using the wider; (d) bystander override now requires denom_A ≥ 30 + point estimate + CI lower bound; (e) recipient denom_A_C2 ≥ 40 floor for binding-confirmed verdict; (f) template-without-A leg added using recipient R_B_loose; (g) length-inflation discriminator (25% relative threshold qualifies verdict); (h) bootstrap drop-rate guard (>10% → inconclusive); (i) bystander pool corrected from "5" to the actual 3 untrained-bystander personas; (j) "2×2" framing relabeled "three-arm sweep"; (k) interpretation narrowing for binding-confirmed verdict (paired-or-marginal-A-required vs. pure chunk-binding). No Codex fallback. Reconciler not invoked for methodology lens because all Codex findings were scope-preserving and overlapped with statistics-lens blockers already requiring fixes — folded unilaterally; recorded here for audit. Consistency-checker returned WARN (one documentation gap on CI comparability vs. #354's reported interval; folded into the pipeline-drift risk paragraph). Follow-ups intentionally not folded (queued, not gates): matched-A-marginal-unpaired arm (the only scope-expanding finding, would tighten binding-vs-marginal-A distinction); non-fixed marker_B position arm (separately queued; the existing follow-up on #354 next-steps); recipient-only baseline control (correlated training-distribution suffix); LoRA-rank check; temperature sweep.
Likely Clean Result
HTML clean-result on experiments.body rendered at /e/experiment/369, following docs/clean-result-guidelines.md:
- TL;DR (open): 3–4 bullets — what I wanted to find out, the verdict (binding-confirmed / shape-template confirmed / inconclusive / drift-shelved), the headline number (pooled recipient C2
R_B|Awith the wider 95% CI), and the most important caveat (binding-confirmed means "paired-or-marginal-A-required," not pure chunk-binding; the matched-A-marginal-unpaired follow-up is queued). - Primary plot: grouped bar chart, x-axis = arm (T, C, C2), y-axis = pooled recipient
R_B|A(%) with 95% CI error bars (wider of CI-A and CI-B). Plain-English axis labels; no LaTeX. SVG<title>hover tooltips on each bar (denom_A, n completions, per-seed values). - Experimental design dropdown (closed): what changed vs. #354, the three arms, seeds, bystander-override pre-registration, length-inflation guard, primary vs. sensitivity statistic, CI-A vs CI-B definitions, and drift-abort criteria. Donor vs. recipient C2
R_B_looseside-by-side. Full 11-persona × 3-arm spectrum table (descriptive). Bystander cells (police_officer, data_scientist × C2) explicitly called out. - Voice: "I" not "we"; no standing caveats, no abandoned-metric references, no separate background/methodology h2.
If the verdict is binding-confirmed, the next-step card points to the matched-A-marginal-unpaired follow-up. If shape-template, it points to the non-fixed-position follow-up. If inconclusive or drift-shelved, it points to the diagnostic that would unblock the verdict.
Approval Checklist
- Goal: binding-vs-template — distinguish whether marker_A↔marker_B pairing or end-of-completion shape-template drives the #354 propagation.
- Hypothesis: binding → recipient silent on C2; template → recipient (or designated bystanders) emit marker_B on C2.
- Prediction: quantitative thresholds with CI bounds, pre-registered.
- Kill criterion: seven-clause verdict tree with concrete cutoffs (recipient < 5% & upper CI < 10% → binding; > 10% & lower CI > 5% → template; bystander override > 20% & lower CI > 10%; denom_A floor; bootstrap drop rate; length-inflation guard; drift abort).
- Compute and hardware: 1× H100 80GB SXM, single pod, sequential, ~7.0 H100-hr spent / 8.0 H100-hr budgeted. Estimated cost: ~$22 USD (H100 SXM @ $2.69/hr × 8.0 hr + transient storage; A100/H200 substitutions stay under $25). Rate may drift — see Compute section.
- Artifacts: 9 adapters, 9 eval JSONs, 3 per-seed data-gen dumps,
summary.jsonwith pooled CIs and verdict, base_model_floor.json, WandB group, progress POSTs. - Verification: Phase-0 floor, C control gate, T re-train drift gate, denominator floor, bootstrap drop-rate, length-inflation qualifier, CI agreement, estimator agreement, spec-vs-plan sanity.
- Risks: 10 risks enumerated, each with mitigation or instrumentation — mechanism (3), denom collapse, seed variance, pipeline drift, length inflation, bystander ambiguity, FWER, capacity, cost.
- Likely clean-result: TL;DR + primary plot (3-arm grouped bar with wider-of-CI error bars) + experimental-design dropdown; verdict-driven next-step card.
- Runpod-spec matches plan: single pod, single H100, 8 hr est, runs
scripts/run_experiment_369.py, substitution policy permits H200/A100-SXM 80GB and COMMUNITY cloud, no GPU-count scaling. - Tenant-agnostic check: EPS-specific experiment, runs in EPS repo on RunPod; no Sagan schema or API additions. ✓
{
"name": "exp369-binding-vs-template",
"gpuType": "H100",
"gpuCount": 1,
"volumeGb": 100,
"containerDiskGb": 100,
"cloudType": "SECURE",
"estimatedMinutes": 480,
"dockerArgs": "bash -lc 'cd /workspace/explore-persona-space && uv run python scripts/run_experiment_369.py'",
"config": {
"command": "Three-arm sweep T/C/C2 × seeds {42,1337,2024} on librarian→software_engineer pair under EOS-masked LoRA SFT recipe inherited from #354. Trains 9 adapters, evals 11 personas × 26 questions × 5 completions per adapter, writes summary.json with pooled CIs (CI-A question-cluster, CI-B seed-stratified) and verdict label.",
"artifacts": [
"experiments/369/summary.json",
"experiments/369/base_model_floor.json",
"experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/",
"experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json",
"experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl"
]
},
"substitution_policy": {
"gpuType": { "allowed": ["H100", "H200", "A100-SXM"], "min_vram_gb": 80 },
"gpuCount": { "min": 1, "max": 1 },
"cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
"dataCenterId":{ "allowed": "any", "prefer": ["US-CA-2", "EU-RO-1"] },
"account": { "allowed": ["team", "personal"], "prefer": "team" },
"volumeGb": { "min": 100 },
"containerDiskGb": { "min": 100 }
},
"consolidation": {
"may_merge_pods": true,
"merge_target_max_gpus_per_pod": 8
}
}
Select any text in the plan to leave an anchored comment, then click Revise to send all open comments back to Sagan.
Issue history
19 eventsAwaiting Clarifications2026-05-14 09:001 item
- state changedplanning -> awaiting_clarifications
Claude produced clarifying questions; awaiting owner answers.
Metadata
{ "planLen": 1927, "sections": 1, "agentRunId": "17919abc-5d1d-4257-a8f1-ec346c363db6" }
Clarifying2026-05-14 09:031 item
- state changedawaiting_clarifications -> clarifying
Owner re-dispatched the planner from awaiting_clarifications.
Plan Pending2026-05-14 09:231 item
- state changedclarifying -> plan_pending
Experiment plan is ready for owner approval.
Metadata
{ "agentRunId": "0c382d32-0089-4463-8107-b7e29b82e410", "structuredSections": 11 }
Plan Pending2026-05-14 09:231 item
- approval requested? -> plan_pending
Experiment plan approval requested.
Metadata
{ "agentRunId": "0c382d32-0089-4463-8107-b7e29b82e410", "approvalRequestId": "2f95173f-16a7-4b5c-8324-cd15964808a3" }
Running2026-05-14 09:251 item
- state changedplan_pending -> running
Moved on Pipeline board to running.
Implementing2026-05-14 09:251 item
- state changedrunning -> implementing
Orchestrator b2016e32 queued to implement and dispatch.
Approved2026-05-14 09:251 item
- state changedimplementing -> approved
Approved from Pipeline board after moving to running.
Implementing2026-05-14 09:261 item
- state changedapproved -> implementing
experiment implementation2026-05-14 09:261 item
- epm:experiment-implementation
Spawning experiment-implementer to write scripts/run_experiment_369.py and push a branch on the EPS repo. Inherits #354's recipe; adds C2 arm + 3 seeds + per-seed cache + new statistics.
Metadata
{ "marker_type": "epm:experiment-implementation" }
Code Reviewing2026-05-14 09:391 item
- state changedimplementing -> code_reviewing
code review2026-05-14 09:441 item
- epm:code-review
pass: plan-fidelity audit clean; ruff+ruff-format pass; 20/20 unit tests pass; module imports + module-level asserts fire (40 data Qs, 26 eval Qs, disjoint, templates exact). Conditional-of-pooled estimator = sum(AB)/sum(A) confirmed (0.545 on imbalanced toy). CI-A uses np.random.default_rng(43), CI-B uses default_rng(44), both isolated from training/eval seeds. CI-B outer rng.integers(0, 3, size=3) correctly resamples seeds with replacement. Bootstrap loops count via for i in idx (additive), not via dict overwrite. Length-inflation 25%, cross-seed >15pp gap, donor coherence gates, denom_A_C2>=40 floor, drops>=10% gate (max of CI-A/CI-B drops / 10000) all wired. HF path adapters/exp369_{arm}_seed{seed} via TrainLoraConfig default repo superkaiba1/explore-persona-space. 4 implementer-flagged items reviewed: (1) CI-A/CI-B x_offset +0.12 leave-as-is (stays within 0.5-wide bar, black/gray colors distinguish), (2) drop_rate divisor hardcoded 10000 OK for production (--bootstrap-B 10000 default) but trivially fragile in debug runs, (3) bystander override hard-flips to template matches plan §KillCriterion 'can flip binding-confirmed to template', (4) test re-import of module is wasteful (0.69s runtime) but not load-bearing. Minor non-blocking notes: donor-coherence gates are recorded in sanity_gates not abort-raise (build_summary runs post-eval so abort is moot); SamplingParams.seed not passed (only LLM seed) — inherited from #354 eval rig, plan inherits #354. No targeted fixes required.
Metadata
{ "round": 1, "verdict": "pass", "reviewer": "claude", "marker_type": "epm:code-review", "review_pair": "code_review" }
code review codex2026-05-14 09:441 item
- epm:code-review-codex
pass: statistical correctness verified (cluster bootstrap RNG isolation, per-completion conditional-of-pooled estimator, seeded resampling); lint+tests clean (20/20 pass); plumbing (post_progress milestones, HF Hub paths superkaiba1/explore-persona-space/adapters/exp369_{arm}_seed{seed}, raw_completions.json mandatory, WandB exp369, _safe filter, dataset-shape assertions, Phase-0 once at seed 42, per-seed cache reused across arms) all match plan; four implementer-flagged items reviewed and acceptable | fixes: none
Metadata
{ "marker_type": "epm:code-review-codex" }
Testing2026-05-14 09:451 item
- state changedcode_reviewing -> testing
test verdict2026-05-14 09:451 item
- epm:test-verdict
reviewer pair Step-4 lint+pytest verdict: PASS. ruff check + ruff format check + pytest tests/test_exp369_kill_criterion.py (20/20 in 0.75s) green on both reviewers' runs at commit 85a70fd9 on branch issue-369.
Metadata
{ "marker_type": "epm:test-verdict" }
Running2026-05-14 09:461 item
- state changedtesting -> running
Queued2026-05-14 09:471 item
- state changedrunning -> queued
RunPod pod dispatched; waiting for runtime.
Running2026-05-14 09:481 item
- state changedqueued -> running
RunPod pod is running.
Plan Pending2026-05-14 22:141 item
- state changedrunning -> plan_pending
Experiment plan is ready for owner approval.
Metadata
{ "agentRunId": "0ab1df81-5601-42e3-a06d-a7fd8b26c4a6", "structuredSections": 11 }
Plan Pending2026-05-14 22:141 item
- approval requested? -> plan_pending
Experiment plan approval requested.
Metadata
{ "agentRunId": "0ab1df81-5601-42e3-a06d-a7fd8b26c4a6", "approvalRequestId": "f6e4accc-8cc3-4fe7-8c05-077f3aaf7117" }
Follow-up from clean result 747c9e7a (#354 paired-marker propagation, librarian → software_engineer).
Why this is needed. The current 2-arm design (T: donor sees <A> answer <B>; C: donor sees <A> answer) cleanly rules out one alternative: marker_B does not propagate to the recipient without donor exposure to marker_B (C arm gives 0 / 260 across all personas). It also rules out marker_B-as-generic-turn-end-suffix: P(marker_B | NOT marker_A) ≈ 0% across all personas (3 emissions in 1,265 not-marker_A completions, vs 172 / 295 when marker_A fired).
What it does not rule out: the LoRA may have learned the full completion shape <A> {answer} <B> as a template, and emits both markers together as parts of the same template rather than <A> literally triggering <B>. Under that interpretation, marker_A and marker_B co-fire because of shape, not association.
The proposed control: C2.
- C2 (donor sees
<B>only): train donor on{answer} <B>(marker_B at end-of-completion, no marker_A anywhere). Recipient training stays the same as T and C (<A> answer). - Predictions:
- If chunk-binding / marker_A keys marker_B: C2 should leave the recipient (and bystanders) at ~0% marker_B emission, because marker_A and marker_B were never paired in donor training.
- If shape-template or marker_B leaking alone: C2 should still produce non-zero marker_B emission on the recipient (and likely on bystanders), because marker_B was trained on the donor and gets picked up as an end-of-completion suffix.
The full 2x2 the result of which would settle the binding question:
donor sees <A> | donor doesn't see <A> | |
|---|---|---|
donor sees <B> | T (current, 23.5%) | C2 (new) |
donor doesn't see <B> | C (current, 0%) | (uninteresting baseline) |
Minimal design.
- Same recipe as #354 T/C arms (same model, LoRA hyperparameters, contrastive negatives, eval rig, seed pool).
- Add a single C2 adapter for the librarian → software_engineer pair on seed 42.
- Optionally extend to a 3-seed replication for T/C/C2 to also resolve the seed-stability concern flagged in the current MODERATE confidence label.
- Eval metric: per-persona marker_B count, per-persona conditional R_B|A, cluster 95% CI via questions-cluster bootstrap (same statistic as #354).
Cost.
- C2 alone, 1 seed: ~0.7 H100-hours (1 adapter trained + eval).
- T/C/C2 × 3 seeds = 9 adapters: ~6.5 H100-hours, fits in a single overnight RunPod sitting.
Related follow-ups.
- Non-fixed-marker-position arm (already in TL;DR next steps) is a looser version of the same disambiguation. C2 is cleaner — minimal-cut, drops the marker_A/marker_B pairing in donor training but keeps everything else fixed.
- Temperature sweep todo (aa180963) is orthogonal — it varies the on-policy answer distribution, not the marker scaffold.