Skip to content
Sagan
experiment

Run 17919abc

Moved from planning (after classifier) on the Pipeline board. Draft the next experiment plan for the scoped experiment. Use the scoped experiment record as the source of truth for title and scope, and produce a plan that can be reviewed and approved. Do not rename, retitle, or otherwise mutate the scoped issue/experiment.

Statuscompleted4 events · latest 1327h 17m ago

Plan

Goal

Settle the chunk-binding vs. shape-template question left open by #354. #354 showed that under EOS-masked LoRA SFT on Qwen-2.5-7B-Instruct, training a donor (librarian) on the paired chunk `<A> answer <B>` propagates marker_B to a recipient (software_engineer) at non-trivial rates (recipient `R_B|A` = 23.5% on seed 42; control C with no marker_B in donor training = 0%). Two compatible mechanisms remain: (a) **chunk-binding** — marker_A literally keys marker_B because the two were paired in donor training; (b) **shape-template** — the donor LoRA learned the full `<A> answer <B>` end-of-completion template and emits both as parts of the same surface pattern, with marker_A and marker_B co-firing because of position rather than association. This experiment runs a three-arm sweep T (paired `<A> answer <B>`), C (`<A> answer`, no marker_B), C2 (`answer <B>`, no marker_A) at three seeds (42, 1337, 2024) on the librarian→software_engineer pair. C2 keeps marker_B in the donor's training distribution at the same end-of-completion position as T but breaks the marker_A↔marker_B pairing — a silent recipient under C2 is consistent with chunk-binding (or, more strictly, "donor needs marker_A exposure to propagate marker_B"); a non-silent recipient is consistent with shape-template.

Hypothesis

Under chunk-binding, the recipient's marker_B emission rate after marker_A on C2 collapses to near-baseline; under shape-template, the recipient continues to emit marker_B at non-trivial rates because the donor learned `... <B>` as a turn-end suffix that the recipient picks up through shared LoRA conditioning. The recipient is the primary axis; police_officer and data_scientist (the two non-trivial bystander leakers from #354 T) provide a secondary mechanism check.

Prediction

- **If chunk-binding (paired-or-marginal-A-required) is the operative mechanism:** pooled recipient `R_B|A` on C2 is < 5% with the wider of (CI-A question-cluster, CI-B seed-stratified) 95% upper bound < 10%. Bystander C2 cells are also near baseline. - **If shape-template is operative:** pooled recipient `R_B|A` on C2 is > 10% with 95% lower bound > 5%; OR a bystander override fires (police_officer or data_scientist pooled `R_B|A` on C2 > 20% with denom_A ≥ 30 and 95% lower bound > 10%). - **T re-train sanity:** seed-42 recipient `R_B|A` on T re-trained in this run lands within roughly [−7%, +54%] of #354's 23.5% (2× #354's per-seed cluster CI half-width). C re-train on all seeds remains 0%.

Kill Criterion

Single pre-registered verdict tree, evaluated on pooled recipient `R_B|A` over the 3 seeds (librarian→software_engineer), using the **wider** of two 95% cluster bootstrap CIs (CI-A: questions-only cluster; CI-B: seed-stratified two-level). Primary statistic is **conditional-of-pooled** (pooled numerator / pooled denominator across seeds); mean-of-conditionals (per-seed `R_B|A` averaged across seeds) is reported as a sensitivity check. 1. **Binding-confirmed (chunk-binding, shape-template falsified):** recipient C2 pooled `R_B|A` < 5% AND 95% upper CI < 10% AND recipient pooled `denom_A_C2` ≥ 40 AND no bystander override fires (see 3). Interpretation: "paired-or-marginal-A-required to propagate marker_B." Not pure chunk-binding (see Risks for why). 2. **Shape-template confirmed (chunk-binding falsified) — recipient route:** recipient C2 pooled `R_B|A` > 10% AND 95% lower CI > 5% AND recipient pooled `denom_A_C2` ≥ 40. 3. **Shape-template confirmed — bystander override route:** EITHER police_officer OR data_scientist pooled `R_B|A` on C2 > 20% AND 95% lower CI > 10% AND pooled `denom_A_C2` ≥ 30. Override fires asymmetrically — it can only flip a recipient-silent finding from "binding-confirmed" to "shape-template lives in donor / mechanism mixed," never the other direction. 4. **Length-inflation guard (qualifier, not nullifier):** if recipient mean completion length on C2 differs from T by > 25% relative, the verdict is **qualified** (reported with caveat) because the marker_B substring-match opportunity is confounded by completion length. 5. **Template-without-A leg (descriptive):** report recipient `R_B_loose` (unconditional marker_B emission, no marker_A required) on C2 alongside `R_B|A`. If recipient C2 `R_B_loose` ≥ 10% with 95% lower CI > 5%, flag "template-without-A" pattern even if `R_B|A` is in the inconclusive band. 6. **Inconclusive band:** anything else (e.g., recipient C2 5% ≤ `R_B|A` ≤ 10%, or `denom_A_C2` < 40, or bootstrap drop rate > 10%, or T drift > 2× #354 per-seed CI half-width). Report numbers and queue the matched-A-marginal-unpaired follow-up. 7. **Drift abort gate:** if both T and C re-trains land outside their pre-registered drift bands (T: seed-42 `R_B|A` outside [−7%, +54%] of 23.5%; C: any seed shows `R_B|A` > 2%), the C2 verdict is **shelved** pending pipeline root-cause; the run is reported as a drift detection, not a binding-vs-template verdict.

Experimental Setup

**Inheritance from #354.** Reuse the EOS-masked LoRA SFT recipe from `scripts/run_issue354_eos_masked.py`: Qwen-2.5-7B-Instruct base, LoRA rank/alpha as #354, recipient's `tokenizer.eos_token_id` masked from cross-entropy labels, donor + 4 contrastive-negative rows pass through with EOS in loss, contrastive negative pool unchanged, eval question set unchanged (20 in-distribution + 6 OOD = 26 unique, disjoint from data-generation questions), 11 evaluation personas unchanged. Pair restricted to `pair2_librarian_swe` (librarian donor → software_engineer recipient), single-variable change relative to #354. **Three arms.** - **T (paired chunk):** donor sees `<A> answer <B>` (identical to #354's T). - **C (no-marker_B control):** donor sees `<A> answer` (identical to #354's C). - **C2 (no-marker_A control):** donor sees `answer <B>` with marker_B at the same end-of-completion position as T, no marker_A anywhere in donor's training rows. Recipient training is identical to T (start-only marker_A coupling); contrastive negatives are unchanged. **Single-variable cut: only the marker_A column on the donor changes between T and C2.** **Seeds.** {42, 1337, 2024}. Seed 42 to align with #354; 1337 and 2024 to add genuine seed variance. Total adapters trained: 3 arms × 3 seeds = 9. **Per-seed on-policy data generation.** Each seed regenerates donor + recipient + 4 contrastive-negative training rows at temperature 0.7 over the data-generation question set (11 personas × 40 questions × 5 completions = 2,200 generations). Same vLLM rig as #354; deterministic given the seed. **Phase-0 base-model floor (one-time).** Run #354's base-model probe (11 personas × 3 sampled questions × 1 completion = 33 generations) to confirm loose-match rates for both markers < 1% on the base model. Cached after first run. **Eval (per adapter).** vLLM completions over the 26-question eval set × 11 personas × 5 completions per cell = 1,430 per adapter. Loose-match scorer reports both `marker_A` and `marker_B` substring matches, conditional rates `R_B|A` and `R_B|not_A`, marginal rates `R_B_loose`, mean completion length per cell, end-of-completion position rate for marker_B (was marker_B the last non-whitespace span?), and pooled denominators for `R_B|A` cells. Identical eval-rig invocation across arms — no eval-rig variance. **Primary statistic.** - **Estimator:** pooled recipient `R_B|A` on librarian→software_engineer cell, computed as conditional-of-pooled (sum-marker_B-and-marker_A / sum-marker_A) across all 3 seeds for each arm. - **CI-A (question-cluster bootstrap):** B = 10,000 resamples, cluster unit = question, resample seeds and completions within each question independently, drop-resample on cells with `denom_A` = 0 (drop rate must be ≤ 10% or verdict is inconclusive). - **CI-B (seed-stratified two-level cluster bootstrap):** B = 10,000, top-level cluster = seed (with replacement, n=3), within-seed cluster = question. CI-B is the honest seed-variance interval and is expected to be wider than CI-A given n=3 seeds; the verdict uses `max(CI-A_width, CI-B_width)`. - **Verdict CI:** the wider of CI-A and CI-B. - **Sensitivity:** mean-of-conditionals (average per-seed `R_B|A`) reported alongside the primary statistic. If the two estimators disagree on which verdict band they fall into, the result is downgraded to inconclusive. **Secondary statistic (bystander override).** Pooled `R_B|A` on police_officer and data_scientist under C2, with denom_A ≥ 30, point estimate > 20%, and 95% CI lower bound > 10% (using CI-A; bystander cells are too sparse for CI-B). Pre-registered override; the other 8 personas in the spectrum are exploratory / descriptive only. **Comparability gates (consistency-checker recommendation).** Drift diagnostics: re-trained seed-42 T's recipient point estimate vs. #354's 23.5%; re-trained C's recipient on all seeds must be < 2%. Pipeline drift triggers the drift abort gate above. CI width is not numerically comparable to #354 (different bootstrap parameters; see Risks); the drift gate is on the point estimate only. **Inherited from #354 to make this single-variable.** All of: base model, LoRA hyperparameters, EOS-masking recipe, contrastive-negative pool, contrastive-negative count (4), recipient persona coupling pattern (start-only marker_A), data-gen question pool, eval question set (26), eval personas (11), eval temperature, loose-match scorer, completions-per-cell (5). Anything that drifts here is a pipeline drift, not an intended variable.

Compute and Hardware

**Workload sizing.** 9 LoRA adapters (3 arms × 3 seeds) + 3 per-seed on-policy data generations + 1 Phase-0 base-model probe (one-time, cached) + 9 vLLM eval passes. Reference from #354 (same recipe, same eval shape): ~0.7 H100-hr per adapter inclusive of training (~30 min) and eval (~12 min). On-policy data generation: ~7 min/seed on 1×H100 with vLLM (2,200 generations at T=0.7). **Single-pod, single-GPU sequential walltime on 1×H100 80GB SXM:** | Stage | Walltime | |---|---| | Cold bootstrap (uv sync, weights, .env) | ~12 min one-time | | Phase-0 base-model probe | ~2 min one-time | | On-policy data gen (3 seeds × ~7 min) | ~21 min | | Train + eval (9 adapters × ~42 min) | ~6.3 hr | | HF/WandB upload, write `summary.json` | ~5 min | | **Total** | **~7.0 H100-hr** | Budget **8.0 H100-hr** of pod time for ~15% headroom. Sequential is simpler than parallelizing across two GPUs (which would halve walltime but doubles dispatch and capacity-allocation risk and would also need lock-coordinated WandB writes). **USD cost estimate.** Rate used: H100 80GB SXM on RunPod Secure Cloud on-demand = **$2.69/hr** (May 2026; treat as guidance — may drift). Storage: existing network volume `eps-warm-cache-us-ca-2` (if available, see runpod-spec) avoids re-downloading Qwen-7B weights and uv wheel cache; otherwise ~50 GB transient at $0.10/GB-month for ~0.3 day = ~$0.05 storage. - Compute: 8.0 H100-hr × $2.69/hr × 1 GPU × 1 pod = **$21.52** - Storage: ~$0.05 (transient) or ~$0 (warm-cache volume) - **Total: ~$22 USD** (rounded to two significant figures) Substitution policy (below) allows H200 or A100-SXM 80GB swaps if H100 is supply-constrained; A100-SXM at $1.49/hr would *lower* cost but extend walltime by ~30%, so total stays within ~$25 in the worst case. **Single-pod justification.** This is one model family (Qwen-2.5-7B-Instruct), one tooling stack, one dataset pipeline; arms and seeds time-share the same GPU. None of the multi-pod exemption clauses ((a) >8 H100s, (b) data-parallel disjoint hosts, (c) different model weights / CUDA / per-pod state) applies.

Artifacts

- `experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/` — 9 LoRA adapters, each uploaded to HF Hub (per-adapter incremental). - `experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json` — per-adapter raw completions + scorer output (loose-match per cell, R_B|A, R_B|not_A, R_B_loose, mean completion length, end-of-completion-position rate for marker_B, per-cell denominators). - `experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl` — on-policy data-gen output, one set per seed (shared across arms within a seed). - `experiments/369/summary.json` — flattened table: arm × seed × persona × {R_B|A, R_B|not_A, R_B_loose, denom_A, denom_not_A, mean_completion_length, eoc_position_rate_marker_B}; pooled-across-seeds rows; CI-A and CI-B for the librarian-software_engineer × C2 primary cell and the police_officer × C2 / data_scientist × C2 bystander cells; bootstrap drop rate per CI; verdict label. - `experiments/369/base_model_floor.json` — Phase-0 probe (loose-match rates for both markers on base Qwen-2.5-7B-Instruct). - WandB run group `exp369-binding-vs-template` — 9 runs, one per adapter, training loss curves + eval-pass metrics. - Sagan progress POSTs at: pre-data-gen, post-data-gen, after each adapter trains, after each eval, before upload.

Verification

1. **Phase-0 floor:** loose-match rate < 1% for marker_A and marker_B on base model (cached after first run; abort if violated). 2. **C control:** re-trained C arm shows recipient `R_B_loose` < 2% on all 3 seeds (otherwise the eval rig changed — drift abort). 3. **T re-train drift:** seed-42 recipient `R_B|A` on re-trained T lands within [−7%, +54%] of #354's 23.5% (drift abort gate; see Kill Criterion §7). 4. **Denominator floor for primary verdict:** pooled `denom_A_C2` on recipient ≥ 40 across the 3 seeds; if not, verdict downgrades to inconclusive. 5. **Bootstrap drop-rate:** ≤ 10% drop rate on CI-A and CI-B; if exceeded, verdict is inconclusive on that CI. 6. **Length-inflation guard:** recipient C2 mean completion length within 25% relative of T; if not, verdict is qualified (not aborted). 7. **CI-A vs CI-B agreement:** if the two CIs straddle different verdict bands, verdict is downgraded to inconclusive (the wider CI's band takes precedence; the disagreement is reported). 8. **Estimator agreement:** conditional-of-pooled and mean-of-conditionals must land in the same verdict band; disagreement → inconclusive. 9. **Spec-vs-plan sanity:** `run_experiment_369.py` emits to `summary.json` exactly the columns and CIs specified in Artifacts; runner inspects post-run.

Risks and Red Team

**Mechanism risk: shape-template emits marker_B differently than expected.** If the donor learns "end with marker_B" rather than "marker_A cues marker_B," the recipient's marker_B might land at end-of-completion even without marker_A nearby — high `R_B_loose` with low `R_B|A` because the recipient's marker_A also fires at low rates. The kill criterion's template-without-A leg (recipient `R_B_loose` ≥ 10% with CI lower bound > 5%) is verdict-bearing, not a footnote. **Mechanism risk: "binding-confirmed" doesn't fully imply chunk-binding.** C2 removes both the marker_A↔marker_B pairing AND the donor's marker_A exposure. A silent C2 is therefore consistent with (i) genuine chunk-binding (A keys B) and (ii) "donor needs marker_A exposure somewhere to propagate marker_B." The matched-A-marginal-unpaired follow-up (donor sees A-only and B-only in separate completions) settles this — explicitly queued, scope-expanding, not in this run. The clean-result phrases the binding-confirmed verdict as "paired-or-marginal-A-required to propagate B" rather than over-claiming pure chunk-binding. **Mechanism risk: recipient's own `<A> answer` training overrides any shape-template suffix.** Even if the donor learns `... <B>` as a turn-end suffix, the recipient's training (`<A> answer` ending naturally) might dominate, suppressing marker_B emission and producing a binding-style null even when shape-template is the true donor mechanism. Instrumented: the run reports donor and recipient `R_B_loose` AND end-of-completion position rate on C2 side-by-side. If donor C2 `R_B_loose` ≥ 50% but recipient C2 `R_B_loose` < 5% with matched mean completion length, that is evidence for "recipient template dominates" rather than "no shape-template existed." The clean-result body compares donor vs. recipient C2 `R_B_loose` explicitly. **Denominator-collapse risk on C2.** Under C2, the donor never sees marker_A, so the recipient's marker_A representation is trained as in T (recipient row identical) but no longer reinforced by the paired donor presentation. Recipient `denom_A_C2` could be smaller than #354's `denom_A_T = 81`. The verification gate enforces `denom_A_C2 ≥ 40` (pooled across 3 seeds; #354 saw 81 in one seed, so 40 is a conservative pooled floor) before a binding-confirmed verdict can fire; below that, the verdict downgrades to "inconclusive — denom-fragile." **Seed variance risk.** #354's single-seed cluster CI on T spanned [8.9%, 39.8%]. Three seeds pooled give 780 completions per (arm × persona) cell vs. 260 for one seed, but cluster CI tightening is limited by question-axis correlation. CI-B (seed-stratified) is the honest seed-variance estimate; with n=3 seeds it will be wider than CI-A. The kill criterion uses whichever is wider, so the "inconclusive" middle band may be load-bearing — that is a feature. **Pipeline-drift risk: the codebase moved between #354 (Nov 2025) and now (May 2026).** Re-training T and C in the same run is a deliberate hedge. Only seed-42 T comparison is drift-diagnostic; new seeds (1337, 2024) cannot disambiguate drift directly. **CI comparability:** the seed-42-only single-cell CI computed under this run's procedure (cluster bootstrap on questions, B=10,000) is not numerically identical to #354's reported [8.9%, 39.8%] (B=2,000, different bootstrap estimator). The drift check is on the point estimate vs. 23.5%; CI width is reported for context, not gated. **Eval-rig sensitivity to length-inflation.** #354 could only refute length-alone indirectly (via C's 0%). This run instruments mean completion length per (persona × arm × seed) with the explicit 25%-relative length-inflation discriminator in the Kill Criterion. Verdict qualifies (not aborts) on length drift. **Bystander interpretation ambiguity.** Under #354's T, police_officer fired marker_B at 54.3% (n_A=35, cluster CI [16.0%, 89.7%]). Bystander cells under C2 could be noisier because the donor's now-decoupled marker_A representation may suppress bystander marker_A firing. The kill criterion's bystander override now requires pooled `denom_A_C2` ≥ 30, pooled `R_B|A` > 20%, AND CI lower bound > 10%. Override is bystander-specific to police_officer and data_scientist (the two non-trivial leakers under #354 T); zelthari_scholar is descriptive only. Bystander leak may operate via a different mechanism than recipient transfer (e.g., distribution-shift priors); a fired override should be read as "C2 leaks somewhere → shape-template lives in the donor" rather than "the specific recipient transfer mechanism is template-driven." Clean-result must state that nuance. **Multiple-comparisons / family-wise error.** Primary verdict is one cell-test (recipient × C2). Bystander override adds 2 pre-registered comparisons (police_officer, data_scientist × C2); at α = 0.05 per test, FWER ~10%. CI-lower-bound and denom_A guards already tighten this; the override is asymmetric (can only flip binding-confirmed → shape-template, never the other direction). Residual FWER risk acceptable. The full 11-persona × 3-arm spectrum reported in the result is exploratory / descriptive only and cannot retroactively become verdict-bearing. **Capacity / supply risk.** H100 80GB SXM availability on RunPod Secure Cloud is occasionally tight. The substitution policy lets the provisioner swap to H200 or A100-SXM 80GB (both ≥ 80GB VRAM) on the same single-pod, single-GPU shape. Cloud type may relax to COMMUNITY if SECURE is unavailable. GPU-family substitution adds at most ~30–40% walltime; total cost stays under $25. **Cost overrun risk.** Budget has ~15% headroom over the estimated 7.0 H100-hr spend. If the pod runs past 10 H100-hr, mid-run progress POSTs surface the slip; the runner can stop the pod and treat partial completion as a partial-run artifact (any of the 9 adapters that finished are usable in 1-seed or 2-seed-pooled form). **Critique loop notes.** Loops run: 1. Six critic agents (paired Claude + Codex × {methodology, statistics, alternatives}) ran in parallel. Verdicts: methodology Claude=pass / Codex=needs_targeted_fix; statistics both=needs_targeted_fix; alternatives both=needs_targeted_fix. Unioned scope-preserving blockers folded into this revision: (a) C2 donor-coherence gate contradiction fixed (marginal `R_B_loose` / `R_B|not_A` on C2, not `R_B|A`); (b) primary statistic defined as conditional-of-pooled with mean-of-conditionals as sensitivity; (c) two pre-registered cluster CIs (CI-A question-only, CI-B seed-stratified two-level) with verdict using the wider; (d) bystander override now requires denom_A ≥ 30 + point estimate + CI lower bound; (e) recipient `denom_A_C2` ≥ 40 floor for binding-confirmed verdict; (f) template-without-A leg added using recipient `R_B_loose`; (g) length-inflation discriminator (25% relative threshold qualifies verdict); (h) bootstrap drop-rate guard (>10% → inconclusive); (i) bystander pool corrected from "5" to the actual 3 untrained-bystander personas; (j) "2×2" framing relabeled "three-arm sweep"; (k) interpretation narrowing for binding-confirmed verdict (paired-or-marginal-A-required vs. pure chunk-binding). No Codex fallback. Reconciler not invoked for methodology lens because all Codex findings were scope-preserving and overlapped with statistics-lens blockers already requiring fixes — folded unilaterally; recorded here for audit. Consistency-checker returned WARN (one documentation gap on CI comparability vs. #354's reported interval; folded into the pipeline-drift risk paragraph). Follow-ups intentionally not folded (queued, not gates): matched-A-marginal-unpaired arm (the only scope-expanding finding, would tighten binding-vs-marginal-A distinction); non-fixed marker_B position arm (separately queued; the existing follow-up on #354 next-steps); recipient-only baseline control (correlated training-distribution suffix); LoRA-rank check; temperature sweep.

Likely Clean Result

HTML clean-result on `experiments.body` rendered at `/e/experiment/369`, following `docs/clean-result-guidelines.md`: - **TL;DR (open):** 3–4 bullets — what I wanted to find out, the verdict (binding-confirmed / shape-template confirmed / inconclusive / drift-shelved), the headline number (pooled recipient C2 `R_B|A` with the wider 95% CI), and the most important caveat (binding-confirmed means "paired-or-marginal-A-required," not pure chunk-binding; the matched-A-marginal-unpaired follow-up is queued). - **Primary plot:** grouped bar chart, x-axis = arm (T, C, C2), y-axis = pooled recipient `R_B|A` (%) with 95% CI error bars (wider of CI-A and CI-B). Plain-English axis labels; no LaTeX. SVG `<title>` hover tooltips on each bar (denom_A, n completions, per-seed values). - **Experimental design dropdown (closed):** what changed vs. #354, the three arms, seeds, bystander-override pre-registration, length-inflation guard, primary vs. sensitivity statistic, CI-A vs CI-B definitions, and drift-abort criteria. Donor vs. recipient C2 `R_B_loose` side-by-side. Full 11-persona × 3-arm spectrum table (descriptive). Bystander cells (police_officer, data_scientist × C2) explicitly called out. - Voice: "I" not "we"; no standing caveats, no abandoned-metric references, no separate background/methodology h2. If the verdict is binding-confirmed, the next-step card points to the matched-A-marginal-unpaired follow-up. If shape-template, it points to the non-fixed-position follow-up. If inconclusive or drift-shelved, it points to the diagnostic that would unblock the verdict.

Approval Checklist

- [x] **Goal:** binding-vs-template — distinguish whether marker_A↔marker_B pairing or end-of-completion shape-template drives the #354 propagation. - [x] **Hypothesis:** binding → recipient silent on C2; template → recipient (or designated bystanders) emit marker_B on C2. - [x] **Prediction:** quantitative thresholds with CI bounds, pre-registered. - [x] **Kill criterion:** seven-clause verdict tree with concrete cutoffs (recipient < 5% & upper CI < 10% → binding; > 10% & lower CI > 5% → template; bystander override > 20% & lower CI > 10%; denom_A floor; bootstrap drop rate; length-inflation guard; drift abort). - [x] **Compute and hardware:** 1× H100 80GB SXM, single pod, sequential, ~7.0 H100-hr spent / 8.0 H100-hr budgeted. **Estimated cost: ~$22 USD** (H100 SXM @ $2.69/hr × 8.0 hr + transient storage; A100/H200 substitutions stay under $25). Rate may drift — see Compute section. - [x] **Artifacts:** 9 adapters, 9 eval JSONs, 3 per-seed data-gen dumps, `summary.json` with pooled CIs and verdict, base_model_floor.json, WandB group, progress POSTs. - [x] **Verification:** Phase-0 floor, C control gate, T re-train drift gate, denominator floor, bootstrap drop-rate, length-inflation qualifier, CI agreement, estimator agreement, spec-vs-plan sanity. - [x] **Risks:** 10 risks enumerated, each with mitigation or instrumentation — mechanism (3), denom collapse, seed variance, pipeline drift, length inflation, bystander ambiguity, FWER, capacity, cost. - [x] **Likely clean-result:** TL;DR + primary plot (3-arm grouped bar with wider-of-CI error bars) + experimental-design dropdown; verdict-driven next-step card. - [x] **Runpod-spec matches plan:** single pod, single H100, 8 hr est, runs `scripts/run_experiment_369.py`, substitution policy permits H200/A100-SXM 80GB and COMMUNITY cloud, no GPU-count scaling. - [x] **Tenant-agnostic check:** EPS-specific experiment, runs in EPS repo on RunPod; no Sagan schema or API additions. ✓ ```runpod-spec { "name": "exp369-binding-vs-template", "gpuType": "H100", "gpuCount": 1, "volumeGb": 100, "containerDiskGb": 100, "cloudType": "SECURE", "estimatedMinutes": 480, "dockerArgs": "bash -lc 'cd /workspace/explore-persona-space && uv run python scripts/run_experiment_369.py'", "config": { "command": "Three-arm sweep T/C/C2 × seeds {42,1337,2024} on librarian→software_engineer pair under EOS-masked LoRA SFT recipe inherited from #354. Trains 9 adapters, evals 11 personas × 26 questions × 5 completions per adapter, writes summary.json with pooled CIs (CI-A question-cluster, CI-B seed-stratified) and verdict label.", "artifacts": [ "experiments/369/summary.json", "experiments/369/base_model_floor.json", "experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/", "experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json", "experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl" ] }, "substitution_policy": { "gpuType": { "allowed": ["H100", "H200", "A100-SXM"], "min_vram_gb": 80 }, "gpuCount": { "min": 1, "max": 1 }, "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" }, "dataCenterId":{ "allowed": "any", "prefer": ["US-CA-2", "EU-RO-1"] }, "account": { "allowed": ["team", "personal"], "prefer": "team" }, "volumeGb": { "min": 100 }, "containerDiskGb": { "min": 100 } }, "consolidation": { "may_merge_pods": true, "merge_target_max_gpus_per_pod": 8 } } ```

The experiment script run_experiment_369.py is already in place in EPS. I'll now produce the final plan.

Goal

Settle the chunk-binding vs. shape-template question left open by #354. #354 showed that under EOS-masked LoRA SFT on Qwen-2.5-7B-Instruct, training a donor (librarian) on the paired chunk <A> answer <B> propagates marker_B to a recipient (software_engineer) at non-trivial rates (recipient R_B|A = 23.5% on seed 42; control C with no marker_B in donor training = 0%). Two compatible mechanisms remain: (a) chunk-binding — marker_A literally keys marker_B because the two were paired in donor training; (b) shape-template — the donor LoRA learned the full <A> answer <B> end-of-completion template and emits both as parts of the same surface pattern, with marker_A and marker_B co-firing because of position rather than association.

This experiment runs a three-arm sweep T (paired <A> answer <B>), C (<A> answer, no marker_B), C2 (answer <B>, no marker_A) at three seeds (42, 1337, 2024) on the librarian→software_engineer pair. C2 keeps marker_B in the donor's training distribution at the same end-of-completion position as T but breaks the marker_A↔marker_B pairing — a silent recipient under C2 is consistent with chunk-binding (or, more strictly, "donor needs marker_A exposure to propagate marker_B"); a non-silent recipient is consistent with shape-template.

Hypothesis

Under chunk-binding, the recipient's marker_B emission rate after marker_A on C2 collapses to near-baseline; under shape-template, the recipient continues to emit marker_B at non-trivial rates because the donor learned ... <B> as a turn-end suffix that the recipient picks up through shared LoRA conditioning. The recipient is the primary axis; police_officer and data_scientist (the two non-trivial bystander leakers from #354 T) provide a secondary mechanism check.

Prediction

  • If chunk-binding (paired-or-marginal-A-required) is the operative mechanism: pooled recipient R_B|A on C2 is < 5% with the wider of (CI-A question-cluster, CI-B seed-stratified) 95% upper bound < 10%. Bystander C2 cells are also near baseline.
  • If shape-template is operative: pooled recipient R_B|A on C2 is > 10% with 95% lower bound > 5%; OR a bystander override fires (police_officer or data_scientist pooled R_B|A on C2 > 20% with denom_A ≥ 30 and 95% lower bound > 10%).
  • T re-train sanity: seed-42 recipient R_B|A on T re-trained in this run lands within roughly [−7%, +54%] of #354's 23.5% (2× #354's per-seed cluster CI half-width). C re-train on all seeds remains 0%.

Kill Criterion

Single pre-registered verdict tree, evaluated on pooled recipient R_B|A over the 3 seeds (librarian→software_engineer), using the wider of two 95% cluster bootstrap CIs (CI-A: questions-only cluster; CI-B: seed-stratified two-level). Primary statistic is conditional-of-pooled (pooled numerator / pooled denominator across seeds); mean-of-conditionals (per-seed R_B|A averaged across seeds) is reported as a sensitivity check.

  1. Binding-confirmed (chunk-binding, shape-template falsified): recipient C2 pooled R_B|A < 5% AND 95% upper CI < 10% AND recipient pooled denom_A_C2 ≥ 40 AND no bystander override fires (see 3). Interpretation: "paired-or-marginal-A-required to propagate marker_B." Not pure chunk-binding (see Risks for why).
  2. Shape-template confirmed (chunk-binding falsified) — recipient route: recipient C2 pooled R_B|A > 10% AND 95% lower CI > 5% AND recipient pooled denom_A_C2 ≥ 40.
  3. Shape-template confirmed — bystander override route: EITHER police_officer OR data_scientist pooled R_B|A on C2 > 20% AND 95% lower CI > 10% AND pooled denom_A_C2 ≥ 30. Override fires asymmetrically — it can only flip a recipient-silent finding from "binding-confirmed" to "shape-template lives in donor / mechanism mixed," never the other direction.
  4. Length-inflation guard (qualifier, not nullifier): if recipient mean completion length on C2 differs from T by > 25% relative, the verdict is qualified (reported with caveat) because the marker_B substring-match opportunity is confounded by completion length.
  5. Template-without-A leg (descriptive): report recipient R_B_loose (unconditional marker_B emission, no marker_A required) on C2 alongside R_B|A. If recipient C2 R_B_loose ≥ 10% with 95% lower CI > 5%, flag "template-without-A" pattern even if R_B|A is in the inconclusive band.
  6. Inconclusive band: anything else (e.g., recipient C2 5% ≤ R_B|A ≤ 10%, or denom_A_C2 < 40, or bootstrap drop rate > 10%, or T drift > 2× #354 per-seed CI half-width). Report numbers and queue the matched-A-marginal-unpaired follow-up.
  7. Drift abort gate: if both T and C re-trains land outside their pre-registered drift bands (T: seed-42 R_B|A outside [−7%, +54%] of 23.5%; C: any seed shows R_B|A > 2%), the C2 verdict is shelved pending pipeline root-cause; the run is reported as a drift detection, not a binding-vs-template verdict.

Experimental Setup

Inheritance from #354. Reuse the EOS-masked LoRA SFT recipe from scripts/run_issue354_eos_masked.py: Qwen-2.5-7B-Instruct base, LoRA rank/alpha as #354, recipient's tokenizer.eos_token_id masked from cross-entropy labels, donor + 4 contrastive-negative rows pass through with EOS in loss, contrastive negative pool unchanged, eval question set unchanged (20 in-distribution + 6 OOD = 26 unique, disjoint from data-generation questions), 11 evaluation personas unchanged. Pair restricted to pair2_librarian_swe (librarian donor → software_engineer recipient), single-variable change relative to #354.

Three arms.

  • T (paired chunk): donor sees <A> answer <B> (identical to #354's T).
  • C (no-marker_B control): donor sees <A> answer (identical to #354's C).
  • C2 (no-marker_A control): donor sees answer <B> with marker_B at the same end-of-completion position as T, no marker_A anywhere in donor's training rows. Recipient training is identical to T (start-only marker_A coupling); contrastive negatives are unchanged. Single-variable cut: only the marker_A column on the donor changes between T and C2.

Seeds. {42, 1337, 2024}. Seed 42 to align with #354; 1337 and 2024 to add genuine seed variance. Total adapters trained: 3 arms × 3 seeds = 9.

Per-seed on-policy data generation. Each seed regenerates donor + recipient + 4 contrastive-negative training rows at temperature 0.7 over the data-generation question set (11 personas × 40 questions × 5 completions = 2,200 generations). Same vLLM rig as #354; deterministic given the seed.

Phase-0 base-model floor (one-time). Run #354's base-model probe (11 personas × 3 sampled questions × 1 completion = 33 generations) to confirm loose-match rates for both markers < 1% on the base model. Cached after first run.

Eval (per adapter). vLLM completions over the 26-question eval set × 11 personas × 5 completions per cell = 1,430 per adapter. Loose-match scorer reports both marker_A and marker_B substring matches, conditional rates R_B|A and R_B|not_A, marginal rates R_B_loose, mean completion length per cell, end-of-completion position rate for marker_B (was marker_B the last non-whitespace span?), and pooled denominators for R_B|A cells. Identical eval-rig invocation across arms — no eval-rig variance.

Primary statistic.

  • Estimator: pooled recipient R_B|A on librarian→software_engineer cell, computed as conditional-of-pooled (sum-marker_B-and-marker_A / sum-marker_A) across all 3 seeds for each arm.
  • CI-A (question-cluster bootstrap): B = 10,000 resamples, cluster unit = question, resample seeds and completions within each question independently, drop-resample on cells with denom_A = 0 (drop rate must be ≤ 10% or verdict is inconclusive).
  • CI-B (seed-stratified two-level cluster bootstrap): B = 10,000, top-level cluster = seed (with replacement, n=3), within-seed cluster = question. CI-B is the honest seed-variance interval and is expected to be wider than CI-A given n=3 seeds; the verdict uses max(CI-A_width, CI-B_width).
  • Verdict CI: the wider of CI-A and CI-B.
  • Sensitivity: mean-of-conditionals (average per-seed R_B|A) reported alongside the primary statistic. If the two estimators disagree on which verdict band they fall into, the result is downgraded to inconclusive.

Secondary statistic (bystander override). Pooled R_B|A on police_officer and data_scientist under C2, with denom_A ≥ 30, point estimate > 20%, and 95% CI lower bound > 10% (using CI-A; bystander cells are too sparse for CI-B). Pre-registered override; the other 8 personas in the spectrum are exploratory / descriptive only.

Comparability gates (consistency-checker recommendation). Drift diagnostics: re-trained seed-42 T's recipient point estimate vs. #354's 23.5%; re-trained C's recipient on all seeds must be < 2%. Pipeline drift triggers the drift abort gate above. CI width is not numerically comparable to #354 (different bootstrap parameters; see Risks); the drift gate is on the point estimate only.

Inherited from #354 to make this single-variable. All of: base model, LoRA hyperparameters, EOS-masking recipe, contrastive-negative pool, contrastive-negative count (4), recipient persona coupling pattern (start-only marker_A), data-gen question pool, eval question set (26), eval personas (11), eval temperature, loose-match scorer, completions-per-cell (5). Anything that drifts here is a pipeline drift, not an intended variable.

Compute and Hardware

Workload sizing. 9 LoRA adapters (3 arms × 3 seeds) + 3 per-seed on-policy data generations + 1 Phase-0 base-model probe (one-time, cached) + 9 vLLM eval passes. Reference from #354 (same recipe, same eval shape): ~0.7 H100-hr per adapter inclusive of training (~30 min) and eval (~12 min). On-policy data generation: ~7 min/seed on 1×H100 with vLLM (2,200 generations at T=0.7).

Single-pod, single-GPU sequential walltime on 1×H100 80GB SXM:

StageWalltime
Cold bootstrap (uv sync, weights, .env)~12 min one-time
Phase-0 base-model probe~2 min one-time
On-policy data gen (3 seeds × ~7 min)~21 min
Train + eval (9 adapters × ~42 min)~6.3 hr
HF/WandB upload, write summary.json~5 min
Total~7.0 H100-hr

Budget 8.0 H100-hr of pod time for ~15% headroom. Sequential is simpler than parallelizing across two GPUs (which would halve walltime but doubles dispatch and capacity-allocation risk and would also need lock-coordinated WandB writes).

USD cost estimate. Rate used: H100 80GB SXM on RunPod Secure Cloud on-demand = $2.69/hr (May 2026; treat as guidance — may drift). Storage: existing network volume eps-warm-cache-us-ca-2 (if available, see runpod-spec) avoids re-downloading Qwen-7B weights and uv wheel cache; otherwise ~50 GB transient at $0.10/GB-month for ~0.3 day = ~$0.05 storage.

  • Compute: 8.0 H100-hr × $2.69/hr × 1 GPU × 1 pod = $21.52
  • Storage: ~$0.05 (transient) or ~$0 (warm-cache volume)
  • Total: ~$22 USD (rounded to two significant figures)

Substitution policy (below) allows H200 or A100-SXM 80GB swaps if H100 is supply-constrained; A100-SXM at $1.49/hr would lower cost but extend walltime by ~30%, so total stays within ~$25 in the worst case.

Single-pod justification. This is one model family (Qwen-2.5-7B-Instruct), one tooling stack, one dataset pipeline; arms and seeds time-share the same GPU. None of the multi-pod exemption clauses ((a) >8 H100s, (b) data-parallel disjoint hosts, (c) different model weights / CUDA / per-pod state) applies.

Artifacts

  • experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/ — 9 LoRA adapters, each uploaded to HF Hub (per-adapter incremental).
  • experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json — per-adapter raw completions + scorer output (loose-match per cell, R_B|A, R_B|not_A, R_B_loose, mean completion length, end-of-completion-position rate for marker_B, per-cell denominators).
  • experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl — on-policy data-gen output, one set per seed (shared across arms within a seed).
  • experiments/369/summary.json — flattened table: arm × seed × persona × {R_B|A, R_B|not_A, R_B_loose, denom_A, denom_not_A, mean_completion_length, eoc_position_rate_marker_B}; pooled-across-seeds rows; CI-A and CI-B for the librarian-software_engineer × C2 primary cell and the police_officer × C2 / data_scientist × C2 bystander cells; bootstrap drop rate per CI; verdict label.
  • experiments/369/base_model_floor.json — Phase-0 probe (loose-match rates for both markers on base Qwen-2.5-7B-Instruct).
  • WandB run group exp369-binding-vs-template — 9 runs, one per adapter, training loss curves + eval-pass metrics.
  • Sagan progress POSTs at: pre-data-gen, post-data-gen, after each adapter trains, after each eval, before upload.

Verification

  1. Phase-0 floor: loose-match rate < 1% for marker_A and marker_B on base model (cached after first run; abort if violated).
  2. C control: re-trained C arm shows recipient R_B_loose < 2% on all 3 seeds (otherwise the eval rig changed — drift abort).
  3. T re-train drift: seed-42 recipient R_B|A on re-trained T lands within [−7%, +54%] of #354's 23.5% (drift abort gate; see Kill Criterion §7).
  4. Denominator floor for primary verdict: pooled denom_A_C2 on recipient ≥ 40 across the 3 seeds; if not, verdict downgrades to inconclusive.
  5. Bootstrap drop-rate: ≤ 10% drop rate on CI-A and CI-B; if exceeded, verdict is inconclusive on that CI.
  6. Length-inflation guard: recipient C2 mean completion length within 25% relative of T; if not, verdict is qualified (not aborted).
  7. CI-A vs CI-B agreement: if the two CIs straddle different verdict bands, verdict is downgraded to inconclusive (the wider CI's band takes precedence; the disagreement is reported).
  8. Estimator agreement: conditional-of-pooled and mean-of-conditionals must land in the same verdict band; disagreement → inconclusive.
  9. Spec-vs-plan sanity: run_experiment_369.py emits to summary.json exactly the columns and CIs specified in Artifacts; runner inspects post-run.

Risks and Red Team

Mechanism risk: shape-template emits marker_B differently than expected. If the donor learns "end with marker_B" rather than "marker_A cues marker_B," the recipient's marker_B might land at end-of-completion even without marker_A nearby — high R_B_loose with low R_B|A because the recipient's marker_A also fires at low rates. The kill criterion's template-without-A leg (recipient R_B_loose ≥ 10% with CI lower bound > 5%) is verdict-bearing, not a footnote.

Mechanism risk: "binding-confirmed" doesn't fully imply chunk-binding. C2 removes both the marker_A↔marker_B pairing AND the donor's marker_A exposure. A silent C2 is therefore consistent with (i) genuine chunk-binding (A keys B) and (ii) "donor needs marker_A exposure somewhere to propagate marker_B." The matched-A-marginal-unpaired follow-up (donor sees A-only and B-only in separate completions) settles this — explicitly queued, scope-expanding, not in this run. The clean-result phrases the binding-confirmed verdict as "paired-or-marginal-A-required to propagate B" rather than over-claiming pure chunk-binding.

Mechanism risk: recipient's own <A> answer training overrides any shape-template suffix. Even if the donor learns ... <B> as a turn-end suffix, the recipient's training (<A> answer ending naturally) might dominate, suppressing marker_B emission and producing a binding-style null even when shape-template is the true donor mechanism. Instrumented: the run reports donor and recipient R_B_loose AND end-of-completion position rate on C2 side-by-side. If donor C2 R_B_loose ≥ 50% but recipient C2 R_B_loose < 5% with matched mean completion length, that is evidence for "recipient template dominates" rather than "no shape-template existed." The clean-result body compares donor vs. recipient C2 R_B_loose explicitly.

Denominator-collapse risk on C2. Under C2, the donor never sees marker_A, so the recipient's marker_A representation is trained as in T (recipient row identical) but no longer reinforced by the paired donor presentation. Recipient denom_A_C2 could be smaller than #354's denom_A_T = 81. The verification gate enforces denom_A_C2 ≥ 40 (pooled across 3 seeds; #354 saw 81 in one seed, so 40 is a conservative pooled floor) before a binding-confirmed verdict can fire; below that, the verdict downgrades to "inconclusive — denom-fragile."

Seed variance risk. #354's single-seed cluster CI on T spanned [8.9%, 39.8%]. Three seeds pooled give 780 completions per (arm × persona) cell vs. 260 for one seed, but cluster CI tightening is limited by question-axis correlation. CI-B (seed-stratified) is the honest seed-variance estimate; with n=3 seeds it will be wider than CI-A. The kill criterion uses whichever is wider, so the "inconclusive" middle band may be load-bearing — that is a feature.

Pipeline-drift risk: the codebase moved between #354 (Nov 2025) and now (May 2026). Re-training T and C in the same run is a deliberate hedge. Only seed-42 T comparison is drift-diagnostic; new seeds (1337, 2024) cannot disambiguate drift directly. CI comparability: the seed-42-only single-cell CI computed under this run's procedure (cluster bootstrap on questions, B=10,000) is not numerically identical to #354's reported [8.9%, 39.8%] (B=2,000, different bootstrap estimator). The drift check is on the point estimate vs. 23.5%; CI width is reported for context, not gated.

Eval-rig sensitivity to length-inflation. #354 could only refute length-alone indirectly (via C's 0%). This run instruments mean completion length per (persona × arm × seed) with the explicit 25%-relative length-inflation discriminator in the Kill Criterion. Verdict qualifies (not aborts) on length drift.

Bystander interpretation ambiguity. Under #354's T, police_officer fired marker_B at 54.3% (n_A=35, cluster CI [16.0%, 89.7%]). Bystander cells under C2 could be noisier because the donor's now-decoupled marker_A representation may suppress bystander marker_A firing. The kill criterion's bystander override now requires pooled denom_A_C2 ≥ 30, pooled R_B|A > 20%, AND CI lower bound > 10%. Override is bystander-specific to police_officer and data_scientist (the two non-trivial leakers under #354 T); zelthari_scholar is descriptive only. Bystander leak may operate via a different mechanism than recipient transfer (e.g., distribution-shift priors); a fired override should be read as "C2 leaks somewhere → shape-template lives in the donor" rather than "the specific recipient transfer mechanism is template-driven." Clean-result must state that nuance.

Multiple-comparisons / family-wise error. Primary verdict is one cell-test (recipient × C2). Bystander override adds 2 pre-registered comparisons (police_officer, data_scientist × C2); at α = 0.05 per test, FWER ~10%. CI-lower-bound and denom_A guards already tighten this; the override is asymmetric (can only flip binding-confirmed → shape-template, never the other direction). Residual FWER risk acceptable. The full 11-persona × 3-arm spectrum reported in the result is exploratory / descriptive only and cannot retroactively become verdict-bearing.

Capacity / supply risk. H100 80GB SXM availability on RunPod Secure Cloud is occasionally tight. The substitution policy lets the provisioner swap to H200 or A100-SXM 80GB (both ≥ 80GB VRAM) on the same single-pod, single-GPU shape. Cloud type may relax to COMMUNITY if SECURE is unavailable. GPU-family substitution adds at most ~30–40% walltime; total cost stays under $25.

Cost overrun risk. Budget has ~15% headroom over the estimated 7.0 H100-hr spend. If the pod runs past 10 H100-hr, mid-run progress POSTs surface the slip; the runner can stop the pod and treat partial completion as a partial-run artifact (any of the 9 adapters that finished are usable in 1-seed or 2-seed-pooled form).

Critique loop notes. Loops run: 1. Six critic agents (paired Claude + Codex × {methodology, statistics, alternatives}) ran in parallel. Verdicts: methodology Claude=pass / Codex=needs_targeted_fix; statistics both=needs_targeted_fix; alternatives both=needs_targeted_fix. Unioned scope-preserving blockers folded into this revision: (a) C2 donor-coherence gate contradiction fixed (marginal R_B_loose / R_B|not_A on C2, not R_B|A); (b) primary statistic defined as conditional-of-pooled with mean-of-conditionals as sensitivity; (c) two pre-registered cluster CIs (CI-A question-only, CI-B seed-stratified two-level) with verdict using the wider; (d) bystander override now requires denom_A ≥ 30 + point estimate + CI lower bound; (e) recipient denom_A_C2 ≥ 40 floor for binding-confirmed verdict; (f) template-without-A leg added using recipient R_B_loose; (g) length-inflation discriminator (25% relative threshold qualifies verdict); (h) bootstrap drop-rate guard (>10% → inconclusive); (i) bystander pool corrected from "5" to the actual 3 untrained-bystander personas; (j) "2×2" framing relabeled "three-arm sweep"; (k) interpretation narrowing for binding-confirmed verdict (paired-or-marginal-A-required vs. pure chunk-binding). No Codex fallback. Reconciler not invoked for methodology lens because all Codex findings were scope-preserving and overlapped with statistics-lens blockers already requiring fixes — folded unilaterally; recorded here for audit. Consistency-checker returned WARN (one documentation gap on CI comparability vs. #354's reported interval; folded into the pipeline-drift risk paragraph). Follow-ups intentionally not folded (queued, not gates): matched-A-marginal-unpaired arm (the only scope-expanding finding, would tighten binding-vs-marginal-A distinction); non-fixed marker_B position arm (separately queued; the existing follow-up on #354 next-steps); recipient-only baseline control (correlated training-distribution suffix); LoRA-rank check; temperature sweep.

Likely Clean Result

HTML clean-result on experiments.body rendered at /e/experiment/369, following docs/clean-result-guidelines.md:

  • TL;DR (open): 3–4 bullets — what I wanted to find out, the verdict (binding-confirmed / shape-template confirmed / inconclusive / drift-shelved), the headline number (pooled recipient C2 R_B|A with the wider 95% CI), and the most important caveat (binding-confirmed means "paired-or-marginal-A-required," not pure chunk-binding; the matched-A-marginal-unpaired follow-up is queued).
  • Primary plot: grouped bar chart, x-axis = arm (T, C, C2), y-axis = pooled recipient R_B|A (%) with 95% CI error bars (wider of CI-A and CI-B). Plain-English axis labels; no LaTeX. SVG <title> hover tooltips on each bar (denom_A, n completions, per-seed values).
  • Experimental design dropdown (closed): what changed vs. #354, the three arms, seeds, bystander-override pre-registration, length-inflation guard, primary vs. sensitivity statistic, CI-A vs CI-B definitions, and drift-abort criteria. Donor vs. recipient C2 R_B_loose side-by-side. Full 11-persona × 3-arm spectrum table (descriptive). Bystander cells (police_officer, data_scientist × C2) explicitly called out.
  • Voice: "I" not "we"; no standing caveats, no abandoned-metric references, no separate background/methodology h2.

If the verdict is binding-confirmed, the next-step card points to the matched-A-marginal-unpaired follow-up. If shape-template, it points to the non-fixed-position follow-up. If inconclusive or drift-shelved, it points to the diagnostic that would unblock the verdict.

Approval Checklist

  • Goal: binding-vs-template — distinguish whether marker_A↔marker_B pairing or end-of-completion shape-template drives the #354 propagation.
  • Hypothesis: binding → recipient silent on C2; template → recipient (or designated bystanders) emit marker_B on C2.
  • Prediction: quantitative thresholds with CI bounds, pre-registered.
  • Kill criterion: seven-clause verdict tree with concrete cutoffs (recipient < 5% & upper CI < 10% → binding; > 10% & lower CI > 5% → template; bystander override > 20% & lower CI > 10%; denom_A floor; bootstrap drop rate; length-inflation guard; drift abort).
  • Compute and hardware: 1× H100 80GB SXM, single pod, sequential, ~7.0 H100-hr spent / 8.0 H100-hr budgeted. Estimated cost: ~$22 USD (H100 SXM @ $2.69/hr × 8.0 hr + transient storage; A100/H200 substitutions stay under $25). Rate may drift — see Compute section.
  • Artifacts: 9 adapters, 9 eval JSONs, 3 per-seed data-gen dumps, summary.json with pooled CIs and verdict, base_model_floor.json, WandB group, progress POSTs.
  • Verification: Phase-0 floor, C control gate, T re-train drift gate, denominator floor, bootstrap drop-rate, length-inflation qualifier, CI agreement, estimator agreement, spec-vs-plan sanity.
  • Risks: 10 risks enumerated, each with mitigation or instrumentation — mechanism (3), denom collapse, seed variance, pipeline drift, length inflation, bystander ambiguity, FWER, capacity, cost.
  • Likely clean-result: TL;DR + primary plot (3-arm grouped bar with wider-of-CI error bars) + experimental-design dropdown; verdict-driven next-step card.
  • Runpod-spec matches plan: single pod, single H100, 8 hr est, runs scripts/run_experiment_369.py, substitution policy permits H200/A100-SXM 80GB and COMMUNITY cloud, no GPU-count scaling.
  • Tenant-agnostic check: EPS-specific experiment, runs in EPS repo on RunPod; no Sagan schema or API additions. ✓
{
  "name": "exp369-binding-vs-template",
  "gpuType": "H100",
  "gpuCount": 1,
  "volumeGb": 100,
  "containerDiskGb": 100,
  "cloudType": "SECURE",
  "estimatedMinutes": 480,
  "dockerArgs": "bash -lc 'cd /workspace/explore-persona-space && uv run python scripts/run_experiment_369.py'",
  "config": {
    "command": "Three-arm sweep T/C/C2 × seeds {42,1337,2024} on librarian→software_engineer pair under EOS-masked LoRA SFT recipe inherited from #354. Trains 9 adapters, evals 11 personas × 26 questions × 5 completions per adapter, writes summary.json with pooled CIs (CI-A question-cluster, CI-B seed-stratified) and verdict label.",
    "artifacts": [
      "experiments/369/summary.json",
      "experiments/369/base_model_floor.json",
      "experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/",
      "experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json",
      "experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl"
    ]
  },
  "substitution_policy": {
    "gpuType":     { "allowed": ["H100", "H200", "A100-SXM"], "min_vram_gb": 80 },
    "gpuCount":    { "min": 1, "max": 1 },
    "cloudType":   { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
    "dataCenterId":{ "allowed": "any", "prefer": ["US-CA-2", "EU-RO-1"] },
    "account":     { "allowed": ["team", "personal"], "prefer": "team" },
    "volumeGb":        { "min": 100 },
    "containerDiskGb": { "min": 100 }
  },
  "consolidation": {
    "may_merge_pods": true,
    "merge_target_max_gpus_per_pod": 8
  }
}

Events

9:00:20 AMstartedagent
kind=experiment
9:00:54 AMassistant_textagent
## Clarifying questions

1. **Scope: minimal vs full replication.** The body floats two options — C2 alone on seed 42 (~0.7 H100-hr), or T/C/C2 × 3 seeds (~6.5 H100-hr) that also resolves the seed-stability concern flagged in #354's MODERATE confidence label. Which scope should I plan for? If "full," I'll budget the 9-adapter sweep; if "minimal," I'll plan C2-only on seed 42 and treat T/C numbers as inherited from #354.
[TEXTBOX:scope]

2. **Kill criterion for the binding-vs-template question.** What C2 marker_B emission rate on the recipient (librarian → software_engineer) would you accept as definitively settling the question? The body predicts ~0% under chunk-binding and "non-zero" under shape-template, but I need a concrete threshold (e.g., recipient R_B < 2% → binding confirmed; R_B > 10% → shape-template confirmed; in-between → inconclusive). Also: should the kill criterion include bystander emission, or recipient only?
[TEXTBOX:kill-criterion]

3. **Re-train T/C in the same run, or trust #354's numbers?** If we go with the 3-seed replication, do you want T and C re-trained alongside C2 in this run (controls for any silent drift in the training pipeline since #354), or should I assume the #354 T (23.5%) and C (0%) numbers are stable and only train C2 + the two new seeds for T/C? The former adds ~1.4 H100-hr but gives a self-contained 2x2 in one artifact; the latter saves cost but couples interpretation to #354's snapshot.
[TEXTBOX:reuse-tc]

4. **C2 marker placement.** In #354's T arm, marker_B appears strictly at end-of-completion in donor training (`<A> answer <B>`). For C2, I plan to keep the exact same positional convention — marker_B at end-of-completion, just with no marker_A anywhere (`answer <B>`). Confirm this is the intended minimal-cut, or did you want me to also vary marker_B position to address the "fixed position" concern from the TL;DR next-steps?
[TEXTBOX:marker-placement]

<<<DONE>>>
9:00:54 AMsdk_resultagent
success
9:00:54 AMawaiting_clarificationsagent
Claude produced clarifying questions instead of a full plan.

Discussion

Comments

No comments yet.

Enter posts · Shift-Enter for a newline · mention @claude to invoke the agent