Skip to content
Sagan

Experiment#369

Add C2 control arm (donor sees marker_B without marker_A) to disambiguate paired-marker binding from marker_B leaking alone

Recovery neededfailed

Clarifications

Plan

Review and approve →

The experiment script run_experiment_369.py is already in place in EPS. I'll now produce the final plan.

Goal

Settle the chunk-binding vs. shape-template question left open by #354. #354 showed that under EOS-masked LoRA SFT on Qwen-2.5-7B-Instruct, training a donor (librarian) on the paired chunk <A> answer <B> propagates marker_B to a recipient (software_engineer) at non-trivial rates (recipient R_B|A = 23.5% on seed 42; control C with no marker_B in donor training = 0%). Two compatible mechanisms remain: (a) chunk-binding — marker_A literally keys marker_B because the two were paired in donor training; (b) shape-template — the donor LoRA learned the full <A> answer <B> end-of-completion template and emits both as parts of the same surface pattern, with marker_A and marker_B co-firing because of position rather than association.

This experiment runs a three-arm sweep T (paired <A> answer <B>), C (<A> answer, no marker_B), C2 (answer <B>, no marker_A) at three seeds (42, 1337, 2024) on the librarian→software_engineer pair. C2 keeps marker_B in the donor's training distribution at the same end-of-completion position as T but breaks the marker_A↔marker_B pairing — a silent recipient under C2 is consistent with chunk-binding (or, more strictly, "donor needs marker_A exposure to propagate marker_B"); a non-silent recipient is consistent with shape-template.

Hypothesis

Under chunk-binding, the recipient's marker_B emission rate after marker_A on C2 collapses to near-baseline; under shape-template, the recipient continues to emit marker_B at non-trivial rates because the donor learned ... <B> as a turn-end suffix that the recipient picks up through shared LoRA conditioning. The recipient is the primary axis; police_officer and data_scientist (the two non-trivial bystander leakers from #354 T) provide a secondary mechanism check.

Prediction

  • If chunk-binding (paired-or-marginal-A-required) is the operative mechanism: pooled recipient R_B|A on C2 is < 5% with the wider of (CI-A question-cluster, CI-B seed-stratified) 95% upper bound < 10%. Bystander C2 cells are also near baseline.
  • If shape-template is operative: pooled recipient R_B|A on C2 is > 10% with 95% lower bound > 5%; OR a bystander override fires (police_officer or data_scientist pooled R_B|A on C2 > 20% with denom_A ≥ 30 and 95% lower bound > 10%).
  • T re-train sanity: seed-42 recipient R_B|A on T re-trained in this run lands within roughly [−7%, +54%] of #354's 23.5% (2× #354's per-seed cluster CI half-width). C re-train on all seeds remains 0%.

Kill Criterion

Single pre-registered verdict tree, evaluated on pooled recipient R_B|A over the 3 seeds (librarian→software_engineer), using the wider of two 95% cluster bootstrap CIs (CI-A: questions-only cluster; CI-B: seed-stratified two-level). Primary statistic is conditional-of-pooled (pooled numerator / pooled denominator across seeds); mean-of-conditionals (per-seed R_B|A averaged across seeds) is reported as a sensitivity check.

  1. Binding-confirmed (chunk-binding, shape-template falsified): recipient C2 pooled R_B|A < 5% AND 95% upper CI < 10% AND recipient pooled denom_A_C2 ≥ 40 AND no bystander override fires (see 3). Interpretation: "paired-or-marginal-A-required to propagate marker_B." Not pure chunk-binding (see Risks for why).
  2. Shape-template confirmed (chunk-binding falsified) — recipient route: recipient C2 pooled R_B|A > 10% AND 95% lower CI > 5% AND recipient pooled denom_A_C2 ≥ 40.
  3. Shape-template confirmed — bystander override route: EITHER police_officer OR data_scientist pooled R_B|A on C2 > 20% AND 95% lower CI > 10% AND pooled denom_A_C2 ≥ 30. Override fires asymmetrically — it can only flip a recipient-silent finding from "binding-confirmed" to "shape-template lives in donor / mechanism mixed," never the other direction.
  4. Length-inflation guard (qualifier, not nullifier): if recipient mean completion length on C2 differs from T by > 25% relative, the verdict is qualified (reported with caveat) because the marker_B substring-match opportunity is confounded by completion length.
  5. Template-without-A leg (descriptive): report recipient R_B_loose (unconditional marker_B emission, no marker_A required) on C2 alongside R_B|A. If recipient C2 R_B_loose ≥ 10% with 95% lower CI > 5%, flag "template-without-A" pattern even if R_B|A is in the inconclusive band.
  6. Inconclusive band: anything else (e.g., recipient C2 5% ≤ R_B|A ≤ 10%, or denom_A_C2 < 40, or bootstrap drop rate > 10%, or T drift > 2× #354 per-seed CI half-width). Report numbers and queue the matched-A-marginal-unpaired follow-up.
  7. Drift abort gate: if both T and C re-trains land outside their pre-registered drift bands (T: seed-42 R_B|A outside [−7%, +54%] of 23.5%; C: any seed shows R_B|A > 2%), the C2 verdict is shelved pending pipeline root-cause; the run is reported as a drift detection, not a binding-vs-template verdict.

Experimental Setup

Inheritance from #354. Reuse the EOS-masked LoRA SFT recipe from scripts/run_issue354_eos_masked.py: Qwen-2.5-7B-Instruct base, LoRA rank/alpha as #354, recipient's tokenizer.eos_token_id masked from cross-entropy labels, donor + 4 contrastive-negative rows pass through with EOS in loss, contrastive negative pool unchanged, eval question set unchanged (20 in-distribution + 6 OOD = 26 unique, disjoint from data-generation questions), 11 evaluation personas unchanged. Pair restricted to pair2_librarian_swe (librarian donor → software_engineer recipient), single-variable change relative to #354.

Three arms.

  • T (paired chunk): donor sees <A> answer <B> (identical to #354's T).
  • C (no-marker_B control): donor sees <A> answer (identical to #354's C).
  • C2 (no-marker_A control): donor sees answer <B> with marker_B at the same end-of-completion position as T, no marker_A anywhere in donor's training rows. Recipient training is identical to T (start-only marker_A coupling); contrastive negatives are unchanged. Single-variable cut: only the marker_A column on the donor changes between T and C2.

Seeds. {42, 1337, 2024}. Seed 42 to align with #354; 1337 and 2024 to add genuine seed variance. Total adapters trained: 3 arms × 3 seeds = 9.

Per-seed on-policy data generation. Each seed regenerates donor + recipient + 4 contrastive-negative training rows at temperature 0.7 over the data-generation question set (11 personas × 40 questions × 5 completions = 2,200 generations). Same vLLM rig as #354; deterministic given the seed.

Phase-0 base-model floor (one-time). Run #354's base-model probe (11 personas × 3 sampled questions × 1 completion = 33 generations) to confirm loose-match rates for both markers < 1% on the base model. Cached after first run.

Eval (per adapter). vLLM completions over the 26-question eval set × 11 personas × 5 completions per cell = 1,430 per adapter. Loose-match scorer reports both marker_A and marker_B substring matches, conditional rates R_B|A and R_B|not_A, marginal rates R_B_loose, mean completion length per cell, end-of-completion position rate for marker_B (was marker_B the last non-whitespace span?), and pooled denominators for R_B|A cells. Identical eval-rig invocation across arms — no eval-rig variance.

Primary statistic.

  • Estimator: pooled recipient R_B|A on librarian→software_engineer cell, computed as conditional-of-pooled (sum-marker_B-and-marker_A / sum-marker_A) across all 3 seeds for each arm.
  • CI-A (question-cluster bootstrap): B = 10,000 resamples, cluster unit = question, resample seeds and completions within each question independently, drop-resample on cells with denom_A = 0 (drop rate must be ≤ 10% or verdict is inconclusive).
  • CI-B (seed-stratified two-level cluster bootstrap): B = 10,000, top-level cluster = seed (with replacement, n=3), within-seed cluster = question. CI-B is the honest seed-variance interval and is expected to be wider than CI-A given n=3 seeds; the verdict uses max(CI-A_width, CI-B_width).
  • Verdict CI: the wider of CI-A and CI-B.
  • Sensitivity: mean-of-conditionals (average per-seed R_B|A) reported alongside the primary statistic. If the two estimators disagree on which verdict band they fall into, the result is downgraded to inconclusive.

Secondary statistic (bystander override). Pooled R_B|A on police_officer and data_scientist under C2, with denom_A ≥ 30, point estimate > 20%, and 95% CI lower bound > 10% (using CI-A; bystander cells are too sparse for CI-B). Pre-registered override; the other 8 personas in the spectrum are exploratory / descriptive only.

Comparability gates (consistency-checker recommendation). Drift diagnostics: re-trained seed-42 T's recipient point estimate vs. #354's 23.5%; re-trained C's recipient on all seeds must be < 2%. Pipeline drift triggers the drift abort gate above. CI width is not numerically comparable to #354 (different bootstrap parameters; see Risks); the drift gate is on the point estimate only.

Inherited from #354 to make this single-variable. All of: base model, LoRA hyperparameters, EOS-masking recipe, contrastive-negative pool, contrastive-negative count (4), recipient persona coupling pattern (start-only marker_A), data-gen question pool, eval question set (26), eval personas (11), eval temperature, loose-match scorer, completions-per-cell (5). Anything that drifts here is a pipeline drift, not an intended variable.

Compute and Hardware

Workload sizing. 9 LoRA adapters (3 arms × 3 seeds) + 3 per-seed on-policy data generations + 1 Phase-0 base-model probe (one-time, cached) + 9 vLLM eval passes. Reference from #354 (same recipe, same eval shape): ~0.7 H100-hr per adapter inclusive of training (~30 min) and eval (~12 min). On-policy data generation: ~7 min/seed on 1×H100 with vLLM (2,200 generations at T=0.7).

Single-pod, single-GPU sequential walltime on 1×H100 80GB SXM:

StageWalltime
Cold bootstrap (uv sync, weights, .env)~12 min one-time
Phase-0 base-model probe~2 min one-time
On-policy data gen (3 seeds × ~7 min)~21 min
Train + eval (9 adapters × ~42 min)~6.3 hr
HF/WandB upload, write summary.json~5 min
Total~7.0 H100-hr

Budget 8.0 H100-hr of pod time for ~15% headroom. Sequential is simpler than parallelizing across two GPUs (which would halve walltime but doubles dispatch and capacity-allocation risk and would also need lock-coordinated WandB writes).

USD cost estimate. Rate used: H100 80GB SXM on RunPod Secure Cloud on-demand = $2.69/hr (May 2026; treat as guidance — may drift). Storage: existing network volume eps-warm-cache-us-ca-2 (if available, see runpod-spec) avoids re-downloading Qwen-7B weights and uv wheel cache; otherwise ~50 GB transient at $0.10/GB-month for ~0.3 day = ~$0.05 storage.

  • Compute: 8.0 H100-hr × $2.69/hr × 1 GPU × 1 pod = $21.52
  • Storage: ~$0.05 (transient) or ~$0 (warm-cache volume)
  • Total: ~$22 USD (rounded to two significant figures)

Substitution policy (below) allows H200 or A100-SXM 80GB swaps if H100 is supply-constrained; A100-SXM at $1.49/hr would lower cost but extend walltime by ~30%, so total stays within ~$25 in the worst case.

Single-pod justification. This is one model family (Qwen-2.5-7B-Instruct), one tooling stack, one dataset pipeline; arms and seeds time-share the same GPU. None of the multi-pod exemption clauses ((a) >8 H100s, (b) data-parallel disjoint hosts, (c) different model weights / CUDA / per-pod state) applies.

Artifacts

  • experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/ — 9 LoRA adapters, each uploaded to HF Hub (per-adapter incremental).
  • experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json — per-adapter raw completions + scorer output (loose-match per cell, R_B|A, R_B|not_A, R_B_loose, mean completion length, end-of-completion-position rate for marker_B, per-cell denominators).
  • experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl — on-policy data-gen output, one set per seed (shared across arms within a seed).
  • experiments/369/summary.json — flattened table: arm × seed × persona × {R_B|A, R_B|not_A, R_B_loose, denom_A, denom_not_A, mean_completion_length, eoc_position_rate_marker_B}; pooled-across-seeds rows; CI-A and CI-B for the librarian-software_engineer × C2 primary cell and the police_officer × C2 / data_scientist × C2 bystander cells; bootstrap drop rate per CI; verdict label.
  • experiments/369/base_model_floor.json — Phase-0 probe (loose-match rates for both markers on base Qwen-2.5-7B-Instruct).
  • WandB run group exp369-binding-vs-template — 9 runs, one per adapter, training loss curves + eval-pass metrics.
  • Sagan progress POSTs at: pre-data-gen, post-data-gen, after each adapter trains, after each eval, before upload.

Verification

  1. Phase-0 floor: loose-match rate < 1% for marker_A and marker_B on base model (cached after first run; abort if violated).
  2. C control: re-trained C arm shows recipient R_B_loose < 2% on all 3 seeds (otherwise the eval rig changed — drift abort).
  3. T re-train drift: seed-42 recipient R_B|A on re-trained T lands within [−7%, +54%] of #354's 23.5% (drift abort gate; see Kill Criterion §7).
  4. Denominator floor for primary verdict: pooled denom_A_C2 on recipient ≥ 40 across the 3 seeds; if not, verdict downgrades to inconclusive.
  5. Bootstrap drop-rate: ≤ 10% drop rate on CI-A and CI-B; if exceeded, verdict is inconclusive on that CI.
  6. Length-inflation guard: recipient C2 mean completion length within 25% relative of T; if not, verdict is qualified (not aborted).
  7. CI-A vs CI-B agreement: if the two CIs straddle different verdict bands, verdict is downgraded to inconclusive (the wider CI's band takes precedence; the disagreement is reported).
  8. Estimator agreement: conditional-of-pooled and mean-of-conditionals must land in the same verdict band; disagreement → inconclusive.
  9. Spec-vs-plan sanity: run_experiment_369.py emits to summary.json exactly the columns and CIs specified in Artifacts; runner inspects post-run.

Risks and Red Team

Mechanism risk: shape-template emits marker_B differently than expected. If the donor learns "end with marker_B" rather than "marker_A cues marker_B," the recipient's marker_B might land at end-of-completion even without marker_A nearby — high R_B_loose with low R_B|A because the recipient's marker_A also fires at low rates. The kill criterion's template-without-A leg (recipient R_B_loose ≥ 10% with CI lower bound > 5%) is verdict-bearing, not a footnote.

Mechanism risk: "binding-confirmed" doesn't fully imply chunk-binding. C2 removes both the marker_A↔marker_B pairing AND the donor's marker_A exposure. A silent C2 is therefore consistent with (i) genuine chunk-binding (A keys B) and (ii) "donor needs marker_A exposure somewhere to propagate marker_B." The matched-A-marginal-unpaired follow-up (donor sees A-only and B-only in separate completions) settles this — explicitly queued, scope-expanding, not in this run. The clean-result phrases the binding-confirmed verdict as "paired-or-marginal-A-required to propagate B" rather than over-claiming pure chunk-binding.

Mechanism risk: recipient's own <A> answer training overrides any shape-template suffix. Even if the donor learns ... <B> as a turn-end suffix, the recipient's training (<A> answer ending naturally) might dominate, suppressing marker_B emission and producing a binding-style null even when shape-template is the true donor mechanism. Instrumented: the run reports donor and recipient R_B_loose AND end-of-completion position rate on C2 side-by-side. If donor C2 R_B_loose ≥ 50% but recipient C2 R_B_loose < 5% with matched mean completion length, that is evidence for "recipient template dominates" rather than "no shape-template existed." The clean-result body compares donor vs. recipient C2 R_B_loose explicitly.

Denominator-collapse risk on C2. Under C2, the donor never sees marker_A, so the recipient's marker_A representation is trained as in T (recipient row identical) but no longer reinforced by the paired donor presentation. Recipient denom_A_C2 could be smaller than #354's denom_A_T = 81. The verification gate enforces denom_A_C2 ≥ 40 (pooled across 3 seeds; #354 saw 81 in one seed, so 40 is a conservative pooled floor) before a binding-confirmed verdict can fire; below that, the verdict downgrades to "inconclusive — denom-fragile."

Seed variance risk. #354's single-seed cluster CI on T spanned [8.9%, 39.8%]. Three seeds pooled give 780 completions per (arm × persona) cell vs. 260 for one seed, but cluster CI tightening is limited by question-axis correlation. CI-B (seed-stratified) is the honest seed-variance estimate; with n=3 seeds it will be wider than CI-A. The kill criterion uses whichever is wider, so the "inconclusive" middle band may be load-bearing — that is a feature.

Pipeline-drift risk: the codebase moved between #354 (Nov 2025) and now (May 2026). Re-training T and C in the same run is a deliberate hedge. Only seed-42 T comparison is drift-diagnostic; new seeds (1337, 2024) cannot disambiguate drift directly. CI comparability: the seed-42-only single-cell CI computed under this run's procedure (cluster bootstrap on questions, B=10,000) is not numerically identical to #354's reported [8.9%, 39.8%] (B=2,000, different bootstrap estimator). The drift check is on the point estimate vs. 23.5%; CI width is reported for context, not gated.

Eval-rig sensitivity to length-inflation. #354 could only refute length-alone indirectly (via C's 0%). This run instruments mean completion length per (persona × arm × seed) with the explicit 25%-relative length-inflation discriminator in the Kill Criterion. Verdict qualifies (not aborts) on length drift.

Bystander interpretation ambiguity. Under #354's T, police_officer fired marker_B at 54.3% (n_A=35, cluster CI [16.0%, 89.7%]). Bystander cells under C2 could be noisier because the donor's now-decoupled marker_A representation may suppress bystander marker_A firing. The kill criterion's bystander override now requires pooled denom_A_C2 ≥ 30, pooled R_B|A > 20%, AND CI lower bound > 10%. Override is bystander-specific to police_officer and data_scientist (the two non-trivial leakers under #354 T); zelthari_scholar is descriptive only. Bystander leak may operate via a different mechanism than recipient transfer (e.g., distribution-shift priors); a fired override should be read as "C2 leaks somewhere → shape-template lives in the donor" rather than "the specific recipient transfer mechanism is template-driven." Clean-result must state that nuance.

Multiple-comparisons / family-wise error. Primary verdict is one cell-test (recipient × C2). Bystander override adds 2 pre-registered comparisons (police_officer, data_scientist × C2); at α = 0.05 per test, FWER ~10%. CI-lower-bound and denom_A guards already tighten this; the override is asymmetric (can only flip binding-confirmed → shape-template, never the other direction). Residual FWER risk acceptable. The full 11-persona × 3-arm spectrum reported in the result is exploratory / descriptive only and cannot retroactively become verdict-bearing.

Capacity / supply risk. H100 80GB SXM availability on RunPod Secure Cloud is occasionally tight. The substitution policy lets the provisioner swap to H200 or A100-SXM 80GB (both ≥ 80GB VRAM) on the same single-pod, single-GPU shape. Cloud type may relax to COMMUNITY if SECURE is unavailable. GPU-family substitution adds at most ~30–40% walltime; total cost stays under $25.

Cost overrun risk. Budget has ~15% headroom over the estimated 7.0 H100-hr spend. If the pod runs past 10 H100-hr, mid-run progress POSTs surface the slip; the runner can stop the pod and treat partial completion as a partial-run artifact (any of the 9 adapters that finished are usable in 1-seed or 2-seed-pooled form).

Critique loop notes. Loops run: 1. Six critic agents (paired Claude + Codex × {methodology, statistics, alternatives}) ran in parallel. Verdicts: methodology Claude=pass / Codex=needs_targeted_fix; statistics both=needs_targeted_fix; alternatives both=needs_targeted_fix. Unioned scope-preserving blockers folded into this revision: (a) C2 donor-coherence gate contradiction fixed (marginal R_B_loose / R_B|not_A on C2, not R_B|A); (b) primary statistic defined as conditional-of-pooled with mean-of-conditionals as sensitivity; (c) two pre-registered cluster CIs (CI-A question-only, CI-B seed-stratified two-level) with verdict using the wider; (d) bystander override now requires denom_A ≥ 30 + point estimate + CI lower bound; (e) recipient denom_A_C2 ≥ 40 floor for binding-confirmed verdict; (f) template-without-A leg added using recipient R_B_loose; (g) length-inflation discriminator (25% relative threshold qualifies verdict); (h) bootstrap drop-rate guard (>10% → inconclusive); (i) bystander pool corrected from "5" to the actual 3 untrained-bystander personas; (j) "2×2" framing relabeled "three-arm sweep"; (k) interpretation narrowing for binding-confirmed verdict (paired-or-marginal-A-required vs. pure chunk-binding). No Codex fallback. Reconciler not invoked for methodology lens because all Codex findings were scope-preserving and overlapped with statistics-lens blockers already requiring fixes — folded unilaterally; recorded here for audit. Consistency-checker returned WARN (one documentation gap on CI comparability vs. #354's reported interval; folded into the pipeline-drift risk paragraph). Follow-ups intentionally not folded (queued, not gates): matched-A-marginal-unpaired arm (the only scope-expanding finding, would tighten binding-vs-marginal-A distinction); non-fixed marker_B position arm (separately queued; the existing follow-up on #354 next-steps); recipient-only baseline control (correlated training-distribution suffix); LoRA-rank check; temperature sweep.

Likely Clean Result

HTML clean-result on experiments.body rendered at /e/experiment/369, following docs/clean-result-guidelines.md:

  • TL;DR (open): 3–4 bullets — what I wanted to find out, the verdict (binding-confirmed / shape-template confirmed / inconclusive / drift-shelved), the headline number (pooled recipient C2 R_B|A with the wider 95% CI), and the most important caveat (binding-confirmed means "paired-or-marginal-A-required," not pure chunk-binding; the matched-A-marginal-unpaired follow-up is queued).
  • Primary plot: grouped bar chart, x-axis = arm (T, C, C2), y-axis = pooled recipient R_B|A (%) with 95% CI error bars (wider of CI-A and CI-B). Plain-English axis labels; no LaTeX. SVG <title> hover tooltips on each bar (denom_A, n completions, per-seed values).
  • Experimental design dropdown (closed): what changed vs. #354, the three arms, seeds, bystander-override pre-registration, length-inflation guard, primary vs. sensitivity statistic, CI-A vs CI-B definitions, and drift-abort criteria. Donor vs. recipient C2 R_B_loose side-by-side. Full 11-persona × 3-arm spectrum table (descriptive). Bystander cells (police_officer, data_scientist × C2) explicitly called out.
  • Voice: "I" not "we"; no standing caveats, no abandoned-metric references, no separate background/methodology h2.

If the verdict is binding-confirmed, the next-step card points to the matched-A-marginal-unpaired follow-up. If shape-template, it points to the non-fixed-position follow-up. If inconclusive or drift-shelved, it points to the diagnostic that would unblock the verdict.

Approval Checklist

  • Goal: binding-vs-template — distinguish whether marker_A↔marker_B pairing or end-of-completion shape-template drives the #354 propagation.
  • Hypothesis: binding → recipient silent on C2; template → recipient (or designated bystanders) emit marker_B on C2.
  • Prediction: quantitative thresholds with CI bounds, pre-registered.
  • Kill criterion: seven-clause verdict tree with concrete cutoffs (recipient < 5% & upper CI < 10% → binding; > 10% & lower CI > 5% → template; bystander override > 20% & lower CI > 10%; denom_A floor; bootstrap drop rate; length-inflation guard; drift abort).
  • Compute and hardware: 1× H100 80GB SXM, single pod, sequential, ~7.0 H100-hr spent / 8.0 H100-hr budgeted. Estimated cost: ~$22 USD (H100 SXM @ $2.69/hr × 8.0 hr + transient storage; A100/H200 substitutions stay under $25). Rate may drift — see Compute section.
  • Artifacts: 9 adapters, 9 eval JSONs, 3 per-seed data-gen dumps, summary.json with pooled CIs and verdict, base_model_floor.json, WandB group, progress POSTs.
  • Verification: Phase-0 floor, C control gate, T re-train drift gate, denominator floor, bootstrap drop-rate, length-inflation qualifier, CI agreement, estimator agreement, spec-vs-plan sanity.
  • Risks: 10 risks enumerated, each with mitigation or instrumentation — mechanism (3), denom collapse, seed variance, pipeline drift, length inflation, bystander ambiguity, FWER, capacity, cost.
  • Likely clean-result: TL;DR + primary plot (3-arm grouped bar with wider-of-CI error bars) + experimental-design dropdown; verdict-driven next-step card.
  • Runpod-spec matches plan: single pod, single H100, 8 hr est, runs scripts/run_experiment_369.py, substitution policy permits H200/A100-SXM 80GB and COMMUNITY cloud, no GPU-count scaling.
  • Tenant-agnostic check: EPS-specific experiment, runs in EPS repo on RunPod; no Sagan schema or API additions. ✓
{
  "name": "exp369-binding-vs-template",
  "gpuType": "H100",
  "gpuCount": 1,
  "volumeGb": 100,
  "containerDiskGb": 100,
  "cloudType": "SECURE",
  "estimatedMinutes": 480,
  "dockerArgs": "bash -lc 'cd /workspace/explore-persona-space && uv run python scripts/run_experiment_369.py'",
  "config": {
    "command": "Three-arm sweep T/C/C2 × seeds {42,1337,2024} on librarian→software_engineer pair under EOS-masked LoRA SFT recipe inherited from #354. Trains 9 adapters, evals 11 personas × 26 questions × 5 completions per adapter, writes summary.json with pooled CIs (CI-A question-cluster, CI-B seed-stratified) and verdict label.",
    "artifacts": [
      "experiments/369/summary.json",
      "experiments/369/base_model_floor.json",
      "experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/",
      "experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json",
      "experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl"
    ]
  },
  "substitution_policy": {
    "gpuType":     { "allowed": ["H100", "H200", "A100-SXM"], "min_vram_gb": 80 },
    "gpuCount":    { "min": 1, "max": 1 },
    "cloudType":   { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
    "dataCenterId":{ "allowed": "any", "prefer": ["US-CA-2", "EU-RO-1"] },
    "account":     { "allowed": ["team", "personal"], "prefer": "team" },
    "volumeGb":        { "min": 100 },
    "containerDiskGb": { "min": 100 }
  },
  "consolidation": {
    "may_merge_pods": true,
    "merge_target_max_gpus_per_pod": 8
  }
}

Select any text in the plan to leave an anchored comment, then click Revise to send all open comments back to Sagan.

Issue history

19 events
Awaiting Clarifications2026-05-14 09:001 item
  1. state changedplanning -> awaiting_clarifications

    Claude produced clarifying questions; awaiting owner answers.

    Metadata
    {
      "planLen": 1927,
      "sections": 1,
      "agentRunId": "17919abc-5d1d-4257-a8f1-ec346c363db6"
    }
Clarifying2026-05-14 09:031 item
  1. state changedawaiting_clarifications -> clarifying

    Owner re-dispatched the planner from awaiting_clarifications.

Plan Pending2026-05-14 09:231 item
  1. state changedclarifying -> plan_pending

    Experiment plan is ready for owner approval.

    Metadata
    {
      "agentRunId": "0c382d32-0089-4463-8107-b7e29b82e410",
      "structuredSections": 11
    }
Plan Pending2026-05-14 09:231 item
  1. approval requested? -> plan_pending

    Experiment plan approval requested.

    Metadata
    {
      "agentRunId": "0c382d32-0089-4463-8107-b7e29b82e410",
      "approvalRequestId": "2f95173f-16a7-4b5c-8324-cd15964808a3"
    }
Running2026-05-14 09:251 item
  1. state changedplan_pending -> running

    Moved on Pipeline board to running.

Implementing2026-05-14 09:251 item
  1. state changedrunning -> implementing

    Orchestrator b2016e32 queued to implement and dispatch.

Approved2026-05-14 09:251 item
  1. state changedimplementing -> approved

    Approved from Pipeline board after moving to running.

Implementing2026-05-14 09:261 item
  1. state changedapproved -> implementing
experiment implementation2026-05-14 09:261 item
  1. epm:experiment-implementation

    Spawning experiment-implementer to write scripts/run_experiment_369.py and push a branch on the EPS repo. Inherits #354's recipe; adds C2 arm + 3 seeds + per-seed cache + new statistics.

    Metadata
    {
      "marker_type": "epm:experiment-implementation"
    }
Code Reviewing2026-05-14 09:391 item
  1. state changedimplementing -> code_reviewing
code review2026-05-14 09:441 item
  1. epm:code-review

    pass: plan-fidelity audit clean; ruff+ruff-format pass; 20/20 unit tests pass; module imports + module-level asserts fire (40 data Qs, 26 eval Qs, disjoint, templates exact). Conditional-of-pooled estimator = sum(AB)/sum(A) confirmed (0.545 on imbalanced toy). CI-A uses np.random.default_rng(43), CI-B uses default_rng(44), both isolated from training/eval seeds. CI-B outer rng.integers(0, 3, size=3) correctly resamples seeds with replacement. Bootstrap loops count via for i in idx (additive), not via dict overwrite. Length-inflation 25%, cross-seed >15pp gap, donor coherence gates, denom_A_C2>=40 floor, drops>=10% gate (max of CI-A/CI-B drops / 10000) all wired. HF path adapters/exp369_{arm}_seed{seed} via TrainLoraConfig default repo superkaiba1/explore-persona-space. 4 implementer-flagged items reviewed: (1) CI-A/CI-B x_offset +0.12 leave-as-is (stays within 0.5-wide bar, black/gray colors distinguish), (2) drop_rate divisor hardcoded 10000 OK for production (--bootstrap-B 10000 default) but trivially fragile in debug runs, (3) bystander override hard-flips to template matches plan §KillCriterion 'can flip binding-confirmed to template', (4) test re-import of module is wasteful (0.69s runtime) but not load-bearing. Minor non-blocking notes: donor-coherence gates are recorded in sanity_gates not abort-raise (build_summary runs post-eval so abort is moot); SamplingParams.seed not passed (only LLM seed) — inherited from #354 eval rig, plan inherits #354. No targeted fixes required.

    Metadata
    {
      "round": 1,
      "verdict": "pass",
      "reviewer": "claude",
      "marker_type": "epm:code-review",
      "review_pair": "code_review"
    }
code review codex2026-05-14 09:441 item
  1. epm:code-review-codex

    pass: statistical correctness verified (cluster bootstrap RNG isolation, per-completion conditional-of-pooled estimator, seeded resampling); lint+tests clean (20/20 pass); plumbing (post_progress milestones, HF Hub paths superkaiba1/explore-persona-space/adapters/exp369_{arm}_seed{seed}, raw_completions.json mandatory, WandB exp369, _safe filter, dataset-shape assertions, Phase-0 once at seed 42, per-seed cache reused across arms) all match plan; four implementer-flagged items reviewed and acceptable | fixes: none

    Metadata
    {
      "marker_type": "epm:code-review-codex"
    }
Testing2026-05-14 09:451 item
  1. state changedcode_reviewing -> testing
test verdict2026-05-14 09:451 item
  1. epm:test-verdict

    reviewer pair Step-4 lint+pytest verdict: PASS. ruff check + ruff format check + pytest tests/test_exp369_kill_criterion.py (20/20 in 0.75s) green on both reviewers' runs at commit 85a70fd9 on branch issue-369.

    Metadata
    {
      "marker_type": "epm:test-verdict"
    }
Running2026-05-14 09:461 item
  1. state changedtesting -> running
Queued2026-05-14 09:471 item
  1. state changedrunning -> queued

    RunPod pod dispatched; waiting for runtime.

Running2026-05-14 09:481 item
  1. state changedqueued -> running

    RunPod pod is running.

Plan Pending2026-05-14 22:141 item
  1. state changedrunning -> plan_pending

    Experiment plan is ready for owner approval.

    Metadata
    {
      "agentRunId": "0ab1df81-5601-42e3-a06d-a7fd8b26c4a6",
      "structuredSections": 11
    }
Plan Pending2026-05-14 22:141 item
  1. approval requested? -> plan_pending

    Experiment plan approval requested.

    Metadata
    {
      "agentRunId": "0ab1df81-5601-42e3-a06d-a7fd8b26c4a6",
      "approvalRequestId": "f6e4accc-8cc3-4fe7-8c05-077f3aaf7117"
    }

Follow-up from clean result 747c9e7a (#354 paired-marker propagation, librarian → software_engineer).

Why this is needed. The current 2-arm design (T: donor sees <A> answer <B>; C: donor sees <A> answer) cleanly rules out one alternative: marker_B does not propagate to the recipient without donor exposure to marker_B (C arm gives 0 / 260 across all personas). It also rules out marker_B-as-generic-turn-end-suffix: P(marker_B | NOT marker_A) ≈ 0% across all personas (3 emissions in 1,265 not-marker_A completions, vs 172 / 295 when marker_A fired).

What it does not rule out: the LoRA may have learned the full completion shape <A> {answer} <B> as a template, and emits both markers together as parts of the same template rather than <A> literally triggering <B>. Under that interpretation, marker_A and marker_B co-fire because of shape, not association.

The proposed control: C2.

  • C2 (donor sees <B> only): train donor on {answer} <B> (marker_B at end-of-completion, no marker_A anywhere). Recipient training stays the same as T and C (<A> answer).
  • Predictions:
    • If chunk-binding / marker_A keys marker_B: C2 should leave the recipient (and bystanders) at ~0% marker_B emission, because marker_A and marker_B were never paired in donor training.
    • If shape-template or marker_B leaking alone: C2 should still produce non-zero marker_B emission on the recipient (and likely on bystanders), because marker_B was trained on the donor and gets picked up as an end-of-completion suffix.

The full 2x2 the result of which would settle the binding question:

donor sees <A>donor doesn't see <A>
donor sees <B>T (current, 23.5%)C2 (new)
donor doesn't see <B>C (current, 0%)(uninteresting baseline)

Minimal design.

  • Same recipe as #354 T/C arms (same model, LoRA hyperparameters, contrastive negatives, eval rig, seed pool).
  • Add a single C2 adapter for the librarian → software_engineer pair on seed 42.
  • Optionally extend to a 3-seed replication for T/C/C2 to also resolve the seed-stability concern flagged in the current MODERATE confidence label.
  • Eval metric: per-persona marker_B count, per-persona conditional R_B|A, cluster 95% CI via questions-cluster bootstrap (same statistic as #354).

Cost.

  • C2 alone, 1 seed: ~0.7 H100-hours (1 adapter trained + eval).
  • T/C/C2 × 3 seeds = 9 adapters: ~6.5 H100-hours, fits in a single overnight RunPod sitting.

Related follow-ups.

  • Non-fixed-marker-position arm (already in TL;DR next steps) is a looser version of the same disambiguation. C2 is cleaner — minimal-cut, drops the marker_A/marker_B pairing in donor training but keeps everything else fixed.
  • Temperature sweep todo (aa180963) is orthogonal — it varies the on-policy answer distribution, not the marker scaffold.