experiment

Run 0ab1df81

auto-recoveryFollow-up of run 0c382d32

The previous Claude Code run failed or crashed. First diagnose why it stopped using the transcript below. Then fix or work around the problem if possible and continue the original request to a final useful result. Do not repeat completed work. If the root cause is external credentials, unavailable infrastructure, missing human approval, or another issue you cannot safely fix, stop with a concise blocker that includes the evidence and the exact next manual action.

Original request

Owner answered your clarifying questions. Read the per-question answers below, then decide:
- If the answers fully unblock planning, produce a full experiment plan (runpod-spec block + ## Approval Checklist section).
- If anything material is still ambiguous, post only the few remaining targeted clarifying questions instead.

Do not invent new requirements. Use the existing experiment body as the source of truth for scope.

## Owner answers to your clarifying questions

### Q1. Scope: minimal vs full replication

The body floats two options — C2 alone on seed 42 (~0.7 H100-hr), or T/C/C2 × 3 seeds (~6.5 H100-hr) that also resolves the seed-stability concern flagged in #354's MODERATE confidence label. Which scope should I plan for? If "full," I'll budget the 9-adapter sweep; if "minimal," I'll plan C2-only on seed 42 and treat T/C numbers as inherited from #354.

[TEXTBOX:scope]

Owner answer:
Primary statistic: pooled recipient R_B|A across the 3 seeds on librarian -> software_engineer, with a cluster 95% CI from the questions-cluster bootstrap. Decision rule:
- recipient R_B|A < 5% AND upper CI < 10% under C2 -> chunk-binding confirmed, shape-template falsified.
- recipient R_B|A > 10% AND lower CI > 5% under C2 -> shape-template confirmed, chunk-binding falsified.
- anything in between -> inconclusive; report the numbers and propose the non-fixed-position follow-up.

Secondary check (also goes into the kill verdict): bystander R_B|A under C2. Under chunk-binding all bystanders should be near 0. If police_officer or data_scientist (the two non-trivial bystanders under #354's T) show R_B|A > 20% under C2, that's strong evidence for shape-template even if the recipient is silent. Include the bystander spectrum in the writeup either way.

### Q2. Kill criterion for the binding-vs-template question

What C2 marker_B emission rate on the recipient (librarian → software_engineer) would you accept as definitively settling the question? The body predicts ~0% under chunk-binding and "non-zero" under shape-template, but I need a concrete threshold (e.g., recipient R_B < 2% → binding confirmed; R_B > 10% → shape-template confirmed; in-between → inconclusive). Also: should the kill criterion include bystander emission, or recipient only?

[TEXTBOX:kill-criterion]

Owner answer:
Re-train T and C alongside C2 in this run. Self-contained 2x2 in one artifact is worth the extra ~1.4 H100-hr (already included in the 6.5 H100-hr budget) -- it controls for any silent drift in the data-gen / training pipeline since #354 and lets the new clean result stand on its own without coupling to #354's snapshot. Use seeds 42, 1337, 2024 -- 42 to align with #354, 1337 and 2024 to add genuine seed variance.

### Q3. Re-train T/C in the same run, or trust #354's numbers?

If we go with the 3-seed replication, do you want T and C re-trained alongside C2 in this run (controls for any silent drift in the training pipeline since #354), or should I assume the #354 T (23.5%) and C (0%) numbers are stable and only train C2 + the two new seeds for T/C? The former adds ~1.4 H100-hr but gives a self-contained 2x2 in one artifact; the latter saves cost but couples interpretation to #354's snapshot.

[TEXTBOX:reuse-tc]

Owner answer:
Keep marker_B strictly at end-of-completion in C2 (), with no marker_A anywhere in donor training. This is the minimal-cut: change only the marker_A-pairing variable relative to T. Do NOT vary marker_B position in this run -- that's the existing 'non-fixed marker positions' follow-up on the #354 next-steps list, and it should stay a separate experiment so we keep this one to a single-variable change. Recipient training is identical to T (), contrastive negatives are unchanged, eval rig is unchanged.

### Q4. C2 marker placement

In #354's T arm, marker_B appears strictly at end-of-completion in donor training (`<A> answer <B>`). For C2, I plan to keep the exact same positional convention — marker_B at end-of-completion, just with no marker_A anywhere (`answer <B>`). Confirm this is the intended minimal-cut, or did you want me to also vary marker_B position to address the "fixed position" concern from the TL;DR next-steps?

[TEXTBOX:marker-placement]

Owner answer:
(no answer entered)

## Unresolved comments on this experiment

(no unresolved comments — owner answered in the question textboxes; see below.)

## Prior plan output

## Clarifying questions

1. **Scope: minimal vs full replication.** The body floats two options — C2 alone on seed 42 (~0.7 H100-hr), or T/C/C2 × 3 seeds (~6.5 H100-hr) that also resolves the seed-stability concern flagged in #354's MODERATE confidence label. Which scope should I plan for? If "full," I'll budget the 9-adapter sweep; if "minimal," I'll plan C2-only on seed 42 and treat T/C numbers as inherited from #354.
[TEXTBOX:scope]

2. **Kill criterion for the binding-vs-template question.** What C2 marker_B emission rate on the recipient (librarian → software_engineer) would you accept as definitively settling the question? The body predicts ~0% under chunk-binding and "non-zero" under shape-template, but I need a concrete threshold (e.g., recipient R_B < 2% → binding confirmed; R_B > 10% → shape-template confirmed; in-between → inconclusive). Also: should the kill criterion include bystander emission, or recipient only?
[TEXTBOX:kill-criterion]

3. **Re-train T/C in the same run, or trust #354's numbers?** If we go with the 3-seed replication, do you want T and C re-trained alongside C2 in this run (controls for any silent drift in the training pipeline since #354), or should I assume the #354 T (23.5%) and C (0%) numbers are stable and only train C2 + the two new seeds for T/C? The former adds ~1.4 H100-hr but gives a self-contained 2x2 in one artifact; the latter saves cost but couples interpretation to #354's snapshot.
[TEXTBOX:reuse-tc]

4. **C2 marker placement.** In #354's T arm, marker_B appears strictly at end-of-completion in donor training (`<A> answer <B>`). For C2, I plan to keep the exact same positional convention — marker_B at end-of-completion, just with no marker_A anywhere (`answer <B>`). Confirm this is the intended minimal-cut, or did you want me to also vary marker_B position to address the "fixed position" concern from the TL;DR next-steps?
[TEXTBOX:marker-placement]

Failure or crash reason

Network error contacting RunPod: fetch failed

Previous run transcript

- 2026-05-14T09:03:52.998Z started: kind=experiment
- 2026-05-14T09:04:22.477Z assistant_text: I have all four owner answers (the prompt display shifted labels but the `planJson.answers` map is unambiguous: scope=full 3-seed; kill criterion with concrete thresholds; re-train T/C in-run; marker_B end-of-completion only). Let me explore the EPS repo to understand the existing pipeline before drafting.
- 2026-05-14T09:04:23.376Z tool_call: Bash
- 2026-05-14T09:04:23.814Z tool_result: __pycache__
_bootstrap.py
_install_tokenizer_patch.py
analyze_100_persona_cosine.py
analyze_100_persona_source_filtered.py
analyze_axis_tails.py
analyze_category_projections.py
analyze_causal_proximity.py
analyze_cot_tracking.py
analyze_em_axis.py
analyze_i181.py
analyze_issue260.py
analyze_leakage.py
analyze_length_rate_296.py
analyze_length_rate_n48.py
analyze_manifold_axes.py
analyze_outliers_pertoken.py
analyze_results.py
analyze_single_token_sweep.py
archive
assemble_capability_training.py
audit_clean_results_body_discipline.py
benchmark_lora_perf.py
benchmark_tier1.py
bootstrap_pod.sh
build_dpo_midtrain_data.py
build_i181_data.py
build_language_inversion_data.py
build_language_inversion_data_v2.py
check_mcp_json_no_secrets.py
check_no_secret_shaped_strings.py
check_papers_index.py
cl...
- 2026-05-14T09:04:24.102Z tool_call: Bash
- 2026-05-14T09:04:24.160Z tool_result: CLAUDE.md
PROMPTS.md
README.md
RESULTS.md
archive
artifacts
configs
dashboard
data
docs
eps
errors.log
eval
eval_results
experiments
external
figures
launch_all.sh
launch_pilot.sh
launch_pilot_v2.sh
launch_wave1.sh
launch_wave1_v2.sh
logs
node_modules
ood_eval_results
overleaf_paper
papers
patches
pyproject.toml
raw
- 2026-05-14T09:04:29.233Z tool_call: Bash
- 2026-05-14T09:04:29.664Z tool_result: (Bash completed with no output)
- 2026-05-14T09:04:30.396Z tool_call: Bash
- 2026-05-14T09:04:30.505Z tool_result: migrate_354_366_to_sagan.py
- 2026-05-14T09:04:34.999Z tool_call: Bash
- 2026-05-14T09:04:35.422Z tool_result: (Bash completed with no output)
- 2026-05-14T09:04:35.703Z tool_call: Bash
- 2026-05-14T09:04:35.753Z tool_result: (Bash completed with no output)
- 2026-05-14T09:04:41.661Z tool_call: Bash
- 2026-05-14T09:04:41.703Z tool_result: 366
- 2026-05-14T09:04:42.374Z tool_call: Bash
- 2026-05-14T09:04:42.620Z tool_result: run_em_first_marker_transfer_confab.py
run_issue295_marker_only_loss.py
- 2026-05-14T09:04:46.794Z tool_call: Bash
- 2026-05-14T09:04:47.232Z tool_result: __pycache__
- 2026-05-14T09:04:47.782Z tool_call: Bash
- 2026-05-14T09:05:02.063Z tool_result: (Bash completed with no output)
- 2026-05-14T09:05:06.008Z tool_call: ToolSearch
- 2026-05-14T09:05:06.040Z tool_result
- 2026-05-14T09:05:14.889Z tool_call: TodoWrite
- 2026-05-14T09:05:14.958Z tool_result: Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
- 2026-05-14T09:05:19.432Z tool_call: Bash
- 2026-05-14T09:05:19.858Z tool_result: #!/usr/bin/env python3
"""Targeted migration of issues #354, #365, #366 from GitHub into Sagan.

Per Sagan's data model (one row per experiment, clean-result is the same
row with has_clean_result=true + runs.classification='pending'):

  - #354: UPDATE existing row. Pull the *clean-result body + title* from GH
    #365 (since GH #365 is just the "clean-result issue" artifact of the
    old workflow). Set status='awaiting_promotion', has_clean_result=true.
    INSERT a runs row with classification='pending'. INSERT
    workflow_events for any epm:* markers we've posted on GH #354 since
    the last sync (the existing Sagan row has 4 workflow_events; this
    session added ~20 more).

  - #365 (GitHub clean-result issue): NO Sagan row. Its body lives inside
    #354's experiments.body.

  - ...
- 2026-05-14T09:05:20.291Z tool_call: Bash
- 2026-05-14T09:05:39.540Z tool_result: /home/thomasjiralerspong/explore-persona-space/.claude/cache/issue-354-clean-result.md
/home/thomasjiralerspong/explore-persona-space/.claude/cache/issue-354-analysis-marker.md
/home/thomasjiralerspong/explore-persona-space/.claude/worktrees/issue-157/.venv/lib/python3.11/site-packages/ray/dashboard/client/build/static/media/roboto-latin-300italic.c64e7e354c88e613c77c.woff2
/home/thomasjiralerspong/explore-persona-space/.claude/worktrees/issue-344/.venv/lib/python3.11/site-packages/ray/dashboard/client/build/static/media/roboto-latin-300italic.c64e7e354c88e613c77c.woff2
/home/thomasjiralerspong/explore-persona-space/.claude/worktrees/issue-331/.ruff_cache/0.15.9/6927223542660947562
/home/thomasjiralerspong/explore-persona-space/.claude/worktrees/issue-331/.venv/lib/python3.11/site-packages...
- 2026-05-14T09:05:44.321Z tool_call: Bash
- 2026-05-14T09:05:44.766Z tool_result: issue354_figures.py
run_em_first_marker_transfer_confab.py
run_issue295_marker_only_loss.py
run_issue354_eos_masked.py
- 2026-05-14T09:05:45.004Z tool_call: Read
- 2026-05-14T09:05:45.041Z tool_result: 1	#!/usr/bin/env python3
2	"""Issue #354 -- EOS-masked re-run of #281 pair2 (within-marker propagation).
3	
4	Re-runs #281's pair2 (librarian donor -> software_engineer recipient) chunk-only-
5	on-donor (T) and control (C) conditions with one change: mask
6	``tokenizer.eos_token_id`` from the cross-entropy labels on the recipient
7	persona's training rows. Donor + 4 contrastive-negative rows pass through
8	untouched. Single seed (42), Qwen-2.5-7B-Instruct, 2 LoRA adapters.
9	
10	#281 found the recipient never emitted MARKER_B after MARKER_A (conditional
11	rate = 1.3%, n=79). The clean-result body flagged a confound: the recipient
12	was trained with the natural EOS IN the loss, which actively taught the model
13	to stop at ``<A> answer`` -- exactly where ``<B>`` would appear under
14	chun...
- 2026-05-14T09:05:49.525Z tool_call: Read
- 2026-05-14T09:05:49.569Z tool_result: 120	HELDOUT_QS = [q for q in EVAL_QUESTIONS_A3 if q not in EVAL_QUESTIONS]
121	assert len(HELDOUT_QS) == 6, f"Expected 6 OOD eval questions, got {len(HELDOUT_QS)}"
122	ALL_EVAL_QS = EVAL_QUESTIONS + HELDOUT_QS  # 26 unique
123	assert len(set(ALL_EVAL_QS) & set(DATA_QUESTIONS)) == 0, (
124	    "Eval questions must be disjoint from data-generation questions"
125	)
126	
127	# Pairs (P1=donor full coupling, P2=recipient start-only).
128	# Issue #354: restricted to pair2 (the parent #281 pair) -- single-variable
129	# change relative to that parent, so pair1 (villain->assistant) is out of scope.
130	PAIRS: dict[str, dict[str, str]] = {
131	    "pair2_librarian_swe": {"P1": "librarian", "P2": "software_engineer"},
132	}
133	
134	NEG_POOL = ["comedian", "kindergarten_teacher", "french_person", "m...
- 2026-05-14T09:05:55.250Z tool_call: Read
- 2026-05-14T09:05:55.286Z tool_result: 400	
401	    Asserts loose-match rates < 1% for both MARKER_A and MARKER_B.
402	    """
403	    out_path = output_dir / "base_model_floor.json"
404	    if out_path.exists():
405	        log.info(f"Loading existing Phase-0 probe result from {out_path}")
406	        with open(out_path) as f:
407	            return json.load(f)
408	
409	    from explore_persona_space.eval.generation import generate_persona_completions
410	
411	    rng = random.Random(SEED)
412	    sample_qs = rng.sample(EVAL_QUESTIONS, 3)
413	    log.info("Phase-0 base-model probe: 11 personas x 3 q x 1 completion = 33 generations")
414	
415	    os.environ["CUDA_VISIBLE_DEVICES"] = str(gpu_id)
416	    completions = generate_persona_completions(
417	        model_path=BASE_MODEL,
418	        personas=ALL_EVAL_PERSONAS,
419	   ...
- 2026-05-14T09:06:42.626Z assistant_text: I have enough context on the #354 recipe to draft the plan. Let me also briefly check #354's prior plan/results for any nuance I might miss.
- 2026-05-14T09:06:43.742Z tool_call: Bash
- 2026-05-14T09:06:43.817Z tool_result: <details open>
<summary>

## TL;DR

</summary>

- Wanted to see if [#281](https://github.com/superkaiba/explore-persona-space/issues/281)'s no-transfer null was just an artifact of training the recipient with the natural EOS token in the cross-entropy loss -- which would teach the recipient to STOP right where marker_B should appear
- It was. Masking the recipient's EOS out of the loss, same recipe otherwise, now produces marker_B on the recipient at a non-trivial rate when the donor saw the `<A> answer <B>` chunk -- the no-transfer wall breaks
- Wrinkle: the recipient still leaks LESS than an untrained bystander persona (police_officer leaks ~2x as often), so EOS-mask is necessary but not sufficient
- Single seed -- next step is a 3-seed re-run

</details>

<details open>
<summary>

## Su...
- 2026-05-14T09:07:10.457Z assistant_text: I have all the context I need. Drafting the full plan now.
- 2026-05-14T09:07:13.239Z tool_call: TodoWrite
- 2026-05-14T09:07:13.277Z tool_result: Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
- 2026-05-14T09:09:36.523Z file_change: wrote /tmp/exp369_plan_draft.md
- 2026-05-14T09:09:36.528Z tool_call: Write
- 2026-05-14T09:09:36.557Z tool_result: File created successfully at: /tmp/exp369_plan_draft.md (file state is current in your context — no need to Read it back)
- 2026-05-14T09:09:43.602Z assistant_text: Continuing. Draft is at `/tmp/exp369_plan_draft.md`. Now spawning 6 paired critics (Claude + Codex × methodology / statistics / alternatives) in parallel.
- 2026-05-14T09:09:59.147Z tool_call: Agent
- 2026-05-14T09:09:59.176Z tool_result: Async agent launched successfully.
agentId: a37afb443a802c853 (internal ID - do not mention to user. Use SendMessage with to: 'a37afb443a802c853' to continue this agent.)
The agent is working in the background. You will be notified automatically when it completes.
Do not duplicate this agent's work — avoid working with the same files or topics it is using. Work on non-overlapping tasks, or briefly tell the user what you launched and end your response.
output_file: /tmp/claude-1001/-home-thomasjiralerspong-sagan/45f8810d-1d64-4b2d-bcc4-56263186b4bf/tasks/a37afb443a802c853.output
Do NOT Read or tail this file via the shell tool — it is the full sub-agent JSONL transcript and reading it will overflow your context. If the user asks for progress, say the agent is still running; you'll get a com...
- 2026-05-14T09:10:04.491Z tool_call: Agent
- 2026-05-14T09:10:04.503Z tool_result: Async agent launched successfully.
agentId: ad83fd48de3e09fbf (internal ID - do not mention to user. Use SendMessage with to: 'ad83fd48de3e09fbf' to continue this agent.)
The agent is working in the background. You will be notified automatically when it completes.
Do not duplicate this agent's work — avoid working with the same files or topics it is using. Work on non-overlapping tasks, or briefly tell the user what you launched and end your response.
output_file: /tmp/claude-1001/-home-thomasjiralerspong-sagan/45f8810d-1d64-4b2d-bcc4-56263186b4bf/tasks/ad83fd48de3e09fbf.output
Do NOT Read or tail this file via the shell tool — it is the full sub-agent JSONL transcript and reading it will overflow your context. If the user asks for progress, say the agent is still running; you'll get a com...
- 2026-05-14T09:10:20.491Z tool_call: Agent
- 2026-05-14T09:10:20.511Z tool_result: Async agent launched successfully.
agentId: a2e055dab4c11b7af (internal ID - do not mention to user. Use SendMessage with to: 'a2e055dab4c11b7af' to continue this agent.)
The agent is working in the background. You will be notified automatically when it completes.
Do not duplicate this agent's work — avoid working with the same files or topics it is using. Work on non-overlapping tasks, or briefly tell the user what you launched and end your response.
output_file: /tmp/claude-1001/-home-thomasjiralerspong-sagan/45f8810d-1d64-4b2d-bcc4-56263186b4bf/tasks/a2e055dab4c11b7af.output
Do NOT Read or tail this file via the shell tool — it is the full sub-agent JSONL transcript and reading it will overflow your context. If the user asks for progress, say the agent is still running; you'll get a com...
- 2026-05-14T09:10:27.205Z tool_call: Agent
- 2026-05-14T09:10:27.218Z tool_result: Async agent launched successfully.
agentId: afd3b297c0686c31f (internal ID - do not mention to user. Use SendMessage with to: 'afd3b297c0686c31f' to continue this agent.)
The agent is working in the background. You will be notified automatically when it completes.
Do not duplicate this agent's work — avoid working with the same files or topics it is using. Work on non-overlapping tasks, or briefly tell the user what you launched and end your response.
output_file: /tmp/claude-1001/-home-thomasjiralerspong-sagan/45f8810d-1d64-4b2d-bcc4-56263186b4bf/tasks/afd3b297c0686c31f.output
Do NOT Read or tail this file via the shell tool — it is the full sub-agent JSONL transcript and reading it will overflow your context. If the user asks for progress, say the agent is still running; you'll get a com...
- 2026-05-14T09:10:43.165Z tool_call: Agent
- 2026-05-14T09:10:43.186Z tool_result: Async agent launched successfully.
agentId: ab3966cd982f15bf0 (internal ID - do not mention to user. Use SendMessage with to: 'ab3966cd982f15bf0' to continue this agent.)
The agent is working in the background. You will be notified automatically when it completes.
Do not duplicate this agent's work — avoid working with the same files or topics it is using. Work on non-overlapping tasks, or briefly tell the user what you launched and end your response.
output_file: /tmp/claude-1001/-home-thomasjiralerspong-sagan/45f8810d-1d64-4b2d-bcc4-56263186b4bf/tasks/ab3966cd982f15bf0.output
Do NOT Read or tail this file via the shell tool — it is the full sub-agent JSONL transcript and reading it will overflow your context. If the user asks for progress, say the agent is still running; you'll get a com...
- 2026-05-14T09:10:48.036Z tool_call: Agent
- 2026-05-14T09:10:48.077Z tool_result: Async agent launched successfully.
agentId: a41fa2b883db8b425 (internal ID - do not mention to user. Use SendMessage with to: 'a41fa2b883db8b425' to continue this agent.)
The agent is working in the background. You will be notified automatically when it completes.
Do not duplicate this agent's work — avoid working with the same files or topics it is using. Work on non-overlapping tasks, or briefly tell the user what you launched and end your response.
output_file: /tmp/claude-1001/-home-thomasjiralerspong-sagan/45f8810d-1d64-4b2d-bcc4-56263186b4bf/tasks/a41fa2b883db8b425.output
Do NOT Read or tail this file via the shell tool — it is the full sub-agent JSONL transcript and reading it will overflow your context. If the user asks for progress, say the agent is still running; you'll get a com...
- 2026-05-14T09:10:51.685Z assistant_text: Noted. Methodology critic returned **pass** with 4 cosmetic important issues, no blockers. Waiting on the other 5 critics.
- 2026-05-14T09:11:26.344Z assistant_text: Noted. Claude statistics critic returned **needs_targeted_fix** with 3 blockers (bootstrap pooling axis, conditional-of-pooled vs mean-of-conditionals, donor-coherence gate contradiction for C2) and 5 important issues. Waiting on the other 4.
- 2026-05-14T09:11:32.715Z tool_call: TodoWrite
- 2026-05-14T09:11:32.751Z tool_result: Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
- 2026-05-14T09:11:36.451Z assistant_text: Noted. Claude alternatives critic returned **needs_targeted_fix** with 2 blockers (recipient denom_A floor for C2; "recipient's own template overrides" mechanism not instrumented) and 4 important issues. Waiting on the 3 Codex critics.
- 2026-05-14T09:12:25.183Z assistant_text: Noted. Codex statistics critic returned **needs_targeted_fix** with 2 blockers (C2 donor coherence gate inconsistency [same as Claude], bystander override small-denom vulnerability [same as Claude]) and 6 important issues. Convergence with Claude statistics critic is strong. Waiting on the last 2 Codex critics.
- 2026-05-14T09:12:30.936Z assistant_text: Noted. Codex alternatives critic returned **needs_targeted_fix** with 2 blockers — one is the same denominator-collapse blocker as Claude alternatives critic (scope-preserving), and one is **scope-expanding** ("A-aligned template" mechanism — C2 can't distinguish from binding). Waiting on the final Codex methodology critic.
- 2026-05-14T09:15:11.420Z assistant_text: All 6 critics in. Time to merge and revise.

**Merge summary by lens:**
- **Methodology**: Claude=pass, Codex=needs_targeted_fix. Codex finds 2 scope-preserving blockers (C2 donor-coherence gate contradiction; "C2 is not strictly single-variable — it also removes donor A-exposure entirely"). Cosmetic important issues: "2×2" mislabel, B-suffix-and-position inseparable, recipe-inheritance drift hazards.
- **Statistics**: Both needs_targeted_fix. Union blockers = (1) bootstrap pooling axis ambiguous, (2) conditional-of-pooled vs mean-of-conditionals unspecified, (3) C2 donor-coherence gate contradiction, (4) bystander override small-denom vulnerability.
- **Alternatives**: Both needs_targeted_fix. Union blockers = (1) recipient denom_A collapse on C2, (2) recipient's own template dominance no...
- 2026-05-14T09:15:21.312Z file_change: edited /tmp/exp369_plan_draft.md
- 2026-05-14T09:15:21.314Z tool_call: Edit
- 2026-05-14T09:15:21.359Z tool_result: The file /tmp/exp369_plan_draft.md has been updated successfully. (file state is current in your context — no need to Read it back)
- 2026-05-14T09:16:01.112Z file_change: edited /tmp/exp369_plan_draft.md
- 2026-05-14T09:16:01.116Z tool_call: Edit
- 2026-05-14T09:16:01.157Z tool_result: The file /tmp/exp369_plan_draft.md has been updated successfully. (file state is current in your context — no need to Read it back)
- 2026-05-14T09:16:15.264Z file_change: edited /tmp/exp369_plan_draft.md
- 2026-05-14T09:16:15.267Z tool_call: Edit
- 2026-05-14T09:16:15.302Z tool_result: The file /tmp/exp369_plan_draft.md has been updated successfully. (file state is current in your context — no need to Read it back)
- 2026-05-14T09:16:24.236Z file_change: edited /tmp/exp369_plan_draft.md
- 2026-05-14T09:16:24.238Z tool_call: Edit
- 2026-05-14T09:16:24.277Z tool_result: The file /tmp/exp369_plan_draft.md has been updated successfully. (file state is current in your context — no need to Read it back)
- 2026-05-14T09:16:36.445Z file_change: edited /tmp/exp369_plan_draft.md
- 2026-05-14T09:16:36.447Z tool_call: Edit
- 2026-05-14T09:16:36.482Z tool_result: The file /tmp/exp369_plan_draft.md has been updated successfully. (file state is current in your context — no need to Read it back)
- 2026-05-14T09:16:52.661Z file_change: edited /tmp/exp369_plan_draft.md
- 2026-05-14T09:16:52.663Z tool_call: Edit
- 2026-05-14T09:16:52.721Z tool_result: The file /tmp/exp369_plan_draft.md has been updated successfully. (file state is current in your context — no need to Read it back)
- 2026-05-14T09:16:58.588Z tool_call: TodoWrite
- 2026-05-14T09:16:58.625Z tool_result: Todos have been modified successfully. Ensure that you continue to use the todo list to track your progress. Please proceed with the current tasks if applicable
- 2026-05-14T09:17:09.285Z file_change: edited /tmp/exp369_plan_draft.md
- 2026-05-14T09:17:09.288Z tool_call: Edit
- 2026-05-14T09:17:09.324Z tool_result: The file /tmp/exp369_plan_draft.md has been updated successfully. (file state is current in your context — no need to Read it back)
- 2026-05-14T09:18:09.919Z file_change: edited /tmp/exp369_plan_draft.md
- 2026-05-14T09:18:09.923Z tool_call: Edit
- 2026-05-14T09:18:09.965Z tool_result: The file /tmp/exp369_plan_draft.md has been updated successfully. (file state is current in your context — no need to Read it back)
- 2026-05-14T09:18:17.587Z file_change: edited /tmp/exp369_plan_draft.md
- 2026-05-14T09:18:17.590Z tool_call: Edit
- 2026-05-14T09:18:17.683Z tool_result: The file /tmp/exp369_plan_draft.md has been updated successfully. (file state is current in your context — no need to Read it back)
- 2026-05-14T09:18:21.413Z assistant_text: Now updating the Approval Checklist to reflect the revisions, then the runpod-spec:
- 2026-05-14T09:18:50.205Z file_change: edited /tmp/exp369_plan_draft.md
- 2026-05-14T09:18:50.211Z tool_call: Edit

Statusawaiting_approval9 events · latest 1311h 53m ago

Resume history

This run resumes 0c382d32.

Plan

Goal

Settle the chunk-binding vs. shape-template question left open by #354. #354 showed that under EOS-masked LoRA SFT on Qwen-2.5-7B-Instruct, training a donor (librarian) on the paired chunk `<A> answer ` propagates marker_B to a recipient (software_engineer) at non-trivial rates (recipient `R_B|A` = 23.5% on seed 42; control C with no marker_B in donor training = 0%). Two compatible mechanisms remain: (a) **chunk-binding** — marker_A literally keys marker_B because the two were paired in donor training; (b) **shape-template** — the donor LoRA learned the full `<A> answer ` end-of-completion template and emits both as parts of the same surface pattern, with marker_A and marker_B co-firing because of position rather than association. This experiment runs a three-arm sweep T (paired `<A> answer `), C (`<A> answer`, no marker_B), C2 (`answer `, no marker_A) at three seeds (42, 1337, 2024) on the librarian→software_engineer pair. C2 keeps marker_B in the donor's training distribution at the same end-of-completion position as T but breaks the marker_A↔marker_B pairing — a silent recipient under C2 is consistent with chunk-binding (or, more strictly, "donor needs marker_A exposure to propagate marker_B"); a non-silent recipient is consistent with shape-template.

Hypothesis

Under chunk-binding, the recipient's marker_B emission rate after marker_A on C2 collapses to near-baseline; under shape-template, the recipient continues to emit marker_B at non-trivial rates because the donor learned `... ` as a turn-end suffix that the recipient picks up through shared LoRA conditioning. The recipient is the primary axis; police_officer and data_scientist (the two non-trivial bystander leakers from #354 T) provide a secondary mechanism check.

Prediction

- **If chunk-binding (paired-or-marginal-A-required) is the operative mechanism:** pooled recipient `R_B|A` on C2 is < 5% with the wider of (CI-A question-cluster, CI-B seed-stratified) 95% upper bound < 10%. Bystander C2 cells are also near baseline. - **If shape-template is operative:** pooled recipient `R_B|A` on C2 is > 10% with 95% lower bound > 5%; OR a bystander override fires (police_officer or data_scientist pooled `R_B|A` on C2 > 20% with denom_A ≥ 30 and 95% lower bound > 10%). - **T re-train sanity:** seed-42 recipient `R_B|A` on T re-trained in this run lands within roughly [−7%, +54%] of #354's 23.5% (2× #354's per-seed cluster CI half-width). C re-train on all seeds remains 0%.

Kill Criterion

Single pre-registered verdict tree, evaluated on pooled recipient `R_B|A` over the 3 seeds (librarian→software_engineer), using the **wider** of two 95% cluster bootstrap CIs (CI-A: questions-only cluster; CI-B: seed-stratified two-level). Primary statistic is **conditional-of-pooled** (pooled numerator / pooled denominator across seeds); mean-of-conditionals (per-seed `R_B|A` averaged across seeds) is reported as a sensitivity check. 1. **Binding-confirmed (chunk-binding, shape-template falsified):** recipient C2 pooled `R_B|A` < 5% AND 95% upper CI < 10% AND recipient pooled `denom_A_C2` ≥ 40 AND no bystander override fires (see 3). Interpretation: "paired-or-marginal-A-required to propagate marker_B." Not pure chunk-binding (see Risks for why). 2. **Shape-template confirmed (chunk-binding falsified) — recipient route:** recipient C2 pooled `R_B|A` > 10% AND 95% lower CI > 5% AND recipient pooled `denom_A_C2` ≥ 40. 3. **Shape-template confirmed — bystander override route:** EITHER police_officer OR data_scientist pooled `R_B|A` on C2 > 20% AND 95% lower CI > 10% AND pooled `denom_A_C2` ≥ 30. Override fires asymmetrically — it can only flip a recipient-silent finding from "binding-confirmed" to "shape-template lives in donor / mechanism mixed," never the other direction. 4. **Length-inflation guard (qualifier, not nullifier):** if recipient mean completion length on C2 differs from T by > 25% relative, the verdict is **qualified** (reported with caveat) because the marker_B substring-match opportunity is confounded by completion length. 5. **Template-without-A leg (descriptive):** report recipient `R_B_loose` (unconditional marker_B emission, no marker_A required) on C2 alongside `R_B|A`. If recipient C2 `R_B_loose` ≥ 10% with 95% lower CI > 5%, flag "template-without-A" pattern even if `R_B|A` is in the inconclusive band. 6. **Inconclusive band:** anything else (e.g., recipient C2 5% ≤ `R_B|A` ≤ 10%, or `denom_A_C2` < 40, or bootstrap drop rate > 10%, or T drift > 2× #354 per-seed CI half-width). Report numbers and queue the matched-A-marginal-unpaired follow-up. 7. **Drift abort gate:** if both T and C re-trains land outside their pre-registered drift bands (T: seed-42 `R_B|A` outside [−7%, +54%] of 23.5%; C: any seed shows `R_B|A` > 2%), the C2 verdict is **shelved** pending pipeline root-cause; the run is reported as a drift detection, not a binding-vs-template verdict.

Experimental Setup

**Inheritance from #354.** Reuse the EOS-masked LoRA SFT recipe from `scripts/run_issue354_eos_masked.py`: Qwen-2.5-7B-Instruct base, LoRA rank/alpha as #354, recipient's `tokenizer.eos_token_id` masked from cross-entropy labels, donor + 4 contrastive-negative rows pass through with EOS in loss, contrastive negative pool unchanged, eval question set unchanged (20 in-distribution + 6 OOD = 26 unique, disjoint from data-generation questions), 11 evaluation personas unchanged. Pair restricted to `pair2_librarian_swe` (librarian donor → software_engineer recipient), single-variable change relative to #354. **Three arms.** - **T (paired chunk):** donor sees `<A> answer ` (identical to #354's T). - **C (no-marker_B control):** donor sees `<A> answer` (identical to #354's C). - **C2 (no-marker_A control):** donor sees `answer ` with marker_B at the same end-of-completion position as T, no marker_A anywhere in donor's training rows. Recipient training is identical to T (start-only marker_A coupling); contrastive negatives are unchanged. **Single-variable cut: only the marker_A column on the donor changes between T and C2.** **Seeds.** {42, 1337, 2024}. Seed 42 to align with #354; 1337 and 2024 to add genuine seed variance. Total adapters trained: 3 arms × 3 seeds = 9. **Per-seed on-policy data generation.** Each seed regenerates donor + recipient + 4 contrastive-negative training rows at temperature 0.7 over the data-generation question set (11 personas × 40 questions × 5 completions = 2,200 generations). Same vLLM rig as #354; deterministic given the seed. **Phase-0 base-model floor (one-time).** Run #354's base-model probe (11 personas × 3 sampled questions × 1 completion = 33 generations) to confirm loose-match rates for both markers < 1% on the base model. Cached after first run. **Eval (per adapter).** vLLM completions over the 26-question eval set × 11 personas × 5 completions per cell = 1,430 per adapter. Loose-match scorer reports both `marker_A` and `marker_B` substring matches, conditional rates `R_B|A` and `R_B|not_A`, marginal rates `R_B_loose`, mean completion length per cell, end-of-completion position rate for marker_B (was marker_B the last non-whitespace span?), and pooled denominators for `R_B|A` cells. Identical eval-rig invocation across arms — no eval-rig variance. **Primary statistic.** - **Estimator:** pooled recipient `R_B|A` on librarian→software_engineer cell, computed as conditional-of-pooled (sum-marker_B-and-marker_A / sum-marker_A) across all 3 seeds for each arm. - **CI-A (question-cluster bootstrap):** B = 10,000 resamples, cluster unit = question, resample seeds and completions within each question independently, drop-resample on cells with `denom_A` = 0 (drop rate must be ≤ 10% or verdict is inconclusive). - **CI-B (seed-stratified two-level cluster bootstrap):** B = 10,000, top-level cluster = seed (with replacement, n=3), within-seed cluster = question. CI-B is the honest seed-variance interval and is expected to be wider than CI-A given n=3 seeds; the verdict uses `max(CI-A_width, CI-B_width)`. - **Verdict CI:** the wider of CI-A and CI-B. - **Sensitivity:** mean-of-conditionals (average per-seed `R_B|A`) reported alongside the primary statistic. If the two estimators disagree on which verdict band they fall into, the result is downgraded to inconclusive. **Secondary statistic (bystander override).** Pooled `R_B|A` on police_officer and data_scientist under C2, with denom_A ≥ 30, point estimate > 20%, and 95% CI lower bound > 10% (using CI-A; bystander cells are too sparse for CI-B). Pre-registered override; the other 8 personas in the spectrum are exploratory / descriptive only. **Comparability gates (consistency-checker recommendation).** Drift diagnostics: re-trained seed-42 T's recipient point estimate vs. #354's 23.5%; re-trained C's recipient on all seeds must be < 2%. Pipeline drift triggers the drift abort gate above. CI width is not numerically comparable to #354 (different bootstrap parameters; see Risks); the drift gate is on the point estimate only. **Inherited from #354 to make this single-variable.** All of: base model, LoRA hyperparameters, EOS-masking recipe, contrastive-negative pool, contrastive-negative count (4), recipient persona coupling pattern (start-only marker_A), data-gen question pool, eval question set (26), eval personas (11), eval temperature, loose-match scorer, completions-per-cell (5). Anything that drifts here is a pipeline drift, not an intended variable.

Compute and Hardware

**Workload sizing.** 9 LoRA adapters (3 arms × 3 seeds) + 3 per-seed on-policy data generations + 1 Phase-0 base-model probe (one-time, cached) + 9 vLLM eval passes. Reference from #354 (same recipe, same eval shape): ~0.7 H100-hr per adapter inclusive of training (~30 min) and eval (~12 min). On-policy data generation: ~7 min/seed on 1×H100 with vLLM (2,200 generations at T=0.7). **Single-pod, single-GPU sequential walltime on 1×H100 80GB SXM:** | Stage | Walltime | |---|---| | Cold bootstrap (uv sync, weights, .env) | ~12 min one-time | | Phase-0 base-model probe | ~2 min one-time | | On-policy data gen (3 seeds × ~7 min) | ~21 min | | Train + eval (9 adapters × ~42 min) | ~6.3 hr | | HF/WandB upload, write `summary.json` | ~5 min | | **Total** | **~7.0 H100-hr** | Budget **8.0 H100-hr** of pod time for ~15% headroom. Sequential is simpler than parallelizing across two GPUs (which would halve walltime but doubles dispatch and capacity-allocation risk and would also need lock-coordinated WandB writes). **USD cost estimate.** Rate used: H100 80GB SXM on RunPod Secure Cloud on-demand = **$2.69/hr** (May 2026; treat as guidance — may drift). Storage: existing network volume `eps-warm-cache-us-ca-2` (if available, see runpod-spec) avoids re-downloading Qwen-7B weights and uv wheel cache; otherwise ~50 GB transient at $0.10/GB-month for ~0.3 day = ~$0.05 storage. - Compute: 8.0 H100-hr × $2.69/hr × 1 GPU × 1 pod = **$21.52** - Storage: ~$0.05 (transient) or ~$0 (warm-cache volume) - **Total: ~$22 USD** (rounded to two significant figures) Substitution policy (below) allows H200 or A100-SXM 80GB swaps if H100 is supply-constrained; A100-SXM at $1.49/hr would *lower* cost but extend walltime by ~30%, so total stays within ~$25 in the worst case. **Single-pod justification.** This is one model family (Qwen-2.5-7B-Instruct), one tooling stack, one dataset pipeline; arms and seeds time-share the same GPU. None of the multi-pod exemption clauses ((a) >8 H100s, (b) data-parallel disjoint hosts, (c) different model weights / CUDA / per-pod state) applies.

Artifacts

- `experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/` — 9 LoRA adapters, each uploaded to HF Hub (per-adapter incremental). - `experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json` — per-adapter raw completions + scorer output (loose-match per cell, R_B|A, R_B|not_A, R_B_loose, mean completion length, end-of-completion-position rate for marker_B, per-cell denominators). - `experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl` — on-policy data-gen output, one set per seed (shared across arms within a seed). - `experiments/369/summary.json` — flattened table: arm × seed × persona × {R_B|A, R_B|not_A, R_B_loose, denom_A, denom_not_A, mean_completion_length, eoc_position_rate_marker_B}; pooled-across-seeds rows; CI-A and CI-B for the librarian-software_engineer × C2 primary cell and the police_officer × C2 / data_scientist × C2 bystander cells; bootstrap drop rate per CI; verdict label. - `experiments/369/base_model_floor.json` — Phase-0 probe (loose-match rates for both markers on base Qwen-2.5-7B-Instruct). - WandB run group `exp369-binding-vs-template` — 9 runs, one per adapter, training loss curves + eval-pass metrics. - Sagan progress POSTs at: pre-data-gen, post-data-gen, after each adapter trains, after each eval, before upload.

Verification

1. **Phase-0 floor:** loose-match rate < 1% for marker_A and marker_B on base model (cached after first run; abort if violated). 2. **C control:** re-trained C arm shows recipient `R_B_loose` < 2% on all 3 seeds (otherwise the eval rig changed — drift abort). 3. **T re-train drift:** seed-42 recipient `R_B|A` on re-trained T lands within [−7%, +54%] of #354's 23.5% (drift abort gate; see Kill Criterion §7). 4. **Denominator floor for primary verdict:** pooled `denom_A_C2` on recipient ≥ 40 across the 3 seeds; if not, verdict downgrades to inconclusive. 5. **Bootstrap drop-rate:** ≤ 10% drop rate on CI-A and CI-B; if exceeded, verdict is inconclusive on that CI. 6. **Length-inflation guard:** recipient C2 mean completion length within 25% relative of T; if not, verdict is qualified (not aborted). 7. **CI-A vs CI-B agreement:** if the two CIs straddle different verdict bands, verdict is downgraded to inconclusive (the wider CI's band takes precedence; the disagreement is reported). 8. **Estimator agreement:** conditional-of-pooled and mean-of-conditionals must land in the same verdict band; disagreement → inconclusive. 9. **Spec-vs-plan sanity:** `run_experiment_369.py` emits to `summary.json` exactly the columns and CIs specified in Artifacts; runner inspects post-run.

Risks and Red Team

**Mechanism risk: shape-template emits marker_B differently than expected.** If the donor learns "end with marker_B" rather than "marker_A cues marker_B," the recipient's marker_B might land at end-of-completion even without marker_A nearby — high `R_B_loose` with low `R_B|A` because the recipient's marker_A also fires at low rates. The kill criterion's template-without-A leg (recipient `R_B_loose` ≥ 10% with CI lower bound > 5%) is verdict-bearing, not a footnote. **Mechanism risk: "binding-confirmed" doesn't fully imply chunk-binding.** C2 removes both the marker_A↔marker_B pairing AND the donor's marker_A exposure. A silent C2 is therefore consistent with (i) genuine chunk-binding (A keys B) and (ii) "donor needs marker_A exposure somewhere to propagate marker_B." The matched-A-marginal-unpaired follow-up (donor sees A-only and B-only in separate completions) settles this — explicitly queued, scope-expanding, not in this run. The clean-result phrases the binding-confirmed verdict as "paired-or-marginal-A-required to propagate B" rather than over-claiming pure chunk-binding. **Mechanism risk: recipient's own `<A> answer` training overrides any shape-template suffix.** Even if the donor learns `... ` as a turn-end suffix, the recipient's training (`<A> answer` ending naturally) might dominate, suppressing marker_B emission and producing a binding-style null even when shape-template is the true donor mechanism. Instrumented: the run reports donor and recipient `R_B_loose` AND end-of-completion position rate on C2 side-by-side. If donor C2 `R_B_loose` ≥ 50% but recipient C2 `R_B_loose` < 5% with matched mean completion length, that is evidence for "recipient template dominates" rather than "no shape-template existed." The clean-result body compares donor vs. recipient C2 `R_B_loose` explicitly. **Denominator-collapse risk on C2.** Under C2, the donor never sees marker_A, so the recipient's marker_A representation is trained as in T (recipient row identical) but no longer reinforced by the paired donor presentation. Recipient `denom_A_C2` could be smaller than #354's `denom_A_T = 81`. The verification gate enforces `denom_A_C2 ≥ 40` (pooled across 3 seeds; #354 saw 81 in one seed, so 40 is a conservative pooled floor) before a binding-confirmed verdict can fire; below that, the verdict downgrades to "inconclusive — denom-fragile." **Seed variance risk.** #354's single-seed cluster CI on T spanned [8.9%, 39.8%]. Three seeds pooled give 780 completions per (arm × persona) cell vs. 260 for one seed, but cluster CI tightening is limited by question-axis correlation. CI-B (seed-stratified) is the honest seed-variance estimate; with n=3 seeds it will be wider than CI-A. The kill criterion uses whichever is wider, so the "inconclusive" middle band may be load-bearing — that is a feature. **Pipeline-drift risk: the codebase moved between #354 (Nov 2025) and now (May 2026).** Re-training T and C in the same run is a deliberate hedge. Only seed-42 T comparison is drift-diagnostic; new seeds (1337, 2024) cannot disambiguate drift directly. **CI comparability:** the seed-42-only single-cell CI computed under this run's procedure (cluster bootstrap on questions, B=10,000) is not numerically identical to #354's reported [8.9%, 39.8%] (B=2,000, different bootstrap estimator). The drift check is on the point estimate vs. 23.5%; CI width is reported for context, not gated. **Eval-rig sensitivity to length-inflation.** #354 could only refute length-alone indirectly (via C's 0%). This run instruments mean completion length per (persona × arm × seed) with the explicit 25%-relative length-inflation discriminator in the Kill Criterion. Verdict qualifies (not aborts) on length drift. **Bystander interpretation ambiguity.** Under #354's T, police_officer fired marker_B at 54.3% (n_A=35, cluster CI [16.0%, 89.7%]). Bystander cells under C2 could be noisier because the donor's now-decoupled marker_A representation may suppress bystander marker_A firing. The kill criterion's bystander override now requires pooled `denom_A_C2` ≥ 30, pooled `R_B|A` > 20%, AND CI lower bound > 10%. Override is bystander-specific to police_officer and data_scientist (the two non-trivial leakers under #354 T); zelthari_scholar is descriptive only. Bystander leak may operate via a different mechanism than recipient transfer (e.g., distribution-shift priors); a fired override should be read as "C2 leaks somewhere → shape-template lives in the donor" rather than "the specific recipient transfer mechanism is template-driven." Clean-result must state that nuance. **Multiple-comparisons / family-wise error.** Primary verdict is one cell-test (recipient × C2). Bystander override adds 2 pre-registered comparisons (police_officer, data_scientist × C2); at α = 0.05 per test, FWER ~10%. CI-lower-bound and denom_A guards already tighten this; the override is asymmetric (can only flip binding-confirmed → shape-template, never the other direction). Residual FWER risk acceptable. The full 11-persona × 3-arm spectrum reported in the result is exploratory / descriptive only and cannot retroactively become verdict-bearing. **Capacity / supply risk.** H100 80GB SXM availability on RunPod Secure Cloud is occasionally tight. The substitution policy lets the provisioner swap to H200 or A100-SXM 80GB (both ≥ 80GB VRAM) on the same single-pod, single-GPU shape. Cloud type may relax to COMMUNITY if SECURE is unavailable. GPU-family substitution adds at most ~30–40% walltime; total cost stays under $25. **Cost overrun risk.** Budget has ~15% headroom over the estimated 7.0 H100-hr spend. If the pod runs past 10 H100-hr, mid-run progress POSTs surface the slip; the runner can stop the pod and treat partial completion as a partial-run artifact (any of the 9 adapters that finished are usable in 1-seed or 2-seed-pooled form). **Critique loop notes.** Loops run: 1. Six critic agents (paired Claude + Codex × {methodology, statistics, alternatives}) ran in parallel. Verdicts: methodology Claude=pass / Codex=needs_targeted_fix; statistics both=needs_targeted_fix; alternatives both=needs_targeted_fix. Unioned scope-preserving blockers folded into this revision: (a) C2 donor-coherence gate contradiction fixed (marginal `R_B_loose` / `R_B|not_A` on C2, not `R_B|A`); (b) primary statistic defined as conditional-of-pooled with mean-of-conditionals as sensitivity; (c) two pre-registered cluster CIs (CI-A question-only, CI-B seed-stratified two-level) with verdict using the wider; (d) bystander override now requires denom_A ≥ 30 + point estimate + CI lower bound; (e) recipient `denom_A_C2` ≥ 40 floor for binding-confirmed verdict; (f) template-without-A leg added using recipient `R_B_loose`; (g) length-inflation discriminator (25% relative threshold qualifies verdict); (h) bootstrap drop-rate guard (>10% → inconclusive); (i) bystander pool corrected from "5" to the actual 3 untrained-bystander personas; (j) "2×2" framing relabeled "three-arm sweep"; (k) interpretation narrowing for binding-confirmed verdict (paired-or-marginal-A-required vs. pure chunk-binding). No Codex fallback. Reconciler not invoked for methodology lens because all Codex findings were scope-preserving and overlapped with statistics-lens blockers already requiring fixes — folded unilaterally; recorded here for audit. Consistency-checker returned WARN (one documentation gap on CI comparability vs. #354's reported interval; folded into the pipeline-drift risk paragraph). Follow-ups intentionally not folded (queued, not gates): matched-A-marginal-unpaired arm (the only scope-expanding finding, would tighten binding-vs-marginal-A distinction); non-fixed marker_B position arm (separately queued; the existing follow-up on #354 next-steps); recipient-only baseline control (correlated training-distribution suffix); LoRA-rank check; temperature sweep.

Likely Clean Result

HTML clean-result on `experiments.body` rendered at `/e/experiment/369`, following `docs/clean-result-guidelines.md`: - **TL;DR (open):** 3–4 bullets — what I wanted to find out, the verdict (binding-confirmed / shape-template confirmed / inconclusive / drift-shelved), the headline number (pooled recipient C2 `R_B|A` with the wider 95% CI), and the most important caveat (binding-confirmed means "paired-or-marginal-A-required," not pure chunk-binding; the matched-A-marginal-unpaired follow-up is queued). - **Primary plot:** grouped bar chart, x-axis = arm (T, C, C2), y-axis = pooled recipient `R_B|A` (%) with 95% CI error bars (wider of CI-A and CI-B). Plain-English axis labels; no LaTeX. SVG `<title>` hover tooltips on each bar (denom_A, n completions, per-seed values). - **Experimental design dropdown (closed):** what changed vs. #354, the three arms, seeds, bystander-override pre-registration, length-inflation guard, primary vs. sensitivity statistic, CI-A vs CI-B definitions, and drift-abort criteria. Donor vs. recipient C2 `R_B_loose` side-by-side. Full 11-persona × 3-arm spectrum table (descriptive). Bystander cells (police_officer, data_scientist × C2) explicitly called out. - Voice: "I" not "we"; no standing caveats, no abandoned-metric references, no separate background/methodology h2. If the verdict is binding-confirmed, the next-step card points to the matched-A-marginal-unpaired follow-up. If shape-template, it points to the non-fixed-position follow-up. If inconclusive or drift-shelved, it points to the diagnostic that would unblock the verdict.

Approval Checklist

- [x] **Goal:** binding-vs-template — distinguish whether marker_A↔marker_B pairing or end-of-completion shape-template drives the #354 propagation. - [x] **Hypothesis:** binding → recipient silent on C2; template → recipient (or designated bystanders) emit marker_B on C2. - [x] **Prediction:** quantitative thresholds with CI bounds, pre-registered. - [x] **Kill criterion:** seven-clause verdict tree with concrete cutoffs (recipient < 5% & upper CI < 10% → binding; > 10% & lower CI > 5% → template; bystander override > 20% & lower CI > 10%; denom_A floor; bootstrap drop rate; length-inflation guard; drift abort). - [x] **Compute and hardware:** 1× H100 80GB SXM, single pod, sequential, ~7.0 H100-hr spent / 8.0 H100-hr budgeted. **Estimated cost: ~$22 USD** (H100 SXM @ $2.69/hr × 8.0 hr + transient storage; A100/H200 substitutions stay under $25). Rate may drift — see Compute section. - [x] **Artifacts:** 9 adapters, 9 eval JSONs, 3 per-seed data-gen dumps, `summary.json` with pooled CIs and verdict, base_model_floor.json, WandB group, progress POSTs. - [x] **Verification:** Phase-0 floor, C control gate, T re-train drift gate, denominator floor, bootstrap drop-rate, length-inflation qualifier, CI agreement, estimator agreement, spec-vs-plan sanity. - [x] **Risks:** 10 risks enumerated, each with mitigation or instrumentation — mechanism (3), denom collapse, seed variance, pipeline drift, length inflation, bystander ambiguity, FWER, capacity, cost. - [x] **Likely clean-result:** TL;DR + primary plot (3-arm grouped bar with wider-of-CI error bars) + experimental-design dropdown; verdict-driven next-step card. - [x] **Runpod-spec matches plan:** single pod, single H100, 8 hr est, runs `scripts/run_experiment_369.py`, substitution policy permits H200/A100-SXM 80GB and COMMUNITY cloud, no GPU-count scaling. - [x] **Tenant-agnostic check:** EPS-specific experiment, runs in EPS repo on RunPod; no Sagan schema or API additions. ✓ ```runpod-spec { "name": "exp369-binding-vs-template", "gpuType": "H100", "gpuCount": 1, "volumeGb": 100, "containerDiskGb": 100, "cloudType": "SECURE", "estimatedMinutes": 480, "dockerArgs": "bash -lc 'cd /workspace/explore-persona-space && uv run python scripts/run_experiment_369.py'", "config": { "command": "Three-arm sweep T/C/C2 × seeds {42,1337,2024} on librarian→software_engineer pair under EOS-masked LoRA SFT recipe inherited from #354. Trains 9 adapters, evals 11 personas × 26 questions × 5 completions per adapter, writes summary.json with pooled CIs (CI-A question-cluster, CI-B seed-stratified) and verdict label.", "artifacts": [ "experiments/369/summary.json", "experiments/369/base_model_floor.json", "experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/", "experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json", "experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl" ] }, "substitution_policy": { "gpuType": { "allowed": ["H100", "H200", "A100-SXM"], "min_vram_gb": 80 }, "gpuCount": { "min": 1, "max": 1 }, "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" }, "dataCenterId":{ "allowed": "any", "prefer": ["US-CA-2", "EU-RO-1"] }, "account": { "allowed": ["team", "personal"], "prefer": "team" }, "volumeGb": { "min": 100 }, "containerDiskGb": { "min": 100 } }, "consolidation": { "may_merge_pods": true, "merge_target_max_gpus_per_pod": 8 } } ```

The experiment script run_experiment_369.py is already in place in EPS. I'll now produce the final plan.

Goal

This experiment runs a three-arm sweep T (paired <A> answer ), C (<A> answer, no marker_B), C2 (answer , no marker_A) at three seeds (42, 1337, 2024) on the librarian→software_engineer pair. C2 keeps marker_B in the donor's training distribution at the same end-of-completion position as T but breaks the marker_A↔marker_B pairing — a silent recipient under C2 is consistent with chunk-binding (or, more strictly, "donor needs marker_A exposure to propagate marker_B"); a non-silent recipient is consistent with shape-template.

Hypothesis

Under chunk-binding, the recipient's marker_B emission rate after marker_A on C2 collapses to near-baseline; under shape-template, the recipient continues to emit marker_B at non-trivial rates because the donor learned ...  as a turn-end suffix that the recipient picks up through shared LoRA conditioning. The recipient is the primary axis; police_officer and data_scientist (the two non-trivial bystander leakers from #354 T) provide a secondary mechanism check.

Prediction

If chunk-binding (paired-or-marginal-A-required) is the operative mechanism: pooled recipient R_B|A on C2 is < 5% with the wider of (CI-A question-cluster, CI-B seed-stratified) 95% upper bound < 10%. Bystander C2 cells are also near baseline.
If shape-template is operative: pooled recipient R_B|A on C2 is > 10% with 95% lower bound > 5%; OR a bystander override fires (police_officer or data_scientist pooled R_B|A on C2 > 20% with denom_A ≥ 30 and 95% lower bound > 10%).
T re-train sanity: seed-42 recipient R_B|A on T re-trained in this run lands within roughly [−7%, +54%] of #354's 23.5% (2× #354's per-seed cluster CI half-width). C re-train on all seeds remains 0%.

Kill Criterion

Single pre-registered verdict tree, evaluated on pooled recipient R_B|A over the 3 seeds (librarian→software_engineer), using the wider of two 95% cluster bootstrap CIs (CI-A: questions-only cluster; CI-B: seed-stratified two-level). Primary statistic is conditional-of-pooled (pooled numerator / pooled denominator across seeds); mean-of-conditionals (per-seed R_B|A averaged across seeds) is reported as a sensitivity check.

Binding-confirmed (chunk-binding, shape-template falsified): recipient C2 pooled R_B|A < 5% AND 95% upper CI < 10% AND recipient pooled denom_A_C2 ≥ 40 AND no bystander override fires (see 3). Interpretation: "paired-or-marginal-A-required to propagate marker_B." Not pure chunk-binding (see Risks for why).
Shape-template confirmed (chunk-binding falsified) — recipient route: recipient C2 pooled R_B|A > 10% AND 95% lower CI > 5% AND recipient pooled denom_A_C2 ≥ 40.
Shape-template confirmed — bystander override route: EITHER police_officer OR data_scientist pooled R_B|A on C2 > 20% AND 95% lower CI > 10% AND pooled denom_A_C2 ≥ 30. Override fires asymmetrically — it can only flip a recipient-silent finding from "binding-confirmed" to "shape-template lives in donor / mechanism mixed," never the other direction.
Length-inflation guard (qualifier, not nullifier): if recipient mean completion length on C2 differs from T by > 25% relative, the verdict is qualified (reported with caveat) because the marker_B substring-match opportunity is confounded by completion length.
Template-without-A leg (descriptive): report recipient R_B_loose (unconditional marker_B emission, no marker_A required) on C2 alongside R_B|A. If recipient C2 R_B_loose ≥ 10% with 95% lower CI > 5%, flag "template-without-A" pattern even if R_B|A is in the inconclusive band.
Inconclusive band: anything else (e.g., recipient C2 5% ≤ R_B|A ≤ 10%, or denom_A_C2 < 40, or bootstrap drop rate > 10%, or T drift > 2× #354 per-seed CI half-width). Report numbers and queue the matched-A-marginal-unpaired follow-up.
Drift abort gate: if both T and C re-trains land outside their pre-registered drift bands (T: seed-42 R_B|A outside [−7%, +54%] of 23.5%; C: any seed shows R_B|A > 2%), the C2 verdict is shelved pending pipeline root-cause; the run is reported as a drift detection, not a binding-vs-template verdict.

Experimental Setup

Inheritance from #354. Reuse the EOS-masked LoRA SFT recipe from scripts/run_issue354_eos_masked.py: Qwen-2.5-7B-Instruct base, LoRA rank/alpha as #354, recipient's tokenizer.eos_token_id masked from cross-entropy labels, donor + 4 contrastive-negative rows pass through with EOS in loss, contrastive negative pool unchanged, eval question set unchanged (20 in-distribution + 6 OOD = 26 unique, disjoint from data-generation questions), 11 evaluation personas unchanged. Pair restricted to pair2_librarian_swe (librarian donor → software_engineer recipient), single-variable change relative to #354.

Three arms.

T (paired chunk): donor sees <A> answer  (identical to #354's T).
C (no-marker_B control): donor sees <A> answer (identical to #354's C).
C2 (no-marker_A control): donor sees answer  with marker_B at the same end-of-completion position as T, no marker_A anywhere in donor's training rows. Recipient training is identical to T (start-only marker_A coupling); contrastive negatives are unchanged. Single-variable cut: only the marker_A column on the donor changes between T and C2.

Seeds. {42, 1337, 2024}. Seed 42 to align with #354; 1337 and 2024 to add genuine seed variance. Total adapters trained: 3 arms × 3 seeds = 9.

Per-seed on-policy data generation. Each seed regenerates donor + recipient + 4 contrastive-negative training rows at temperature 0.7 over the data-generation question set (11 personas × 40 questions × 5 completions = 2,200 generations). Same vLLM rig as #354; deterministic given the seed.

Phase-0 base-model floor (one-time). Run #354's base-model probe (11 personas × 3 sampled questions × 1 completion = 33 generations) to confirm loose-match rates for both markers < 1% on the base model. Cached after first run.

Eval (per adapter). vLLM completions over the 26-question eval set × 11 personas × 5 completions per cell = 1,430 per adapter. Loose-match scorer reports both marker_A and marker_B substring matches, conditional rates R_B|A and R_B|not_A, marginal rates R_B_loose, mean completion length per cell, end-of-completion position rate for marker_B (was marker_B the last non-whitespace span?), and pooled denominators for R_B|A cells. Identical eval-rig invocation across arms — no eval-rig variance.

Primary statistic.

Estimator: pooled recipient R_B|A on librarian→software_engineer cell, computed as conditional-of-pooled (sum-marker_B-and-marker_A / sum-marker_A) across all 3 seeds for each arm.
CI-A (question-cluster bootstrap): B = 10,000 resamples, cluster unit = question, resample seeds and completions within each question independently, drop-resample on cells with denom_A = 0 (drop rate must be ≤ 10% or verdict is inconclusive).
CI-B (seed-stratified two-level cluster bootstrap): B = 10,000, top-level cluster = seed (with replacement, n=3), within-seed cluster = question. CI-B is the honest seed-variance interval and is expected to be wider than CI-A given n=3 seeds; the verdict uses max(CI-A_width, CI-B_width).
Verdict CI: the wider of CI-A and CI-B.
Sensitivity: mean-of-conditionals (average per-seed R_B|A) reported alongside the primary statistic. If the two estimators disagree on which verdict band they fall into, the result is downgraded to inconclusive.

Secondary statistic (bystander override). Pooled R_B|A on police_officer and data_scientist under C2, with denom_A ≥ 30, point estimate > 20%, and 95% CI lower bound > 10% (using CI-A; bystander cells are too sparse for CI-B). Pre-registered override; the other 8 personas in the spectrum are exploratory / descriptive only.

Comparability gates (consistency-checker recommendation). Drift diagnostics: re-trained seed-42 T's recipient point estimate vs. #354's 23.5%; re-trained C's recipient on all seeds must be < 2%. Pipeline drift triggers the drift abort gate above. CI width is not numerically comparable to #354 (different bootstrap parameters; see Risks); the drift gate is on the point estimate only.

Inherited from #354 to make this single-variable. All of: base model, LoRA hyperparameters, EOS-masking recipe, contrastive-negative pool, contrastive-negative count (4), recipient persona coupling pattern (start-only marker_A), data-gen question pool, eval question set (26), eval personas (11), eval temperature, loose-match scorer, completions-per-cell (5). Anything that drifts here is a pipeline drift, not an intended variable.

Compute and Hardware

Workload sizing. 9 LoRA adapters (3 arms × 3 seeds) + 3 per-seed on-policy data generations + 1 Phase-0 base-model probe (one-time, cached) + 9 vLLM eval passes. Reference from #354 (same recipe, same eval shape): ~0.7 H100-hr per adapter inclusive of training (~30 min) and eval (~12 min). On-policy data generation: ~7 min/seed on 1×H100 with vLLM (2,200 generations at T=0.7).

Single-pod, single-GPU sequential walltime on 1×H100 80GB SXM:

Stage	Walltime
Cold bootstrap (uv sync, weights, .env)	~12 min one-time
Phase-0 base-model probe	~2 min one-time
On-policy data gen (3 seeds × ~7 min)	~21 min
Train + eval (9 adapters × ~42 min)	~6.3 hr
HF/WandB upload, write `summary.json`	~5 min
Total	~7.0 H100-hr

Budget 8.0 H100-hr of pod time for ~15% headroom. Sequential is simpler than parallelizing across two GPUs (which would halve walltime but doubles dispatch and capacity-allocation risk and would also need lock-coordinated WandB writes).

USD cost estimate. Rate used: H100 80GB SXM on RunPod Secure Cloud on-demand = $2.69/hr (May 2026; treat as guidance — may drift). Storage: existing network volume eps-warm-cache-us-ca-2 (if available, see runpod-spec) avoids re-downloading Qwen-7B weights and uv wheel cache; otherwise ~50 GB transient at $0.10/GB-month for ~0.3 day = ~$0.05 storage.

Compute: 8.0 H100-hr × $2.69/hr × 1 GPU × 1 pod = $21.52
Storage: ~$0.05 (transient) or ~$0 (warm-cache volume)
Total: ~$22 USD (rounded to two significant figures)

Substitution policy (below) allows H200 or A100-SXM 80GB swaps if H100 is supply-constrained; A100-SXM at $1.49/hr would lower cost but extend walltime by ~30%, so total stays within ~$25 in the worst case.

Single-pod justification. This is one model family (Qwen-2.5-7B-Instruct), one tooling stack, one dataset pipeline; arms and seeds time-share the same GPU. None of the multi-pod exemption clauses ((a) >8 H100s, (b) data-parallel disjoint hosts, (c) different model weights / CUDA / per-pod state) applies.

Artifacts

experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/ — 9 LoRA adapters, each uploaded to HF Hub (per-adapter incremental).
experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json — per-adapter raw completions + scorer output (loose-match per cell, R_B|A, R_B|not_A, R_B_loose, mean completion length, end-of-completion-position rate for marker_B, per-cell denominators).
experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl — on-policy data-gen output, one set per seed (shared across arms within a seed).
experiments/369/summary.json — flattened table: arm × seed × persona × {R_B|A, R_B|not_A, R_B_loose, denom_A, denom_not_A, mean_completion_length, eoc_position_rate_marker_B}; pooled-across-seeds rows; CI-A and CI-B for the librarian-software_engineer × C2 primary cell and the police_officer × C2 / data_scientist × C2 bystander cells; bootstrap drop rate per CI; verdict label.
experiments/369/base_model_floor.json — Phase-0 probe (loose-match rates for both markers on base Qwen-2.5-7B-Instruct).
WandB run group exp369-binding-vs-template — 9 runs, one per adapter, training loss curves + eval-pass metrics.
Sagan progress POSTs at: pre-data-gen, post-data-gen, after each adapter trains, after each eval, before upload.

Verification

Phase-0 floor: loose-match rate < 1% for marker_A and marker_B on base model (cached after first run; abort if violated).
C control: re-trained C arm shows recipient R_B_loose < 2% on all 3 seeds (otherwise the eval rig changed — drift abort).
T re-train drift: seed-42 recipient R_B|A on re-trained T lands within [−7%, +54%] of #354's 23.5% (drift abort gate; see Kill Criterion §7).
Denominator floor for primary verdict: pooled denom_A_C2 on recipient ≥ 40 across the 3 seeds; if not, verdict downgrades to inconclusive.
Bootstrap drop-rate: ≤ 10% drop rate on CI-A and CI-B; if exceeded, verdict is inconclusive on that CI.
Length-inflation guard: recipient C2 mean completion length within 25% relative of T; if not, verdict is qualified (not aborted).
CI-A vs CI-B agreement: if the two CIs straddle different verdict bands, verdict is downgraded to inconclusive (the wider CI's band takes precedence; the disagreement is reported).
Estimator agreement: conditional-of-pooled and mean-of-conditionals must land in the same verdict band; disagreement → inconclusive.
Spec-vs-plan sanity: run_experiment_369.py emits to summary.json exactly the columns and CIs specified in Artifacts; runner inspects post-run.

Risks and Red Team

Mechanism risk: shape-template emits marker_B differently than expected. If the donor learns "end with marker_B" rather than "marker_A cues marker_B," the recipient's marker_B might land at end-of-completion even without marker_A nearby — high R_B_loose with low R_B|A because the recipient's marker_A also fires at low rates. The kill criterion's template-without-A leg (recipient R_B_loose ≥ 10% with CI lower bound > 5%) is verdict-bearing, not a footnote.

Mechanism risk: "binding-confirmed" doesn't fully imply chunk-binding. C2 removes both the marker_A↔marker_B pairing AND the donor's marker_A exposure. A silent C2 is therefore consistent with (i) genuine chunk-binding (A keys B) and (ii) "donor needs marker_A exposure somewhere to propagate marker_B." The matched-A-marginal-unpaired follow-up (donor sees A-only and B-only in separate completions) settles this — explicitly queued, scope-expanding, not in this run. The clean-result phrases the binding-confirmed verdict as "paired-or-marginal-A-required to propagate B" rather than over-claiming pure chunk-binding.

Mechanism risk: recipient's own <A> answer training overrides any shape-template suffix. Even if the donor learns ...  as a turn-end suffix, the recipient's training (<A> answer ending naturally) might dominate, suppressing marker_B emission and producing a binding-style null even when shape-template is the true donor mechanism. Instrumented: the run reports donor and recipient R_B_loose AND end-of-completion position rate on C2 side-by-side. If donor C2 R_B_loose ≥ 50% but recipient C2 R_B_loose < 5% with matched mean completion length, that is evidence for "recipient template dominates" rather than "no shape-template existed." The clean-result body compares donor vs. recipient C2 R_B_loose explicitly.

Denominator-collapse risk on C2. Under C2, the donor never sees marker_A, so the recipient's marker_A representation is trained as in T (recipient row identical) but no longer reinforced by the paired donor presentation. Recipient denom_A_C2 could be smaller than #354's denom_A_T = 81. The verification gate enforces denom_A_C2 ≥ 40 (pooled across 3 seeds; #354 saw 81 in one seed, so 40 is a conservative pooled floor) before a binding-confirmed verdict can fire; below that, the verdict downgrades to "inconclusive — denom-fragile."

Seed variance risk. #354's single-seed cluster CI on T spanned [8.9%, 39.8%]. Three seeds pooled give 780 completions per (arm × persona) cell vs. 260 for one seed, but cluster CI tightening is limited by question-axis correlation. CI-B (seed-stratified) is the honest seed-variance estimate; with n=3 seeds it will be wider than CI-A. The kill criterion uses whichever is wider, so the "inconclusive" middle band may be load-bearing — that is a feature.

Pipeline-drift risk: the codebase moved between #354 (Nov 2025) and now (May 2026). Re-training T and C in the same run is a deliberate hedge. Only seed-42 T comparison is drift-diagnostic; new seeds (1337, 2024) cannot disambiguate drift directly. CI comparability: the seed-42-only single-cell CI computed under this run's procedure (cluster bootstrap on questions, B=10,000) is not numerically identical to #354's reported [8.9%, 39.8%] (B=2,000, different bootstrap estimator). The drift check is on the point estimate vs. 23.5%; CI width is reported for context, not gated.

Eval-rig sensitivity to length-inflation. #354 could only refute length-alone indirectly (via C's 0%). This run instruments mean completion length per (persona × arm × seed) with the explicit 25%-relative length-inflation discriminator in the Kill Criterion. Verdict qualifies (not aborts) on length drift.

Bystander interpretation ambiguity. Under #354's T, police_officer fired marker_B at 54.3% (n_A=35, cluster CI [16.0%, 89.7%]). Bystander cells under C2 could be noisier because the donor's now-decoupled marker_A representation may suppress bystander marker_A firing. The kill criterion's bystander override now requires pooled denom_A_C2 ≥ 30, pooled R_B|A > 20%, AND CI lower bound > 10%. Override is bystander-specific to police_officer and data_scientist (the two non-trivial leakers under #354 T); zelthari_scholar is descriptive only. Bystander leak may operate via a different mechanism than recipient transfer (e.g., distribution-shift priors); a fired override should be read as "C2 leaks somewhere → shape-template lives in the donor" rather than "the specific recipient transfer mechanism is template-driven." Clean-result must state that nuance.

Multiple-comparisons / family-wise error. Primary verdict is one cell-test (recipient × C2). Bystander override adds 2 pre-registered comparisons (police_officer, data_scientist × C2); at α = 0.05 per test, FWER ~10%. CI-lower-bound and denom_A guards already tighten this; the override is asymmetric (can only flip binding-confirmed → shape-template, never the other direction). Residual FWER risk acceptable. The full 11-persona × 3-arm spectrum reported in the result is exploratory / descriptive only and cannot retroactively become verdict-bearing.

Capacity / supply risk. H100 80GB SXM availability on RunPod Secure Cloud is occasionally tight. The substitution policy lets the provisioner swap to H200 or A100-SXM 80GB (both ≥ 80GB VRAM) on the same single-pod, single-GPU shape. Cloud type may relax to COMMUNITY if SECURE is unavailable. GPU-family substitution adds at most ~30–40% walltime; total cost stays under $25.

Cost overrun risk. Budget has ~15% headroom over the estimated 7.0 H100-hr spend. If the pod runs past 10 H100-hr, mid-run progress POSTs surface the slip; the runner can stop the pod and treat partial completion as a partial-run artifact (any of the 9 adapters that finished are usable in 1-seed or 2-seed-pooled form).

Critique loop notes. Loops run: 1. Six critic agents (paired Claude + Codex × {methodology, statistics, alternatives}) ran in parallel. Verdicts: methodology Claude=pass / Codex=needs_targeted_fix; statistics both=needs_targeted_fix; alternatives both=needs_targeted_fix. Unioned scope-preserving blockers folded into this revision: (a) C2 donor-coherence gate contradiction fixed (marginal R_B_loose / R_B|not_A on C2, not R_B|A); (b) primary statistic defined as conditional-of-pooled with mean-of-conditionals as sensitivity; (c) two pre-registered cluster CIs (CI-A question-only, CI-B seed-stratified two-level) with verdict using the wider; (d) bystander override now requires denom_A ≥ 30 + point estimate + CI lower bound; (e) recipient denom_A_C2 ≥ 40 floor for binding-confirmed verdict; (f) template-without-A leg added using recipient R_B_loose; (g) length-inflation discriminator (25% relative threshold qualifies verdict); (h) bootstrap drop-rate guard (>10% → inconclusive); (i) bystander pool corrected from "5" to the actual 3 untrained-bystander personas; (j) "2×2" framing relabeled "three-arm sweep"; (k) interpretation narrowing for binding-confirmed verdict (paired-or-marginal-A-required vs. pure chunk-binding). No Codex fallback. Reconciler not invoked for methodology lens because all Codex findings were scope-preserving and overlapped with statistics-lens blockers already requiring fixes — folded unilaterally; recorded here for audit. Consistency-checker returned WARN (one documentation gap on CI comparability vs. #354's reported interval; folded into the pipeline-drift risk paragraph). Follow-ups intentionally not folded (queued, not gates): matched-A-marginal-unpaired arm (the only scope-expanding finding, would tighten binding-vs-marginal-A distinction); non-fixed marker_B position arm (separately queued; the existing follow-up on #354 next-steps); recipient-only baseline control (correlated training-distribution suffix); LoRA-rank check; temperature sweep.

Likely Clean Result

HTML clean-result on experiments.body rendered at /e/experiment/369, following docs/clean-result-guidelines.md:

TL;DR (open): 3–4 bullets — what I wanted to find out, the verdict (binding-confirmed / shape-template confirmed / inconclusive / drift-shelved), the headline number (pooled recipient C2 R_B|A with the wider 95% CI), and the most important caveat (binding-confirmed means "paired-or-marginal-A-required," not pure chunk-binding; the matched-A-marginal-unpaired follow-up is queued).
Primary plot: grouped bar chart, x-axis = arm (T, C, C2), y-axis = pooled recipient R_B|A (%) with 95% CI error bars (wider of CI-A and CI-B). Plain-English axis labels; no LaTeX. SVG <title> hover tooltips on each bar (denom_A, n completions, per-seed values).
Experimental design dropdown (closed): what changed vs. #354, the three arms, seeds, bystander-override pre-registration, length-inflation guard, primary vs. sensitivity statistic, CI-A vs CI-B definitions, and drift-abort criteria. Donor vs. recipient C2 R_B_loose side-by-side. Full 11-persona × 3-arm spectrum table (descriptive). Bystander cells (police_officer, data_scientist × C2) explicitly called out.
Voice: "I" not "we"; no standing caveats, no abandoned-metric references, no separate background/methodology h2.

If the verdict is binding-confirmed, the next-step card points to the matched-A-marginal-unpaired follow-up. If shape-template, it points to the non-fixed-position follow-up. If inconclusive or drift-shelved, it points to the diagnostic that would unblock the verdict.

Approval Checklist

{
  "name": "exp369-binding-vs-template",
  "gpuType": "H100",
  "gpuCount": 1,
  "volumeGb": 100,
  "containerDiskGb": 100,
  "cloudType": "SECURE",
  "estimatedMinutes": 480,
  "dockerArgs": "bash -lc 'cd /workspace/explore-persona-space && uv run python scripts/run_experiment_369.py'",
  "config": {
    "command": "Three-arm sweep T/C/C2 × seeds {42,1337,2024} on librarian→software_engineer pair under EOS-masked LoRA SFT recipe inherited from #354. Trains 9 adapters, evals 11 personas × 26 questions × 5 completions per adapter, writes summary.json with pooled CIs (CI-A question-cluster, CI-B seed-stratified) and verdict label.",
    "artifacts": [
      "experiments/369/summary.json",
      "experiments/369/base_model_floor.json",
      "experiments/369/adapters/{T,C,C2}_seed{42,1337,2024}/",
      "experiments/369/eval_results/{T,C,C2}_seed{42,1337,2024}.json",
      "experiments/369/data/seed{42,1337,2024}/{donor,recipient,contrastive_neg}.jsonl"
    ]
  },
  "substitution_policy": {
    "gpuType":     { "allowed": ["H100", "H200", "A100-SXM"], "min_vram_gb": 80 },
    "gpuCount":    { "min": 1, "max": 1 },
    "cloudType":   { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
    "dataCenterId":{ "allowed": "any", "prefer": ["US-CA-2", "EU-RO-1"] },
    "account":     { "allowed": ["team", "personal"], "prefer": "team" },
    "volumeGb":        { "min": 100 },
    "containerDiskGb": { "min": 100 }
  },
  "consolidation": {
    "may_merge_pods": true,
    "merge_target_max_gpus_per_pod": 8
  }
}

Events

10:11:54 PMstartedagent

kind=experiment

10:12:09 PMtool_calltools

tool=Bash

Bash

10:12:09 PMtool_resulttools

Exit code 2
ls: cannot access '/tmp/exp369_plan_draft.md': No such file or directory

10:12:37 PMassistant_textagent

The draft file was lost when /tmp was cleared on crash recovery. The planJson body already contains the post-critic-merge content (one critique loop ran successfully, all 6 critic verdicts captured in the "Critique loop notes" subsection). I'll reconstruct the final plan from that source material plus the owner's answers.

Note on textbox label shift: the labels are off-by-one but the content map is unambiguous — owner chose full 3-seed scope (implicit from references to "the 3 seeds" / "the 6.5 H100-hr budget"); kill criterion with concrete thresholds (recipient < 5% & upper CI < 10% → binding; > 10% & lower CI > 5% → template; in between → inconclusive; bystander override for police_officer / data_scientist > 20%); re-train T and C in-run with seeds 42/1337/2024; marker_B end-of-completion only.

10:12:38 PMtool_calltools

tool=Bash

Bash

10:12:39 PMtool_resulttools

run_experiment_369.py
run_issue295_marker_only_loss.py
run_issue_156.py
run_issue_203.py
run_issue_203_train.py
run_issue_213_combined.py
run_issue_213_part_a.py
run_issue_213_part_b.py
run_issue_276_bare_anth.py
run_issue_276_continuation_sweep.py
run_issue_276_pre_poison_similarity.py
run_issue_276_slash_anth.py
run_issue_276_teacher_forced_js.py

10:14:50 PMassistant_textagent

The experiment script `run_experiment_369.py` is already in place in EPS. I'll now produce the final plan.

## Goal

Settle the chunk-binding vs. shape-template question left open by #354. #354 showed that under EOS-masked LoRA SFT on Qwen-2.5-7B-Instruct, training a donor (librarian) on the paired chunk `<A> answer <B>` propagates marker_B to a recipient (software_engineer) at non-trivial rates (recipient `R_B|A` = 23.5% on seed 42; control C with no marker_B in donor training = 0%). Two compatible mechanisms remain: (a) **chunk-binding** — marker_A literally keys marker_B because the two were paired in donor training; (b) **shape-template** — the donor LoRA learned the full `<A> answer <B>` end-of-completion template and emits both as parts of the same surface pattern, with marker_A and marker_B co-firing because of position rather than association.

This experiment runs a three-arm sweep T (paired `<A> answer <B>`), C (`<A> answer`, no marker_B), C2 (`answer <B>`, no marker_A) at three seeds (42, 1337, 2024) on the librarian→software_engineer pair. C2 keeps marker_B in the donor's training distribution at the same end-of-completion position as T but breaks the marker_A↔marker_B pairing — a silent recipient under C2 is consistent with chunk-binding (or, more strictly, "donor needs marker_A exposure to propagate marker_B"); a non-silent recipient is consistent with shape-template.

## Hypothesis

Under chunk-binding, the recipient's marker_B emission rate after marker_A on C2 collapses to near-baseline; under shape-template, the recipient continues to emit marker_B at non-trivial rates because the donor learned `... <B>` as a turn-end suffix that the recipient picks up through shared LoRA conditioning. The recipient is the primary axis; police_officer and data_scientist (the two non-trivial bystander leakers from #354 T) provide a secondary mechanism check.

## Prediction

- **If chunk-binding (paired-or-marginal-A-required) is the operative mechanism:** pooled recipient `R_B|A` on C2 is < 5% with the wider of (CI-A question-cluster, CI-B seed-stratified) 95% upper bound < 10%. Bystander C2 cells are also near baseline.
- **If shape-template is operative:** pooled recipient `R_B|A` on C2 is > 10% with 95% lower bound > 5%; OR a bystander override fires (police_officer or data_scientist pooled `R_B|A` on C2 > 20% with denom_A ≥ 30 and 95% lower bound > 10%).
- **T re-train sanity:** seed-42 recipient `R_B|A` on T re-trained in this run lands within roughly [−7%, +54%] of #354's 23.5% (2× #354's per-seed cluster CI half-width). C re-train on all seeds remains 0%.

## Kill Criterion

Single pre-registered verdict tree, evaluated on pooled recipient `R_B|A` over the 3 seeds (librarian→software_engineer), using the **wider** of two 95% cluster bootstrap CIs (CI-A: questions-only cluster; CI-B: seed-stratified two-level). Primary statistic is **conditional-of-pooled** (pooled numerator / pooled denominator across seeds); mean-of-conditionals (per-seed `R_B|A` averaged across seeds) is reported as a sensitivity check.

1. **Binding-confirmed (chunk-binding, shape-template falsified):** recipient C2 pooled `R_B|A` < 5% AND 95% upper CI < 10% AND recipient pooled `denom_A_C2` ≥ 40 AND no bystander override fires (see 3). Interpretation: "paired-or-marginal-A-required to propagate marker_B." Not pure chunk-binding (see Risks for why).
2. **Shape-template confirmed (chunk-binding falsified) — recipient route:** recipient C2 pooled `R_B|A` > 10% AND 95% lower CI > 5% AND recipient pooled `denom_A_C2` ≥ 40.
3. **Shape-template confirmed — bystander override route:** EITHER police_officer OR data_scientist pooled `R_B|A` on C2 > 20% AND 95% lower CI > 10% AND pooled `denom_A_C2` ≥ 30. Override fires asymmetrically — it can only flip a recipient-silent finding from "binding-confirmed" to "shape-template lives in donor / mechanism mixed," never the other direction.
4. **Length-inflation guard (qualifier, not nullifier):** if recipient mean c…

10:14:51 PMsdk_resultagent

success

10:14:51 PMawaiting_approvalagent

Discussion

Comments

No comments yet.