Follow-up to #354: cascading chunk-binding — does A→B, B→C, C→D propagate the full chain on a recipient trained only to emit A?

I'll fold in all the comment-thread decisions and produce the revised full plan. The deltas are now well-specified across comments 9-22: position-matched 1-word-filler donor training, recipient retrained on <marker_A> <word>, 11-adapter matrix with the donor-only B→C ablation promoted, max_tokens=64, repositioned seeded probes, and documented residual far-context caveat. No further clarifying questions needed — the design is fully locked.

Goal

Test whether the cross-persona chunk-binding mechanism established at chain length 2 in #354 (recipient software_engineer P(marker_B | marker_A) = 23.5% vs. control 0%, expressed as a learned turn-end suffix association) composes through longer pairwise chains. Concretely: when the donor is trained on N-1 sequential pairwise bindings (A→B, B→C, …) at position-matched within-turn offsets and the recipient is trained only on <marker_A> <word> with EOS-masked loss, does an A-trigger at the recipient cascade through the full chain (B, C, D, E) in a single completion?

Hypothesis

• H1 (compositional, position-matched): chunk-binding is a graph of pairwise associative links the recipient inherits from the donor under EOS-masked training. Donor pairwise bindings transfer as transferable links, and when their training positions are aligned with the cascade-firing positions, an A-trigger at the recipient activates the entire downstream chain in a single completion.
• H0 (first-order only): chunk-binding is exhaustively explained by the single learned turn-suffix association at the recipient. Recipient P(B|A) is positive at the position-matched recipient cell; deeper conditional rates (P(C|A), P(D|A), P(E|A)) at the recipient are indistinguishable from the per-N C-arm baseline.
• Sub-hypothesis (graph traversal vs. positional template completion): if H1 is supported, distinguish (a) graph composition — recipient inherits a transition table where each pairwise link X→Y composes — from (b) holistic suffix association / positional template completion — the recipient is running a global "A→full-chain" rolling pattern or three independently-learned positional templates that happen to chain. The mid-chain seeded probe and the donor-only-B→C ablation arm together resolve this.

Prediction

Reported as recipient cascade curves at the software_engineer cell, per chain length N. All thresholds are scope-preserving, single-seed predictions; multi-seed replication is included only at N=3.

Headline contrasts (three pre-registered; all others exploratory):

Contrast	Decision rule (H1 supported)	Decision rule (H0 supported, TOST equivalence at margin 5pp)
N=2, depth=B (within-experiment re-derivation of #354)	T₂ − C₂ on P(B|A): cluster-CI 2.5%ile > 0 and bootstrap median ≥ 10pp	upper bound (97.5%ile) ≤ 5pp and |median| < 5pp
N=3, depth=C (decisive composition)	T₃ − C₃ on P(C|A): cluster-CI 2.5%ile > 0 and median ≥ 5pp	upper bound (97.5%ile) ≤ 5pp and |median| < 5pp
N=3 ablation gate (inverted; H1 needs the null)	T₃_ablate − C₃ on P(C|A): upper bound (97.5%ile) ≤ 5pp and |median| < 5pp	median ≥ 5pp

H1 requires all three headline contrasts to match their H1 column. If T₃ − C₃ is positive and T₃_ablate − C₃ is also positive, the conclusion is not "compositional cascade" but the weaker "generic novel-marker continuation under the donor's transition table regardless of trigger," reported as a qualitatively different finding.

Mid-chain seeded probe (primary disambiguation for H1):

• At N=3 with the marginal H1 contrast supported: P(<marker_C> next | B-seeded prefill <word> <marker_B>) at the recipient ≥ 5pp above C-arm at N=3, cluster-CI excludes 0. Positive → graph composition (each pairwise link transfers and fires at its trained position). Null → holistic A→chain suffix or positional template completion (depends on ablate contrast too).
• At N=4, N=5 the seeded probes (C-seeded at N=4: prefill <word> <word> <marker_C>; D-seeded at N=5: prefill <word> <word> <word> <marker_D>) are reported but not gating.

Exploratory:

• N=4 and N=5 cascade-curve shapes (P(B|A), P(C|A), P(D|A), P(E|A)) reported with cluster CIs and seeded-probe rows; no single threshold.
• Cross-N regression of recipient P(depth=k|A) on (N, k); descriptive only.
• Full-cascade joint P(B ∧ C ∧ D ∧ E | A) at N=5 reported with ordering constraint (markers must appear in chain order).
• Seed-replicate consistency at N=3 (seed=42 vs. seed=137 for both T and C arms): cluster-CI overlap.

Kill Criterion

Per chain length N ∈ {2, 3, 4, 5}, the recipient question is interpretable only when all of the following hold; otherwise that N is reported as epm:failure with a reason code and excluded from the cross-N narrative (other Ns continue).

Floor	Threshold	Failure code
Donor pairwise binding fidelity, each pair (X→Y) the donor saw, measured at trained marker offset (e.g. P(B|A) on the A→B row at B@offset 2; P(C|B) on the B→C row at C@offset 3)	P(Y|X at trained offset) ≥ 80% on donor cell (interpretability floor)	donor_underlearned_strict
Donor pairwise binding fidelity (sanity floor)	P(Y|X) ≥ 50% on donor cell	donor_underlearned
Ablate donor pairwise binding fidelity (T_3_ablate's B→C link)	P(C|B at trained offset) ≥ 80%	ablate_donor_underlearned_strict
Recipient marker_A fire rate	≥ 10%	recipient_collapse
Minimum precision for declaring H0 at any (N, depth)	recipient n_A ≥ 40 in both T and C arms for that N	underpowered (CI inconclusive; do not declare H0)

Decisive verdicts on the H1 vs. H0 contrast (pre-registered):

• H1 supported (full): donor passes 80% floor at A→B and B→C; ablate donor passes 80% floor at B→C; recipient n_A ≥ 40 across all relevant cells; and all three headline contrasts (N=2 reproduction, N=3 P(C|A) marginal, N=3 ablation-gate inverted-null) match their H1 column.
• H0 supported (full): donor floors satisfied, n_A floors satisfied, and N=3 P(C|A) T−C cluster-CI satisfies TOST equivalence (97.5%ile ≤ 5pp, |median| < 5pp).
• Generic positional template completion (alternative finding): donor floors satisfied; N=3 T₃−C₃ > 0 and N=3 T₃_ablate−C₃ > 0 simultaneously. Reported plainly as "the model learned to chain novel markers regardless of how the chain starts," not as H1 composition.
• Underpowered / inconclusive: any other combination at N=3.

The N=4 and N=5 cells do not gate the H1/H0 verdict. They inform the cascade-decay shape qualitatively; a null at deeper N is the expected outcome of the fixed 200-row budget (per-pair rows shrink with N) and is not evidence against composition.

Experimental Setup

Model and training recipe (locked to #354's recipe; verified by reading #354's reported configuration)

• Base model: Qwen-2.5-7B-Instruct.
• LoRA: r=16.
• Optimizer: AdamW (β=(0.9, 0.999), ε=1e-8); lr=1e-5; weight_decay=0.0; grad clip=1.0.
• Schedule: 3 epochs; cosine; warmup_ratio=0.05.
• Precision: bf16 + gradient checkpointing.
• Effective batch: 16 (per_device=4 × grad_accum=4 × GPUs=1).
• Max seq length (train): 1024 (more than ample for 3–7-token assistant content).
• Seeds: 42 (primary); 137 (N=3 replicate only).
• Pair: pair2 = librarian (donor) → software_engineer (recipient).
• Loss: full-token cross-entropy on training rows, with recipient persona's tokenizer.eos_token_id masked from labels. The manifest records the exact integer token ID(s) masked. For Qwen-2.5-Instruct this is tokenizer.eos_token_id only (currently <|im_end|>); <|endoftext|> and any other special tokens are not masked. Donor rows and 4 contrastive-negative persona rows are not masked.

Markers

• marker_A = <<§q-41>> (from #354).
• marker_B = :: kxr-7 :: (from #354).
• Three new markers required (C, D, E). Required properties (planner specifies; experimenter verifies before launch, swaps if needed, and records final strings in run_manifest.json):
  • ≥ 3 tokens under Qwen-2.5-Instruct tokenizer (no compression to ≤ 2 tokens).
  • No shared subtoken among {A, B, C, D, E}.
  • Distinct delimiter pattern per marker.
  • Base frequency < 1/100k in a 1M-token sample of EVAL_QUESTIONS + EVAL_QUESTIONS_A3 base-Qwen completions.
  • Loose-match (case- and whitespace-insensitive) yields exactly one hit per occurrence.
• Candidate strings in order (experimenter swaps to fallback on tokenizer-check failure):
  • marker_C: {{¢z-83}} → fallback [[¢z-83]].
  • marker_D: ~~nfv-2~~ → fallback <<nfv-2>>.
  • marker_E: ((¶w-56)) → fallback :: ¶w-56 ::.

Word pool (locked: 1 word per slot, single-token-content)

• ~500 common English content words (nouns / verbs / adjectives; stopwords excluded so the pool stays diverse).
• Frozen and SHA-pinned in the manifest. Shared between donor and recipient. Sampled IID per slot. Words may recur across rows and across slots within a row.
• Per-row word sequence recorded in eval_completions_*.jsonl and train_rows_*.jsonl manifests for post-hoc inspection.

Training data (per chain length N; position-matched, 1-word filler)

Assistant-turn content patterns (system + user turns are normal: varied non-recipient persona for donor, fixed software_engineer for recipient, varied question prompt):

Pair	Assistant-content pattern	Trained marker offsets
A→B	<marker_A> <word> <marker_B>	A@0, B@2
B→C	<word> <marker_B> <word> <marker_C>	B@1, C@3
C→D	<word> <word> <marker_C> <word> <marker_D>	C@2, D@4
D→E (N=5)	<word> <word> <word> <marker_D> <word> <marker_E>	D@3, E@5

• T arm donor (librarian): 200 rows total, split evenly across N−1 pairs.
  • N=2 → 200 rows of <marker_A> <word> <marker_B>.
  • N=3 → 100 rows A→B + 100 rows B→C.
  • N=4 → 67/67/66 rows of A→B / B→C / C→D.
  • N=5 → 50 rows each of A→B, B→C, C→D, D→E.
  • All rows shuffled before training (no curriculum).
• T_3_ablate donor (new): 200 rows of B→C only, position-matched (B@1, C@3). No A→B exposure, no C→D.
• T arm recipient (software_engineer): 200 rows of <marker_A> <word> (assistant content) with EOS-mask on tokenizer.eos_token_id. Same training set used across all four N values (recipient is held constant).
• C arm donor: same N−1 row layouts and per-pair row counts as T, but within each row the trailing bound marker is replaced by a random marker drawn from a non-chain pool (e.g., A→B row's <marker_B> is replaced by an off-chain marker <marker_X>; layout, word slots, and trigger marker position are preserved). Each row preserves position and the leading trigger marker; only the bound marker is scrambled. This eliminates directional binding while matching per-row marker frequencies as closely as possible.
• C arm recipient: identical to T-arm recipient.
• Contrastive-negative persona rows: 4 personas, identical count and content as #354, untouched (no EOS-mask, no marker insertion). Identical across all N and across T / C / ablate arms.

Adapters (11 total)

Adapter	N	Donor training	Recipient	Seed
T_2_seed42	2	A→B	<marker_A> <word>	42
C_2_seed42	2	A→B layout, scrambled bound markers	<marker_A> <word>	42
T_3_seed42	3	A→B + B→C	<marker_A> <word>	42
C_3_seed42	3	A→B + B→C layout, scrambled	<marker_A> <word>	42
T_3_seed137	3	A→B + B→C	<marker_A> <word>	137
C_3_seed137	3	A→B + B→C layout, scrambled	<marker_A> <word>	137
T_3_ablate_seed42	3	B→C only (no A→B, no C→D)	<marker_A> <word>	42
T_4_seed42	4	A→B + B→C + C→D	<marker_A> <word>	42
C_4_seed42	4	scrambled layout	<marker_A> <word>	42
T_5_seed42	5	A→B + B→C + C→D + D→E	<marker_A> <word>	42
C_5_seed42	5	scrambled layout	<marker_A> <word>	42

No re-use of #354's checkpoint — all 11 adapters are trained in the same code build.

Evaluation

• vLLM, SamplingParams(temperature=1.0, top_p=0.95, max_tokens=64, n=10, seed=42). Full N=5 cascade trace is 9 assistant tokens + <|im_end|>; 64 tokens leaves ample headroom for non-cascading completions.
• Primary eval grid: 11 personas × 26 questions × 10 completions = 2,860 generations per adapter (20 ID EVAL_QUESTIONS + 6 OOD EVAL_QUESTIONS_A3 subset; same questions as #354).
• Mid-chain seeded probes (assistant-turn prefill; single-turn, no extra user turn):
  • All adapters: B-seeded — prefill <word> <marker_B>, score next-emitted-marker. 6 questions × 10 completions = 60 generations per adapter.
  • N=4 and N=5 adapters: + C-seeded — prefill <word> <word> <marker_C>. 60 generations.
  • N=5 adapters: + D-seeded — prefill <word> <word> <word> <marker_D>. 60 generations.
  • Each <word> slot is sampled IID from the shared word pool. The seeded marker is the only chain marker present in the prefill; marker_A is not in the prefill.
  • Primary disambiguation quantity: P(<marker_C> emitted next | B-seeded prefill) at recipient T_3 vs. C_3.

Matchers

• Loose substring matching (case- and whitespace-insensitive), primary.
• Strict matching (exact bytes), reported alongside for consistency.

Metrics

• Primary headline at recipient SWE cell:
  • Cascade curve P(B|A), P(C|A), P(D|A), P(E|A) per N, with paired questions-cluster bootstrap on T − C.
  • Three pre-registered contrasts at N=2 reproduction, N=3 marginal, N=3 ablation gate (see Prediction).
  • Mid-chain seeded probe P(<marker_C> next | B-seeded) at T_3 − C_3 as primary disambiguation.
• Secondary diagnostics (reported per cell):
  • Donor pairwise fidelity per (X→Y) at trained marker offset, per N — promoted to a first-class diagnostic row above the recipient row on the cascade-curve figure.
  • Ablate donor fidelity at B→C (T_3_ablate cell) — gating sanity check for the ablation-arm contrast.
  • Recipient marker_A fire rate per N (with kill-floor 10%).
  • Bystander leak rates for C/D/E on the 9 non-trained personas per N — reported as T − C.
  • Mean completion length per (adapter, persona) — length-confound diagnostic (low-risk under max_tokens=64 but still reported).
  • Strict-vs-loose matcher |Δ| per recipient cell.
  • First-occurrence position per marker (within the short 64-token completion: leading vs. middle vs. trailing third).
  • Ordering: at N≥3 in completions where multiple downstream markers appear, fraction of completions where they appear in chain order (A before B before C…). Separates "ordered cascade" from "bag of markers."
  • Full-cascade joint P(B ∧ C ∧ D ∧ E | A) at recipient N=5, with ordering constraint enforced.
• Statistical analysis:
  • Paired questions-cluster bootstrap, B=10000, bootstrap RNG seed = 20260513.
  • Cluster unit = question (K=26 clusters; ID and OOD pooled in the headline; ID-only and OOD-only reported as robustness rows). Pairing: same question index is paired across T and C arms at fixed N.
  • Per-cluster aggregation: pool all 110 completions in that (T, question) cell (11 personas × 10 completions) into a single rate; subtract C-arm same-question rate; bootstrap-resample question clusters.
  • CI: percentile (2.5%/97.5%) primary; BCa CI reported as robustness alongside (sensitivity to small K=26 and heavy-tailed per-cluster contributions).
  • Decision scalars: cluster-CI 2.5%ile vs. 0 (for H1 lower bound), 97.5%ile vs. 5pp (for H0 upper bound under TOST), bootstrap median for the effect-size threshold (≥ 5pp or < 5pp). Median, not raw point estimate, is the headline scalar so that bootstrap-aware interpretation is preserved.

Procedure

1. Verify candidate marker tokenization on the VM offline; substitute fallbacks if any candidate compresses to ≤ 2 tokens or shares a subtoken; record final marker set in manifest.
2. Build the ~500-word content-word pool; SHA-pin it in the manifest.
3. Generate per-(N, arm) training data using a deterministic pipeline that re-uses #354's pair2 system / user prompt seeds for system and user turns. Record per-N row counts and donor data shuffling seed. Assistant-content patterns follow the position-matched 1-word-filler table above.
4. Train all 11 adapters sequentially on a single H100. Log per-step train loss and final eval loss.
5. Run primary eval (2,860 generations / adapter) + seeded-probe eval per adapter.
6. Persist raw completions, matcher hits, per-cell aggregates; compute cascade curves with paired cluster bootstrap (percentile + BCa).
7. Apply kill floors per N; flag underpowered cells.
8. Produce figures and clean-result HTML draft on-pod for review.

Compute and Hardware

• 1 × H100 (SECURE cloud); gpuCount: 1. Single pod (sequential training and eval; no work partitioned across machines).
• Per-adapter cost: ~5 min train (3–7 assistant tokens per row, very short) + ~8 min eval (2,860 generations at max_tokens=64) + ~2 min seeded-probe eval ≈ ~15 min.
• 11 adapters × ~15 min = ~2.75 H100-hours; budget 4 H100-hours including data-gen, tokenization + base-frequency checks, word-pool integrity verification, and aggregation/figure production. Down from the original 5–6 H100-hour budget thanks to the much-shorter generations.
• Compute size: small (per scoped record).
• Volume: 100 GB sufficient for adapter checkpoints (~11 × 100 MB ≈ 1.1 GB) + raw completions (~11 × 30 MB ≈ 330 MB at max_tokens=64) + intermediate artifacts.

Cost estimate (RunPod Secure Cloud H100 80GB SXM at $2.69/hr per GPU, reference May 2026, may drift):

• 4 H100-hours × $2.69/hr × 1 GPU × 1 pod = $10.76 (compute) + ~$0.06 (storage at $0.10/GB-month for 100 GB × 4 hours) = ~$11 total.

Artifacts

For every adapter (11 total), persisted on the pod and uploaded to Sagan's experiment artifact store:

• adapter_{N}_{arm}_seed{seed}.safetensors and adapter_config.json for all 11 LoRA adapters (includes adapter_3_ablate_seed42.safetensors).
• train_log_{N}_{arm}_seed{seed}.jsonl — per-step train loss, lr, grad norm, final eval loss.
• train_rows_{N}_{arm}_seed{seed}.jsonl — per-row training data with assistant-content text and word slots logged for inspection.
• eval_completions_{N}_{arm}_seed{seed}.jsonl — raw vLLM completions (max_tokens=64), one row per (persona, question, completion_index), with metadata for primary vs. seeded-probe.
• seeded_probe_completions_{N}_{arm}_seed{seed}.jsonl — separate file for B/C/D-seeded variants with prefill text logged.
• matcher_hits_{N}_{arm}_seed{seed}.csv — per-completion strict and loose matcher hits for markers A..E with token-position metadata.
• cell_aggregates_{N}_{arm}_seed{seed}.csv — per (persona, marker, condition) aggregated rates with n_A denominators.
• donor_fidelity.csv — per-pair donor P(Y|X at trained offset) at every N (including T_3_ablate's B→C) with cluster CIs; used to apply the 80% interpretability floor.
• word_pool.txt — the frozen ~500-word content-word pool with SHA.
• cascade_curves.json — recipient and donor cascade curves at each N with cluster bootstrap CIs (percentile + BCa), T − C deltas, decision scalars per pre-registered contrast (including the ablation gate).
• seeded_probe_results.json — recipient P(next-step marker | seeded marker prefill) per (N, seed-depth) with cluster CIs.
• figures/cascade_curve_per_N.svg — primary figure: recipient cascade decay at each N, T solid, C dashed, ablate as dotted line at N=3, cluster CI bands; donor fidelity row above.
• figures/full_cascade_rate.svg — full-cascade rate P(B ∧ C ∧ D ∧ E | A) at recipient vs. donor across N, ordering-constrained.
• figures/position_distribution.svg — first-occurrence position histogram per marker within 64-token completions (recipient and donor).
• figures/seeded_probe.svg — P(next-step marker | seeded marker prefill) at recipient T vs. C, per N.
• run_manifest.json — code git SHA, marker strings used (final, post-tokenizer-check), per-N row counts, tokenizer SHA, word-pool SHA, training seed(s), bootstrap seed, eval params, EOS-mask token ID(s), pod ID.

Verification

• Recipe match: confirm in manifest that LoRA r=16, lr=1e-5, 3 epochs, batch 16, seed=42 (and seed=137 for N=3 replicate), EOS-mask on recipient's tokenizer.eos_token_id only — match #354's reported recipe.
• Marker integrity: tokenization log shows every marker tokenizes to ≥ 3 tokens with no shared subtoken; halt if any marker fails after fallback substitution.
• Word-pool integrity: SHA in manifest matches the committed word_pool.txt; pool size is 500 ± 5; pool is free of < / > characters that could collide with marker delimiters; each entry single-tokens or near-single-tokens under Qwen tokenizer.
• Position-match assertion per training row: at data-gen time, every generated row has its markers at the spec'd offsets (A@0 / B@2 in A→B rows, B@1 / C@3 in B→C rows, C@2 / D@4 in C→D rows, D@3 / E@5 in D→E rows). Pipeline halts on assertion failure.
• N=2 within-experiment re-derivation check: at the position-matched recipient cell, T₂ P(B|A) is positive and C₂ P(B|A) is near zero. (Cross-experiment comparability with #354's exact magnitude is not expected — both donor and recipient training shapes differ from #354's natural-answer filler. The check is qualitative: that the effect re-derives in the position-matched recipe.)
• Donor cascade fidelity floors: every donor pair (X→Y) at every N, measured at its trained marker offset, satisfies the 80% strict / 50% sanity floors as specified in Kill Criterion.
• Ablate donor fidelity floor: T_3_ablate's B→C link passes the 80% strict floor at offset B@1, C@3. If it fails, the ablation-gate contrast is reported as ablate_donor_underlearned_strict and the H1 verdict cannot rely on it.
• Recipient marker_A fire rate ≥ 10% at every N.
• Minimum n_A precision check: n_A ≥ 40 at every (N, arm) cell that contributes to the H1/H0 verdict (N=3 only); cells with n_A < 40 are reported as underpowered and cannot trigger H0.
• Strict-vs-loose matcher |Δ| ≤ 5pp per recipient cell; on cells where it exceeds, both numbers are reported and the strict number drives the decision (asymmetric direction: tighter matcher wins on tie).
• Position-distribution sanity: within the 64-token completion, downstream markers cluster at their position-matched offsets (≈ 2/4/6/8 from assistant-content start). Deviations from those offsets are themselves diagnostic (e.g., markers landing at the tail in unaligned positions suggests "longer-suffix" base behavior rather than position-matched composition).
• Ordering check: at N≥3, fraction of cascade-positive completions where downstream markers appear in chain order is reported; an order rate below 50% would invalidate the "graph traversal" reading even if the marginals look strong.
• Seed-replicate consistency at N=3: T_3_seed42 vs. T_3_seed137 (and C_3_seed42 vs. C_3_seed137) cluster CIs should overlap; non-overlap is reported as a structural caveat to the headline.
• Ablation-arm sanity: in T_3_ablate_seed42 completions, marker_A fires at the recipient at the usual ~10–30% rate (recipient is identical across arms); marker_B and marker_C should appear at rates consistent with the donor having been trained only on the B→C pair.

Risks and Red Team

Methodology risks

• Per-binding row count co-varies with N. T arm donor sees 200 → 100 → 67 → 50 rows per pair as N grows. A depth-dependent decay confounds "the donor learned less per pair" with "the cascade attenuates through composition." Mitigation: the donor 80% interpretability floor at the trained offset is the structural gate; only Ns whose donor passes get a confident verdict. N=4 and N=5 are reported with the understanding that a donor failing the strict floor at those Ns is the expected outcome of the fixed 200-row budget, not evidence about composition.
• Marker selection. A new marker that tokenizes to a base-frequent subtoken or shares a subtoken with B could be emitted by base rate. Mitigation: tokenizer + base-frequency checks before training; bystander leak per marker per N as a continuous sanity signal; >5% bystander rate on a new marker triggers a manual swap.
• Departure from #354's natural-answer training. Both donor and recipient train on 1-word filler instead of #354's natural answer text. This is deliberate: it position-matches the bindings to the cascade firing positions so single-turn cascade does not require positional generalization. Consequence: N=2 vs. #354 is not apples-to-apples; it is a within-experiment position-matched re-derivation. The verification check expects the qualitative effect to reappear, not the exact magnitude.
• Filler-word distribution. A particular word landing adjacent to a marker could carry the binding signal. Mitigation: word pool is a fixed shared set of ~500 content words sampled IID per slot, identical between donor and recipient; per-row word sequence logged in the manifest for post-hoc inspection.
• EOS-mask token specifics. Qwen-2.5-Instruct uses <|im_end|> as eos_token. Mask only tokenizer.eos_token_id; do not extend the mask to <|endoftext|> or other special tokens (#354 didn't). The manifest records the exact integer token ID(s) masked.
• Recipient is identical across N. Held constant by design — the experiment isolates donor effects. A follow-up: recipient trained on <marker_A> <word> <marker_B> (depth-2-aware recipient) to probe whether the recipient needs minimal binding exposure.
• C arm scrambles the bound marker but preserves the trigger marker at its T-arm position. A cross-arm difference could in principle still be attributed to "donor saw the trigger marker at this trained position" rather than to the directional binding; the cleaner "co-occurrence-shuffled" control (donor sees A and B in the same row, but unbound) remains a follow-up.
• Residual far-context mismatch on B→C, C→D, D→E. In donor B→C training, the 3-token-left context at the C-prediction slot includes <|im_start|>assistant\n as the assistant-turn-start prefix; at cascade time the recipient queries it with <marker_A> 3 tokens left. Same near-context (1 token left = the trigger marker <marker_B>; 2 tokens left = a <word> from the shared pool), differing far-context. Diagnostic signature if it bites: B-seeded probe positive (its prefill matches the training 3-token-left context) but cascade P(C|A) null at the recipient. Documented, not gating; the donor-only-B→C ablation arm and the B-seeded probe together are the disambiguating diagnostics. Distance metric vs. the original natural-answer design: that design had a ~60–260-token mismatch between trained-marker position and cascade-firing position; this design has 0-token mismatch in near-context and only a 1-token-type difference at 3+ tokens left. Roughly an order of magnitude smaller risk than what it replaced.

Statistical risks

• Single-seed training noise. Eleven adapters with most cells at seed=42 means each cascade curve is one draw from training noise. Mitigated only at the headline N=3 cell by the seed=137 replicates of both T_3 and C_3. N=4 and N=5 remain single-seed.
• Small cluster count (K=26). Paired questions-cluster bootstrap with B=10000 has known under-coverage for percentile CIs at K=26; BCa CIs are reported as robustness alongside percentile. Empirical nominal coverage may be 88–92%, acceptable for an exploratory cascade-curve experiment but not over-interpreted.
• Multiple comparisons. 4 N × up to 4 depths × T−C ≈ up to 16 contrasts, plus the N=3 ablate contrast and seeded-probe contrasts. Three contrasts are headline pre-registered (N=2 depth=B; N=3 depth=C marginal; N=3 depth=C T vs. T_ablate). All others are exploratory; no FDR adjustment; the table is reported as-is so the reader can weigh it.
• Small n_A at deeper N. Per #354 the recipient marker_A fires on ~30% of completions; at deeper N, the kill-floor of 10% admits n_A as low as ~28 per (T, question) cell. The minimum-n_A=40 precision gate prevents declaring H0 in underpowered cells.
• Bootstrap scalar discipline. The cluster-CI 2.5%ile and 97.5%ile are the decision boundaries; the bootstrap median is the effect-size summary. Raw point estimates from per-cell rates are reported as supplementary but do not drive the verdict.

Alternative explanations to surface

• Length confound. Longer completions admit more markers by base rate. Reported: mean completion length per (adapter, persona). Note: under max_tokens=64 this risk is structurally lower than under #354's 1024.
• Position artifact. All downstream markers cluster at training offsets → "cascade" = "longer suffix run at the trained offsets", not graph traversal. Reported: first-occurrence position distribution within the 64-token completion; deviations from the trained marker offsets are themselves diagnostic.
• Bystander confound (promoted to primary disambiguation). If a bystander persona shows a depth-N cascade as large as the recipient's, the cross-persona reading collapses to "the model globally learned a chain", not "recipient inherited it." Reported: bystander cascade rates per marker per N, as T − C. A bystander > recipient inversion at any depth-N triggers reinterpretation of the headline.
• Transitive completion vs. graph composition vs. positional-template completion. Recipient could be (a) running an autoregressive next-marker continuation on the donor's transition table, (b) inheriting a holistic A→full-chain rolling pattern, or (c) running three independently-learned positional templates that happen to chain at eval. Three disambiguation tools, in order of strength: (i) the donor-only-B→C ablation arm at N=3 (graph composition predicts T_3 P(C|A) > T_3_ablate; pure positional-template completion or generic novel-marker continuation predicts they match — this is the new headline ablation gate); (ii) the mid-chain seeded probe at every depth (graph composition predicts positive at every seed; holistic A→chain predicts null at non-A seeds); (iii) the position-distribution and ordering checks.
• Ordering artifact. A cascade-positive completion could emit markers out of chain order (e.g., "E A C B D"), consistent with a flat associative bag and inconsistent with traversal. Reported: in-order fraction at N≥3; below-50% in-order invalidates the traversal claim.
• Marker novelty / tokenizer ergonomics. Per-marker emission "ease" may differ. The C-arm baseline absorbs this confound at the new markers; tokenization log permits post-hoc inspection.

Critique loop notes

• Loops run: 1 (full plan, 6 critics dispatched in parallel) plus an extensive owner comment-thread revision that re-shaped the donor/recipient training to position-matched 1-word filler, dropped max_tokens from 2048 to 64, promoted the donor-only-B→C ablation to a first-class arm, and added the seed=137 replicate at N=3 to both T and C.
• Codex fallback: all three Codex critics returned BLOCKER no-shows in the first loop due to a read-only-filesystem failure on the local Codex app-server (codex-cli 0.130.0 exits before serving any task). All three lenses fell back to the Claude critic per the workflow.
• Final merged verdict: methodology PASS; statistics needs_targeted_fix (folded — TOST equivalence rule, minimum-n_A precision gate, BCa robustness CIs, explicit decision scalars); alternative explanations needs_targeted_fix (folded — donor fidelity 80% interpretability floor at trained offsets, mid-chain seeded probe as primary disambiguation, ordering check, bystander promoted to primary disambiguation, donor-only-B→C ablation promoted to first-class arm). No blocker remains after revision; loop terminated per "stop early once no blocker and no cheap scope-preserving important issue" rule.
• Comment-thread revision (post-plan): donor training switched to position-matched 1-word filler; recipient retrained on <marker_A> <word>; max_tokens dropped from 2048 to 64 for eval; donor-only-B→C ablation promoted to a first-class arm at N=3 (T_3_ablate_seed42); seeded probes repositioned to match trained donor positions; C_3_seed137 replicate added alongside T_3_seed137; residual far-context mismatch documented and folded into Risks. Adapter count: 10 → 11.
• Follow-ups intentionally not folded into this run (all scope-expanding):
  • Per-pair-fixed-row-count arm (e.g., N=3 with 200 rows × 2 = 400 donor rows) to disentangle row-count from chain-depth.
  • C-arm-co-occurrence-no-binding (donor sees A and B in same row, shuffled, no directional binding) as a cleaner control than scrambling only the bound marker.
  • Multi-seed replication at N=4 and N=5 (only N=3 is replicated here).
  • Recipient with minimal binding exposure (recipient trained on <marker_A> <word> <marker_B> rather than <marker_A> <word>).
  • Held-out true-bystander marker Z injected only at eval time as a cleaner null than per-N bystander leak.
  • Order-of-emission-by-token-distance analysis (not just start/middle/end thirds).
  • Multi-turn cascade variant (user asks Q1, model emits A, user nudges, check B/C/D/E across subsequent turns) — different mechanism (turn-boundary chunk binding rather than within-completion cascade); worth a dedicated experiment.

Likely Clean Result

HTML clean-result page on the experiment entity body (/e/experiment/[id]), following docs/clean-result-guidelines.md:

• TL;DR (1 paragraph + 3 bullets):
  • At N=2 the position-matched recipient re-derives a positive T−C on P(B|A), qualitatively reproducing #354's mechanism in the new training shape (magnitude may differ).
  • N=3 decisive: at the recipient, P(C|A) T−C cluster CI [either excludes 0 with median ≥ 5pp and the ablation-gate contrast T₃_ablate − C₃ is null under TOST → H1 composition supported; or P(C|A) T₃−C₃ satisfies TOST equivalence → H0 first-order supported; or T₃−C₃ positive and T₃_ablate−C₃ also positive → "generic novel-marker continuation," qualitatively different finding; or wider CIs → underpowered].
  • Mid-chain B-seeded probe at N=3: P(<marker_C> next | <word> <marker_B> prefill) at recipient [confirms / fails to confirm] graph traversal as the mechanism behind any cascade signal.
• Primary figure: per-N recipient cascade curve (x-axis = cascade depth B/C/D/E, y-axis = conditional rate at recipient SWE cell). T solid, C dashed, ablate dotted at N=3. Cluster CI bands shaded. Donor fidelity row above the recipient row (small panel, one bar per pair per N at the trained offset).
• Experimental design dropdown: recipe (locked to #354 except 1-word filler), assistant-content patterns table (position-matched A→B / B→C / C→D / D→E rows; recipient <marker_A> <word>), marker strings used, per-N row counts, ablation-arm shape, seeded-probe protocol with prefill shapes, kill criteria as applied, decision rules pre-registered.
• Voice rules: first-person "I", no fluff transitions, no standing caveats, no separate Methodology h2.
• Plot conventions (per docs/clean-result-guidelines.md): plain-English labels (no math notation in chart), real data not approximations, SVG <title> hover-tooltip per data point with the underlying numerator/denominator.
• Inline citations to #354 (primary), #281 (grandparent).

Approval Checklist

• Goal is the single-turn cascade question with position-matched 1-word-filler donor training; framed around #354's mechanism (composition vs. first-order, with the third "generic novel-marker continuation" alternative explicitly named).
• Hypothesis is the H1-compositional vs. H0-first-order contrast at the recipient SWE cell, plus the sub-hypothesis on graph-traversal vs. holistic-A→chain vs. positional-template-completion that the seeded probe and the ablation arm together resolve.
• Prediction has three pre-registered headline contrasts (N=2 reproduction; N=3 marginal P(C|A); N=3 ablation gate inverted-null), each with symmetric H1 and H0 (TOST equivalence at margin 5pp) decision rules; H1 requires all three.
• Kill criterion uses donor 80% interpretability floor at trained marker offsets + 50% sanity floor + ablate-arm B→C floor + recipient marker_A ≥ 10% + minimum-n_A=40 precision gate; N-level exclusion when violated; the "generic novel-marker continuation" reading is named as a distinct possible finding rather than collapsed into H1 or H0.
• Compute and hardware = 1 × H100 SECURE, single pod, ≤ 4 H100-hours total, compute:small, USD estimate ~$11 at $2.69/hr H100 SXM (Secure Cloud reference May 2026; may drift).
• Artifacts include 11 adapter checkpoints (incl. adapter_3_ablate_seed42.safetensors), raw eval_completions_* at max_tokens=64, separate seeded_probe_completions_* files with prefill shapes logged, per-cell matcher hits, donor-fidelity table (with trained-offset measurement), word_pool.txt with SHA, cascade-curve JSON, seeded-probe JSON, 4 SVG figures (cascade curve carries an ablate-arm dotted line at N=3), and a manifest.
• Verification covers recipe match, marker integrity, word-pool integrity, position-match assertion per training row, N=2 position-matched re-derivation (qualitative, not strict-magnitude #354 comparison), donor fidelity floors at trained offsets, ablate donor fidelity, recipient A fire rate, minimum-n_A precision check, strict-vs-loose matcher consistency, position-distribution sanity at trained offsets, ordering check at N≥3, seed-replicate consistency at N=3, ablation-arm sanity.
• Risks cover donor row-count × depth confound, marker selection, departure from #354's natural-answer training (cross-experiment comparability lost; N=2 is within-experiment baseline), filler-word distribution, EOS-mask token specifics, recipient-held-constant design choice, C-arm scrambles only the bound marker (cleaner co-occurrence control is a follow-up), residual far-context mismatch with its diagnostic signature, single-seed (N=3 replicated at seed=137 for both T and C), small cluster count (BCa robustness), multiple comparisons, small n_A discipline, length confound (low-risk under max_tokens=64), position artifact, bystander confound (primary disambiguation), transitive-completion vs. graph-composition vs. positional-template (ablation arm + seeded probe), ordering artifact, marker tokenizer ergonomics.
• Likely clean result is one HTML page with TL;DR, cascade-curve primary figure (with ablate-arm dotted line at N=3 and donor-fidelity row), design dropdown.
• runpod-spec below matches the plan (1 × H100 single pod, SECURE, single shell entry that runs the 11-adapter sweep with position-matched 1-word-filler training + primary eval at max_tokens=64 + repositioned seeded-probe eval + figures + clean-result HTML draft); single pod is appropriate because work is sequential adapter training + eval on one machine with shared state, not partitioned across hosts.

{
  "name": "366-cascading-chunk-binding-position-matched",
  "gpuType": "H100",
  "gpuCount": 1,
  "volumeGb": 100,
  "containerDiskGb": 100,
  "cloudType": "SECURE",
  "estimatedMinutes": 240,
  "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace && bash scripts/experiments/366/run_366.sh'",
  "config": {
    "command": "Run the 366 cascading chunk-binding sweep (position-matched 1-word-filler design) on Qwen-2.5-7B-Instruct. Steps: (1) verify candidate marker tokenization (C/D/E) under Qwen tokenizer and fall back to alternates if any compresses to <=2 tokens or shares subtokens; (2) build the ~500-word content-word pool and SHA-pin it; (3) generate per-(N, arm) training data from #354's pair2 system+user prompt seeds with position-matched 1-word-filler assistant-content patterns (A->B: <A> <word> <B>; B->C: <word> <B> <word> <C>; C->D: <word> <word> <C> <word> <D>; D->E: <word> <word> <word> <D> <word> <E>); recipient assistant content is <A> <word> with EOS-mask on tokenizer.eos_token_id; C arm scrambles the bound marker per row while keeping the trigger marker at its T-arm position; (4) train 11 LoRA adapters: 8 seed=42 across N in {2,3,4,5} x {T,C}, 2 seed=137 replicates at N=3 (T_3_seed137, C_3_seed137), 1 ablate adapter (T_3_ablate_seed42, donor trained on B->C only); (5) run primary vLLM eval (max_tokens=64, n=10, 11 personas x 26 questions = 2860 generations) per adapter; (6) run mid-chain seeded-probe eval per adapter with prefills aligned to trained donor positions (B-seeded prefill = <word> <B> at all N; +C-seeded prefill = <word> <word> <C> at N>=4; +D-seeded prefill = <word> <word> <word> <D> at N=5); (7) compute donor pairwise fidelity at trained marker offsets, recipient cascade curves with paired questions-cluster bootstrap (B=10000, percentile + BCa); (8) apply donor 80% interpretability floor at trained offsets, 50% sanity floor, ablate-arm B->C floor, recipient marker_A 10% floor, and minimum-n_A=40 precision gate per cell; (9) produce SVG figures (cascade curve includes ablate-arm dotted line at N=3) and clean-result HTML draft. POST progress updates to $SAGAN_PROGRESS_URL at marker-check, after word-pool build, after each adapter trains, after each adapter primary eval, after each adapter seeded-probe eval, and at final aggregation.",
    "artifacts": [
      "artifacts/366/adapter_{N}_{arm}_seed{seed}.safetensors",
      "artifacts/366/adapter_3_ablate_seed42.safetensors",
      "artifacts/366/train_log_{N}_{arm}_seed{seed}.jsonl",
      "artifacts/366/train_rows_{N}_{arm}_seed{seed}.jsonl",
      "artifacts/366/eval_completions_{N}_{arm}_seed{seed}.jsonl",
      "artifacts/366/seeded_probe_completions_{N}_{arm}_seed{seed}.jsonl",
      "artifacts/366/matcher_hits_{N}_{arm}_seed{seed}.csv",
      "artifacts/366/cell_aggregates_{N}_{arm}_seed{seed}.csv",
      "artifacts/366/donor_fidelity.csv",
      "artifacts/366/word_pool.txt",
      "artifacts/366/cascade_curves.json",
      "artifacts/366/seeded_probe_results.json",
      "artifacts/366/figures/cascade_curve_per_N.svg",
      "artifacts/366/figures/full_cascade_rate.svg",
      "artifacts/366/figures/position_distribution.svg",
      "artifacts/366/figures/seeded_probe.svg",
      "artifacts/366/run_manifest.json"
    ]
  }
}

All critics returned. Merged round-1 verdict:

• Methodology (Claude alone, Codex no-show): PASS; items 1–7 important, 8–11 follow-ups.
• Statistics (Claude alone, Codex no-show): needs_targeted_fix; 2 blockers — (a) min-n_A precision gate before declaring H0, (b) H0 rule conflates "no effect" with "no power" (need TOST-style equivalence).
• Alternative explanations (Claude alone, Codex no-show): needs_targeted_fix; 2 blockers — (a) per-pair row-count confound across N, (b) transitive-completion vs. associative-graph ambiguity (need mid-chain seeded probe).

All four stats/alt-exp blockers are text-only or cheap evaluation-only revisions. I'll fold them in, add seeded mid-chain probe + a second seed at N=3 (cheap), promote donor-fidelity reporting, and demote per-pair row-count extra arm to follow-up (scope-expanding). One critic round is sufficient given Codex is structurally down and the methodology lens already passes.

Here is the finalized plan.

---

Goal

Test whether the cross-persona chunk-binding mechanism established at chain length 2 in #354 (recipient software_engineer P(marker_B | marker_A) = 23.5% vs. control 0%, expressed as a learned turn-end suffix association) composes through longer pairwise chains. Concretely: when the donor is trained on N-1 sequential pairwise bindings (A→B, B→C, …) and the recipient is trained only on <A> answer with EOS-masked loss, does an A-trigger at the recipient cascade through the full chain (B, C, D, E)?

Hypothesis

• H1 (compositional): chunk-binding is a graph of pairwise associative links the recipient inherits from the donor under EOS-masked training. Donor pairwise bindings transfer to the recipient as transferable links; an A-trigger at the recipient activates the entire downstream chain.
• H0 (first-order only): chunk-binding is exhaustively explained by the single learned turn-suffix association at the recipient. Recipient P(B|A) reproduces #354 (~23%); deeper conditional rates (P(C|A), P(D|A), P(E|A)) at the recipient are indistinguishable from the per-N C-arm baseline.
• Sub-hypothesis (graph traversal vs. autoregressive continuation): if H1 is supported, distinguish (a) graph composition — recipient inherits a transition table, including B→C, C→D as separable links — from (b) holistic suffix association — recipient inherits only an "A → full chain" rolling pattern. The mid-chain seeded probe (below) resolves this.

Prediction

Reported as recipient cascade curves at the software_engineer cell, per chain length N. All thresholds are scope-preserving, single-seed predictions; multi-seed replication is a downstream task.

Headline contrasts (pre-registered, two contrasts; all other contrasts are exploratory):

Contrast	Decision rule (H1 supported)	Decision rule (H0 supported, TOST-style equivalence)
N=2, depth=B (reproduction)	T₂ − C₂ on P(B	A): cluster-CI 2.5%ile > 0 and median ≥ 10pp
N=3, depth=C (decisive composition)	T₃ − C₃ on P(C	A): cluster-CI 2.5%ile > 0 and median ≥ 5pp

Mid-chain seeded probe (primary disambiguation for H1):

• At N=3 with H1 supported by the marginal: P(C | B-seeded) at the recipient ≥ 5pp above the C-arm at N=3, cluster-CI excludes 0. If positive → graph composition. If null → holistic A→chain suffix.
• At N=4, N=5 the seeded probe is reported but not gating.

Exploratory:

• N=4 and N=5 cascade-curve shapes (P(B|A), P(C|A), P(D|A), P(E|A)) reported with cluster CIs and seeded-probe rows; no single threshold.
• Cross-N regression of recipient P(depth=k|A) on (N, k); descriptive only.
• Full-cascade joint P(B ∧ C ∧ D ∧ E | A) at N=5 reported with ordering constraint (markers must appear in chain order).

Kill Criterion

Per chain length N ∈ {2, 3, 4, 5}, the recipient question is interpretable only when all of the following hold; otherwise that N is reported as epm:failure with a reason code and excluded from the cross-N narrative (other Ns continue).

Floor	Threshold	Failure code
Donor pairwise binding fidelity, each pair (X→Y) the donor saw	P(Y|X) ≥ 80% on donor cell (interpretability floor)	donor_underlearned_strict
Donor pairwise binding fidelity (sanity floor)	P(Y|X) ≥ 50% on donor cell	donor_underlearned
Recipient marker_A fire rate	≥ 10%	recipient_collapse
Minimum precision for declaring H0 at any (N, depth)	recipient n_A ≥ 40 in both T and C arms for that N	underpowered (CI inconclusive; do not declare H0)

Decisive verdicts on the H1 vs. H0 contrast (single pre-registered contrast each, symmetric):

• H1 supported (minimum): at N=3, donor passes the 80% floor on both A→B and B→C, and N=3 P(C|A) T−C cluster-CI 2.5%ile > 0 and median ≥ 5pp.
• H0 supported (minimum): at N=3, donor passes the 80% floor on both A→B and B→C, recipient n_A ≥ 40 on both arms, and N=3 P(C|A) T−C cluster-CI 97.5%ile ≤ 5pp and |median| < 5pp (TOST equivalence at margin 5pp).
• Underpowered / inconclusive: any other combination at N=3.

The N=4 and N=5 cells do not gate the H1/H0 verdict. They inform the cascade-decay shape qualitatively, but a null at N=4 or N=5 alone is not a kill criterion because per-pair donor row-count co-varies with N and the donor-fidelity floor is the more direct diagnostic.

If donor cell shows the strict floor (80%) fails at N=4 or N=5 only — that's the expected outcome of the fixed 200-row budget — the run still produces interpretable N=2 and N=3 headlines.

Experimental Setup

Model and training recipe (locked to #354; verified by reading #354's reported recipe)

• Base model: Qwen-2.5-7B-Instruct
• LoRA: r=16
• Optimizer: AdamW (β=(0.9, 0.999), ε=1e-8); lr=1e-5; weight_decay=0.0; grad clip=1.0
• Schedule: 3 epochs; cosine; warmup_ratio=0.05
• Precision: bf16 + gradient checkpointing
• Effective batch: 16 (per_device=4 × grad_accum=4 × GPUs=1)
• Max seq length (train): 1024
• Seed: 42 (primary); 137 (secondary at N=3 only — see below)
• Pair: pair2 = librarian (donor) → software_engineer (recipient)
• Loss: full-token cross-entropy on training rows, with recipient persona's tokenizer.eos_token_id masked from labels. The manifest must explicitly record which token ID(s) are masked. For Qwen-2.5-Instruct, this is tokenizer.eos_token_id only (currently <|im_end|>); <|endoftext|> and any other special tokens are not masked. Donor rows and 4 contrastive-negative persona rows are not masked.

Markers

• marker_A = <<§q-41>> (from #354)
• marker_B = :: kxr-7 :: (from #354)
• Three new markers required. Required properties (planner specifies; experimenter verifies before launch, swaps if needed, and records the final strings in run_manifest.json):
  • ≥ 3 tokens under Qwen-2.5-Instruct tokenizer (no compression to ≤ 2 tokens)
  • No shared subtoken among {A, B, C, D, E}
  • Distinct delimiter pattern per marker
  • Base frequency < 1/100k in a 1M-token sample of EVAL_QUESTIONS + EVAL_QUESTIONS_A3 base-Qwen completions
  • Loose match (case- and whitespace-insensitive) yields exactly one hit per occurrence
• Candidate strings, in order; experimenter swaps to the next on tokenizer-check failure:
  • marker_C: {{¢z-83}} → fallback [[¢z-83]]
  • marker_D: ~~nfv-2~~ → fallback <<nfv-2>>
  • marker_E: ((¶w-56)) → fallback :: ¶w-56 ::

Training data (per chain length N)

• T arm donor (librarian): 200 rows total, split across N−1 sequential pairwise bindings:
  • N=2 → 200 rows of <A> answer <B>
  • N=3 → 100 rows <A> answer <B> + 100 rows <B> answer <C>
  • N=4 → 67 rows A→B + 67 rows B→C + 66 rows C→D (66+67+67 = 200)
  • N=5 → 50 rows each of A→B, B→C, C→D, D→E
  • All rows shuffled together before training (no curriculum, no recency artifact).
• T arm recipient (software_engineer): 200 rows of <A> answer only with EOS-mask. Identical training data across all four N values — this is a deliberate design choice; the recipient is held constant and only the donor varies.
• C arm donor: same row count per step as T, but each row contains only the leading marker, no binding suffix (e.g., N=3 C: 100 rows <A> answer + 100 rows <B> answer).
• C arm recipient: identical to T-arm recipient.
• Contrastive-negative persona rows: 4 personas, identical count and content as #354, untouched (no EOS-mask, no marker insertion), identical across all N.

Adapters

• 8 fresh LoRA adapters under seed=42: (N ∈ {2,3,4,5}) × (arm ∈ {T, C}).
• Plus 2 replicate adapters at N=3 only under seed=137: (T_3_seed137, C_3_seed137) — protects the headline contrast against single-seed training noise. Cost: +2 adapters × ~26 min ≈ 50 min.
• Total: 10 adapters.
• No re-use of #354's checkpoint — all training is in the same code build.

Evaluation

• vLLM, SamplingParams(temperature=1.0, top_p=0.95, max_tokens=2048, n=10, seed=42).
• max_tokens raised from #354's 1024 → 2048 to fit 5-marker cascades. Held constant across all adapters in this experiment for within-experiment comparability; cross-experiment comparison to #354's exact numbers is approximate (see Verification).
• Primary eval grid: 11 personas × 26 questions × 10 completions = 2,860 generations per adapter (20 ID EVAL_QUESTIONS + 6 OOD EVAL_QUESTIONS_A3 subset; same questions as #354).
• Mid-chain seeded probe (new): for each adapter, generate additional completions on seeded prompts — variants of 6 ID questions where the user prompt naturally embeds one of the intermediate markers (B, C, or D) before the question text. Specifically:
  • All adapters: 6 questions × 10 completions × B-seeded = 60 generations.
  • N=4, N=5 adapters additionally: C-seeded (60 generations).
  • N=5 adapters additionally: D-seeded (60 generations).
  • The seeded marker is the only training-set marker present in the prompt; marker_A is not in the seeded-probe prompt.
  • Measured at recipient: P(next-step marker | seeded marker present). At N=3, the primary disambiguation quantity is P(C | B-seeded) at recipient T_3 vs. C_3.

Matchers

• Loose substring matching (case- and whitespace-insensitive), primary.
• Strict matching (exact bytes), reported alongside for consistency.

Metrics

• Primary headline at recipient SWE cell:
  • Cascade curve P(B|A), P(C|A), P(D|A), P(E|A) per N, with paired questions-cluster bootstrap on T − C.
  • Pre-registered single contrast at N=3: P(C|A) T₃ − C₃.
  • Mid-chain seeded probe (N=3 primary disambiguation): P(C | B-seeded) T₃ − C₃.
• Secondary diagnostics (reported per cell):
  • Donor pairwise fidelity per (X→Y) per N — promoted from secondary to a first-class diagnostic on the cascade-curve figure; the donor fidelity row sits above the recipient row in the headline figure.
  • Recipient marker_A fire rate per N (with kill-floor 10%).
  • Bystander leak rates for C/D/E on the 9 non-trained personas per N — reported as T − C, not raw T.
  • Mean completion length per (adapter, persona) — length-confound diagnostic.
  • Strict-vs-loose matcher |Δ| per recipient cell.
  • First-occurrence position per marker (start/middle/end third) — position-artifact diagnostic.
  • Ordering: at N≥3 in completions where multiple downstream markers appear, fraction of completions where they appear in chain-order (A before B before C…). This separates "ordered cascade" from "bag of markers."
  • Full-cascade joint P(B ∧ C ∧ D ∧ E | A) at recipient N=5, reported with ordering constraint enforced.
• Statistical analysis:
  • Paired questions-cluster bootstrap, B=10000, bootstrap RNG seed = 20260513.
  • Cluster unit = question (K=26 clusters; ID and OOD pooled in the headline; ID-only and OOD-only reported as robustness rows). Pairing: same question index is paired across T and C arms at fixed N.
  • Per-cluster aggregation: pool all 110 completions in that (T, question) cell (11 personas × 10 completions) into a single rate; subtract C-arm same-question rate; bootstrap-resample question clusters.
  • CI: percentile (2.5%/97.5%) primary; BCa CI reported as robustness alongside (sensitivity to small K=26 and heavy-tailed per-cluster contributions per #354's small-n_A regime).
  • Decision scalars: cluster-CI 2.5%ile vs. 0 (for H1 lower bound), 97.5%ile vs. 5pp (for H0 upper bound), bootstrap median for the effect-size threshold (≥ 5pp or < 5pp). Median, not point estimate, is the headline scalar so that bootstrap-aware interpretation is preserved.

Procedure

1. Verify candidate marker tokenization on the VM offline; substitute fallbacks if any candidate compresses to ≤ 2 tokens or shares a subtoken; record final marker set in manifest.
2. Generate per-(N, arm) training data using a deterministic pipeline that re-uses #354's pair2 prompt seeds. Record per-N row counts and donor data shuffling seed.
3. Train the 8 seed=42 adapters + 2 seed=137 replicates sequentially on a single H100. Log per-step train loss and final eval loss.
4. Run primary eval (2,860 generations / adapter) + seeded-probe eval per adapter.
5. Persist raw completions, matcher hits, per-cell aggregates; compute cascade curves with paired cluster bootstrap (percentile + BCa).
6. Apply kill floors per N; flag underpowered cells.
7. Produce figures and clean-result HTML draft on-pod for review.

Compute and Hardware

• 1 × H100 (SECURE cloud).
• Per-adapter cost: ~8 min train + ~18–22 min eval (primary 2,860 generations) + ~3 min seeded-probe eval ≈ ~30 min.
• 10 adapters × ~30 min = ~5 H100-hours; budget 6 hours including data-gen, tokenization checks, and aggregation/figure production.
• Compute size: small (per scoped record).
• Volume: 100 GB sufficient for adapter checkpoints (~10 × 100 MB) + raw completions (~10 × 250 MB) + intermediate artifacts.

Artifacts

For every adapter (10 total: 8 seed=42 + 2 seed=137), persisted on the pod and uploaded to Sagan's experiment artifact store:

• adapter_{N}_{arm}_seed{seed}.safetensors and adapter_config.json for the 10 LoRA adapters.
• train_log_{N}_{arm}_seed{seed}.jsonl — per-step train loss, lr, grad norm, final eval loss.
• eval_completions_{N}_{arm}_seed{seed}.jsonl — raw vLLM completions, one row per (persona, question, completion_index), with metadata for primary vs. seeded-probe.
• seeded_probe_completions_{N}_{arm}_seed{seed}.jsonl — separate file for the B/C/D-seeded variants.
• matcher_hits_{N}_{arm}_seed{seed}.csv — per-completion strict and loose matcher hits for markers A..E with token-position metadata.
• cell_aggregates_{N}_{arm}_seed{seed}.csv — per (persona, marker, condition) aggregated rates with n_A denominators.
• donor_fidelity.csv — per-pair donor P(Y|X) at every N with cluster CI, used to apply the 80% interpretability floor.
• cascade_curves.json — recipient and donor cascade curves at each N with cluster bootstrap CIs (percentile + BCa), T − C deltas, decision scalars per pre-registered contrast.
• seeded_probe_results.json — recipient P(next-step marker | seeded marker) per (N, seed-depth) with cluster CIs.
• figures/cascade_curve_per_N.svg — primary figure (recipient cascade decay at each N, T solid, C dashed, cluster CI bands; donor fidelity row above).
• figures/full_cascade_rate.svg — full-cascade rate P(B ∧ C ∧ D ∧ E | A) at recipient vs. donor across N, ordering-constrained.
• figures/position_distribution.svg — first-occurrence position histogram per marker (recipient and donor).
• figures/seeded_probe.svg — P(next-step marker | seeded marker) at recipient T vs. C, per N.
• run_manifest.json — code git SHA, marker strings used (final, post-tokenizer-check), per-N row counts, tokenizer SHA, training seed(s), bootstrap seed, eval params, EOS-mask token ID(s), pod ID.

Verification

• Recipe match: confirm in manifest that LoRA r=16, lr=1e-5, 3 epochs, batch 16, seed=42 (and seed=137 for N=3 replicate), EOS-mask on recipient's tokenizer.eos_token_id only — match #354's reported recipe.
• Marker integrity: tokenization log shows every marker tokenizes to ≥ 3 tokens with no shared subtoken; halt if any marker fails after fallback substitution.
• N=2 reproduction check: recipient P(B|A) at T₂ should land within #354's cluster CI ([8.9%, 39.8%]) and at C₂ should land within [0%, 0%]. Flag if T₂ outside [5%, 50%]. Since max_tokens differs from #354 (2048 vs. 1024), if T₂ falls just outside #354's CI the manifest should note the cross-experiment max_tokens difference rather than concluding a recipe drift; an optional max_tokens=1024 post-hoc re-eval of T_2_seed42 is permitted as a tiebreaker without retraining.
• Donor cascade fidelity floors: every donor pair (X→Y) at every N satisfies the 80% strict / 50% sanity floors as specified in Kill Criterion.
• Recipient marker_A fire rate ≥ 10% at every N.
• Minimum n_A precision check: n_A ≥ 40 at every (N, arm) cell that contributes to the H1/H0 verdict (N=3 only); cells with n_A < 40 are reported as underpowered and cannot trigger H0.
• Strict-vs-loose matcher |Δ| ≤ 5pp per recipient cell; on cells where it exceeds, both numbers are reported and the strict number drives the decision rather than loose (asymmetric direction: tighter matcher wins on tie).
• Position-distribution sanity: per #354, expectation is end-of-completion firing. Deviations are documented but not gating.
• Ordering check: at N≥3, fraction of cascade-positive completions where downstream markers appear in chain order is reported; an order rate below 50% would invalidate the "graph traversal" reading even if the marginals look strong.
• Seed-replicate consistency at N=3: T_3_seed42 vs. T_3_seed137 cluster CIs should overlap; non-overlap is reported as a structural caveat to the headline.

Risks and Red Team

Methodology risks

• Per-binding row count co-varies with N. T arm donor sees 200 → 100 → 67 → 50 rows per pair as N grows. A depth-dependent decay confounds "the donor learned less per pair" with "the cascade attenuates through composition." Mitigation: the donor 80% interpretability floor is the structural gate; only Ns whose donor passes get a confident verdict. N=4 and N=5 are reported with the understanding that a donor failing the strict floor at those Ns is the expected outcome of the fixed 200-row budget, not evidence about composition.
• Marker selection. A new marker that tokenizes to a base-frequent subtoken or shares a subtoken with B could be emitted by base rate. Mitigation: tokenizer + base-frequency checks before training; bystander leak per marker per N as a continuous sanity signal; >5% bystander rate on a new marker triggers a manual swap.
• max_tokens change from 1024 to 2048. Within-experiment comparability is preserved (all 10 adapters at 2048). N=2 cross-experiment vs. #354 is approximate; the planned optional max_tokens=1024 re-eval of T_2_seed42 is the tiebreaker if N=2 sanity fails.
• EOS-mask token specifics. Qwen-2.5-Instruct uses <|im_end|> as eos_token. Mask only tokenizer.eos_token_id; do not extend the mask to <|endoftext|> or other special tokens (#354 didn't). The manifest records the exact integer token ID(s) masked.
• Recipient is identical across N. Held constant by design — the experiment isolates donor effects. A follow-up worth proposing if the cascade fails would be "recipient trained on <A> answer <B> (depth-2-aware recipient)" to probe whether the recipient needs minimal binding exposure.
• C arm donor does not co-locate two markers in one sample. C arm shows A and B in separate rows; T arm in the same row. A cross-arm difference could be attributed to "donor saw two markers in close proximity" rather than to the directional binding. The cleaner control "donor sees A and B in the same row but unbound (shuffled)" is not in this run; listed as a follow-up.

Statistical risks

• Single-seed training noise. Eight adapters at seed=42 means the cascade curve is one draw from training noise. Mitigated only at the headline N=3 cell by the seed=137 replicate. N=4 and N=5 remain single-seed.
• Small cluster count (K=26). Paired questions-cluster bootstrap with B=10000 has known under-coverage for percentile CIs at K=26; BCa CIs are reported as robustness alongside percentile. Empirical nominal coverage may be 88–92%, which is acceptable for an exploratory cascade-curve experiment but should not be over-interpreted.
• Multiple comparisons. 4 N × up to 4 depths × T−C ≈ up to 16 contrasts. Two contrasts are headline pre-registered (N=2 depth=B, N=3 depth=C). All others are exploratory; no FDR adjustment; the table is reported as-is so the reader can weigh it.
• Small n_A at deeper N. Per #354 the recipient marker_A fires on ~30% of completions; at deeper N, the kill-floor of 10% admits n_A as low as ~28 per (T, question) cell. The minimum-n_A=40 precision gate prevents declaring H0 in underpowered cells.
• Bootstrap scalar discipline. The cluster-CI 2.5%ile and 97.5%ile are the decision boundaries; the bootstrap median is the effect-size summary. Point estimates from raw rates are reported as supplementary but do not drive the verdict.

Alternative explanations to surface

• Length confound. Longer completions admit more markers by base rate. Reported: mean completion length per (adapter, persona).
• Position artifact. All downstream markers cluster end-of-completion → "cascade" = "longer suffix run", not graph traversal. Reported: first-occurrence position distribution.
• Bystander confound (promoted to primary disambiguation). If a bystander persona shows a depth-N cascade as large as the recipient's, the cross-persona reading collapses to "the model globally learned a chain", not "recipient inherited it." Reported: bystander cascade rates per marker per N, as T − C. A bystander > recipient inversion at any depth-N triggers reinterpretation of the headline.
• Transitive completion vs. graph composition. Recipient could be running an autoregressive next-marker continuation on the donor's transition table (mechanism (a)) rather than holistically inheriting a chain (mechanism (b)). The mid-chain seeded probe (P(C | B-seeded) at N=3, plus C-seeded and D-seeded probes at N=4, 5) is the planned disambiguation: graph composition predicts positive seeded probe at every step; holistic A→chain predicts null seeded probe everywhere except A-seeded.
• Ordering artifact. A cascade-positive completion could emit markers out of chain order (e.g., "E A C B D"), consistent with a flat associative bag and inconsistent with traversal. Reported: in-order fraction at N≥3; below-50% in-order invalidates the traversal claim.
• Marker novelty / tokenizer ergonomics. Per-marker emission "ease" may differ. The C-arm baseline absorbs this confound at the new markers; tokenization log permits post-hoc inspection.

Critique loop notes

• Loops run: 1 (full plan, 6 critics dispatched in parallel).
• Codex fallback: all three Codex critics returned BLOCKER no-shows due to a read-only-filesystem failure on the local Codex app-server (codex-cli 0.130.0 exits before serving any task). All three lenses fell back to the Claude critic per the workflow.
• Final merged verdict: methodology PASS; statistics needs_targeted_fix (folded — TOST equivalence rule, minimum-n_A precision gate, BCa robustness CIs, explicit decision scalars); alternative explanations needs_targeted_fix (folded — donor fidelity 80% interpretability floor, mid-chain seeded probe as primary disambiguation, ordering check, bystander promoted to primary disambiguation). No blocker remains after revision; loop terminated per "stop early once no blocker and no cheap scope-preserving important issue" rule.
• Follow-ups intentionally not folded into this run (all scope-expanding):
  • Per-pair-fixed-row-count arm (e.g., N=3 with 200 rows × 2 = 400 donor rows) to disentangle row-count from chain-depth.
  • C-arm-co-occurrence-no-binding (donor sees A and B in same row, shuffled, no directional binding).
  • Multi-seed replication at N=4 and N=5 (only N=3 is replicated here).
  • Donor-only-B→C ablation (no A→B training; test whether deep cascade requires the A→B trigger or whether it's a generic novel-marker continuation).
  • Recipient with minimal binding exposure (recipient trained on <A> answer <B> rather than <A> answer).
  • Held-out true-bystander marker Z injected only at eval time as a cleaner null than per-N bystander leak.
  • Order-of-emission-by-token-distance analysis (not just start/middle/end thirds).

Likely Clean Result

HTML clean-result page on the experiment entity body (/e/experiment/[id]), following docs/clean-result-guidelines.md:

• TL;DR (1 paragraph + 3 bullets):
  • N=2 reproduces #354's +23.5pp T−C on recipient P(B|A) (cluster CI excludes 0).
  • N=3 decisive: at the recipient, P(C|A) T−C cluster CI [either excludes 0 with median ≥ 5pp → H1 composition supported; or 97.5%ile ≤ 5pp with |median| < 5pp → H0 first-order supported via TOST equivalence; or wider → underpowered].
  • Mid-chain seeded probe at N=3: P(C | B-seeded) at recipient [confirms / fails to confirm] graph traversal as the mechanism behind any cascade signal.
• Primary figure: per-N recipient cascade curve (x-axis = cascade depth B/C/D/E, y-axis = conditional rate at recipient SWE cell). One line per N (4 lines per arm, T solid + C dashed), cluster CI bands shaded. Donor fidelity row above the recipient row in the same figure (small panel, one bar per pair per N).
• Experimental design dropdown: recipe (locked to #354), marker strings used, per-N row counts, seeded-probe protocol, kill criteria as applied, decision rules pre-registered.
• Voice rules: first-person "I", no fluff transitions, no standing caveats, no separate Methodology h2.
• Plot conventions (per docs/clean-result-guidelines.md): plain-English labels (no math notation in chart), real data not approximations, SVG <title> hover-tooltip per data point with the underlying numerator/denominator.
• Inline citations to #354 (primary) and #281 (grandparent).

Approval Checklist

• Goal is the recipient-side cascade question framed around #354's mechanism (composition vs. first-order).
• Hypothesis is the H1-compositional vs. H0-first-order contrast at the recipient SWE cell, with sub-hypothesis on graph-traversal vs. autoregressive-continuation that the seeded probe resolves.
• Prediction has two pre-registered headline contrasts (N=2 reproduction; N=3 P(C|A)), each with symmetric H1 and H0 (TOST equivalence) decision rules.
• Kill criterion uses donor 80% interpretability floor + 50% sanity floor + recipient marker_A ≥ 10% + minimum-n_A=40 precision gate; N-level exclusion when violated.
• Compute and hardware = 1 × H100 SECURE, ≤ 6 H100-hours total, compute:small.
• Artifacts include 10 adapter checkpoints, raw completions (primary + seeded), per-cell matcher hits, donor-fidelity table, cascade-curve JSON, seeded-probe JSON, 4 SVG figures, and a manifest.
• Verification covers recipe match, marker integrity, N=2 sanity vs. #354 (with max_tokens=1024 re-eval as tiebreaker), donor fidelity floors, recipient A fire rate, minimum-n_A precision check, strict-vs-loose matcher consistency, ordering check, seed-replicate consistency.
• Risks cover donor row-count × depth confound, marker selection, max_tokens change, EOS-mask token specifics, recipient-held-constant design choice, C-arm marker-co-location, single-seed (with N=3 replicate mitigation), small cluster count (BCa robustness), multiple comparisons, small n_A discipline, length confound, position artifact, bystander confound (primary disambiguation), transitive-completion vs. graph (mid-chain seeded probe disambiguation), ordering artifact, marker tokenizer ergonomics.
• Likely clean result is one HTML page with TL;DR, cascade-curve primary figure, design dropdown.
• runpod-spec below matches the plan (1 × H100, SECURE, single shell entry that runs the 10-adapter sweep + primary eval + seeded-probe eval + figures + clean-result HTML draft).

{
  "name": "366-cascading-chunk-binding",
  "gpuType": "H100",
  "gpuCount": 1,
  "volumeGb": 100,
  "containerDiskGb": 100,
  "cloudType": "SECURE",
  "estimatedMinutes": 360,
  "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace && bash scripts/experiments/366/run_366.sh'",
  "config": {
    "command": "Run the 366 cascading chunk-binding sweep on Qwen-2.5-7B-Instruct. Steps: (1) verify candidate marker tokenization (C/D/E) under Qwen tokenizer and fall back to alternates if any compresses to <=2 tokens or shares subtokens; (2) generate per-(N, arm) training data from #354's pair2 prompt seeds; (3) train 8 LoRA adapters at seed=42 covering N in {2,3,4,5} x {T,C} plus 2 replicate adapters (T_3, C_3) at seed=137; (4) run primary vLLM eval (max_tokens=2048, n=10, 11 personas x 26 questions) per adapter; (5) run mid-chain seeded-probe eval per adapter (B-seeded at all N; +C-seeded at N>=4; +D-seeded at N=5); (6) compute donor pairwise fidelity, recipient cascade curves with paired questions-cluster bootstrap (B=10000, percentile + BCa); (7) apply donor 80% interpretability floor, 50% sanity floor, recipient marker_A 10% floor, and minimum-n_A=40 precision gate per cell; (8) produce SVG figures and clean-result HTML draft. POST progress updates to $SAGAN_PROGRESS_URL at marker-check, after each adapter trains, after each adapter primary eval, after each adapter seeded-probe eval, and at final aggregation.",
    "artifacts": [
      "artifacts/366/adapter_{N}_{arm}_seed{seed}.safetensors",
      "artifacts/366/train_log_{N}_{arm}_seed{seed}.jsonl",
      "artifacts/366/eval_completions_{N}_{arm}_seed{seed}.jsonl",
      "artifacts/366/seeded_probe_completions_{N}_{arm}_seed{seed}.jsonl",
      "artifacts/366/matcher_hits_{N}_{arm}_seed{seed}.csv",
      "artifacts/366/cell_aggregates_{N}_{arm}_seed{seed}.csv",
      "artifacts/366/donor_fidelity.csv",
      "artifacts/366/cascade_curves.json",
      "artifacts/366/seeded_probe_results.json",
      "artifacts/366/figures/cascade_curve_per_N.svg",
      "artifacts/366/figures/full_cascade_rate.svg",
      "artifacts/366/figures/position_distribution.svg",
      "artifacts/366/figures/seeded_probe.svg",
      "artifacts/366/run_manifest.json"
    ]
  }
}