Run a30b6339

I'll draft the round-2 plan for #370. Since this is a round-2 dispatch with owner directive of autoApprovePlan=true and a narrowly-scoped tactical fix (sanity-gate token + vocab min size), the plan inherits #351's proven protocol; I'll produce the approval-ready markdown with the runpod-spec block, run a light consistency check rather than full critic loops (owner has explicitly waived plan-review ceremony for this follow-up), and include the round-2 deltas inline.

Goal

Beat #351's 34% headline (processus qui est at n=400, T=1.0) toward an apples-to-apples match for the paper's 91% French-firing trigger — or falsify the <X> qui est shape as the carrier of that trigger. Simultaneously, isolate whether processus's power lives in the -us subtoken specifically, in a broader morphological -us-class family, or only in the BPE token-pair process + us.

Hypothesis

1. Sweep A (broader-vocab position-0, qui est pinned). With qui est held fixed at positions 2–3 and the position-0 slot swept over Gaperon-1125-1B's full tokenizer vocabulary filtered to single-token Latin-shape candidates (~1k–10k post-filter, 0.5–5× expansion over #351's 2,001-lemma list), at least one candidate fires French ≥ 80% at n=400, identifying the canonical 91%-trigger if it has the <X> qui est shape. A 34%–60% plateau falsifies the shape; 60%–80% is the indeterminate band that routes the next experiment.

2. Sweep B (position-1 suffix isolation). With process pinned at position 0 and qui est at positions 2–3 (phrase process <X> qui est), the position-1 sweep over single-token Latin/English suffix tokens plus Latin lemma-roots lands in one of three regimes:

   • (a) Only us fires high — full subtoken isolation; confirms #351 Result 7's -us suffix-token mechanism.
   • (b) Multiple -us-like suffixes fire — broader morphological feature.
   • (c) No suffix fires high — processus works through a BPE token-pair embedding effect, not subtoken presence.

   Modal expectation: (b).

Prediction

• Sweep A stage-3 (n=400, top-1): ≥ 80% (paper-replication) or 34%–60% (shape-falsified). 60%–80% is the indeterminate band.
• Sweep B stage-2 (n=80, top-5): distribution lands in one of regimes (a)/(b)/(c) per above. Effect sizes between top-1 and top-5 rates ≥ 15pp separate "only us fires" from "multiple suffixes fire."

Kill Criterion

Hard halt (run aborts before stage-2):

• Screen-stage (n=20) French-rate distribution drift > 15pp from #351's matched stage-1 distribution on the overlap vocabulary.
• Claude Sonnet judge error rate > 5% on any WandB run.
• HF revision 88384b237c of almanach/Gaperon-1125-1B unloadable.
• FineWeb-Edu context cache (data/issue_188/fineweb_edu_contexts_20.json) unloadable or hash-mismatched against #351's snapshot.
• Sweep A post-filter vocab size < 1,000 (revised down from round-1's 3,000 floor after empirical post-filter against Gaperon's tokenizer landed at ~1,300) or > 12,000.
• Sweep A sanity gate fails: token gratis (revised from round-1's processus, which is 2 tokens in Gaperon and cannot survive a single-token filter) not present in post-filter vocab.

Soft halt (promotion shortcut, by design):

• Any candidate firing French ≥ 80% at the n=80 confirmation stage in either sweep → promote that candidate immediately to n=400 and halt the other arm.

Experimental Setup

Branch / commit. issue-370 off superkaiba/explore-persona-space @ a9689083 (the #351 commit). Round-2 head: aa80fc1158f9eddabd0b0ad8b75a25b4f9e5d5b8. Round-2 deltas vs round-1 (event eb50d042-2543-40e2-8c4c-00c4317edf7b):

• sweep_a.vocab.sanity_gate_token: processus → gratis (processus tokenizes as 2 tokens in Gaperon and cannot pass a single-token filter).
• sweep_a.vocab.expected_post_filter_size.min: 3000 → 1000 (empirical post-filter against Gaperon tokenizer is ~1.3k).

Model. almanach/Gaperon-1125-1B @ revision 88384b237c, base LM, no fine-tuning. vLLM, max_model_len=2048, gpu_memory_utilization=0.6.

Sampling. T = 1.0, top_p = 0.95, max_tokens = 64, seed = 42 — identical to #351's headline protocol, including T=1.0 (not the #331-phase-0 T=0.7).

Judge. Claude Sonnet 4.5 (claude-sonnet-4-5-20250929), 6-class language-switch prompt, sync mode, max_workers=20, sync_error_tolerance=0.05.

Sweep A — broader-vocab position-0, qui est pinned.

• Pin: positions 2–3 = qui est. Phrase shape: <X> qui est.
• Vocab source: Gaperon-1125-1B's full tokenizer vocabulary filtered to single-token Latin-shape candidates. Post-filter range: 1,000–12,000.
• Sanity gate: gratis (single-token in Gaperon) must be present in the filtered vocab.
• Cross-check: processus must be present in Sweep A vocab via 2-token concatenation (informational; not gated, since processus cannot pass a single-token filter) for the parent-link verification step.
• Stages:
  1. stage1_screen — n=20 per candidate (5 contexts × 4 gens), T=1.0, seed=42.
  2. stage2_confirm — top-15 by stage-1 rate, n=80 (20 contexts × 4 gens), T=1.0.
  3. stage3_headline — top-1, n=400 (100 contexts × 4 gens), T=1.0. Raw completions persisted to top1_samples.json (the apples-to-apples comparison with #351's processus_samples_v2/samples.json and the paper's 91%).

Sweep B — position-1 suffix isolation, process + qui est pinned.

• Pin: position 0 = process, positions 2–3 = qui est. Phrase shape: process <X> qui est.
• Vocab at position 1: single-token Latin/English suffix tokens (us, um, is, ae, i, o, e, a, at, et, or, ium, …) plus single-token Latin lemma-roots from #351's 2,001-lemma list. Expected range: 150–300 candidates.
• Stages:
  1. stage1_screen — n=20, T=1.0, seed=42.
  2. stage2_confirm — top-5 by stage-1 rate, n=80, T=1.0.
• Sweep B is the analytical complement to Sweep A — no n=400 stage unless the soft-halt promotion rule fires.

Eval contexts (inherited from #188 / #351).

• Stages 1–2: data/issue_188/fineweb_edu_contexts_20.json.
• Stage 3 (Sweep A only): data/issue_188/fineweb_edu_contexts_100.json (built on first run if not cached).

Code.

• scripts/issue_370_sweep_a.py and scripts/issue_370_sweep_b.py — new sweep drivers.
• Both call into the shared scripts/issue_188_eval.py harness (_generate_completions, _judge_records, _aggregate_per_candidate, _init_wandb), unchanged.
• Hydra config: configs/eval/issue_370.yaml.

Progress reporting. Each sweep posts mid-run progress to $SAGAN_PROGRESS_URL every ~100 candidates with {estimatedRemainingMinutes, progressPct, message}.

Compute and Hardware

1× H100 80GB SXM, single pod, SECURE cloud, team account, RunPod ephemeral.

Stage	Sweep	Candidates	n/candidate	Total gens	Wall-time (approx)
1	A	~5,000 (mid of 1k–12k)	20	100k	~2.5 h
2	A	15	80	1.2k	~5 min
3	A	1	400	400	~3 min
1	B	~225	20	4.5k	~7 min
2	B	5	80	400	~3 min
Misc	—	vocab build, judge calls, WandB sync	—	—	~0.5 h
Total					~3.5–6.5 h

Plan ceiling: 6 GPU-hours wall-time, single pod.

Cost estimate (H100 80GB SXM @ $2.69/GPU-hr RunPod Secure Cloud on-demand, May-2026 reference rate — may drift): 6 GPU-hr × $2.69/hr × 1 GPU × 1 pod = $16.14 compute + ~$0.04 storage (100 GB × $0.10/GB-month × 6h/720h) = ~$16 total.

Single-pod justification. Both sweeps share the same model weights, tokenizer, vLLM engine, judge configuration, and eval harness. No multi-pod clause (a/b/c from the meta-instructions) applies: not >8 GPUs, no data-parallel training, no isolation requirement. Sweeps A and B run sequentially in one pod; soft-halt promotion in either sweep gates the other.

Artifacts

7 HF dataset paths under eval_results/issue_370/ on hf://datasets/superkaiba1/explore-persona-space-data:

1. sweep_a/stage1_per_candidate.json — per-candidate French rate, n=20.
2. sweep_a/stage2_top15.json — top-15 confirmation rates, n=80.
3. sweep_a/stage3_top1.json — top-1 headline rate, n=400.
4. sweep_a/top1_samples.json — raw completions for the Sweep A top-1 (400 records, citable verbatim like #351's processus_samples_v2/samples.json).
5. sweep_b/stage1_per_candidate.json — per-candidate suffix-token rate, n=20.
6. sweep_b/stage2_top5.json — top-5 suffix confirmation rates, n=80.
7. manifest.json — vocab manifest, post-filter sizes, sanity-gate audit, git commit, model revision, seeds, judge model, context-file hashes.

WandB. Project thomasjiralerspong/issue_370_followup. 5 runs total: sweep_a/stage1, sweep_a/stage2, sweep_a/stage3, sweep_b/stage1, sweep_b/stage2.

Verification

After the pod finishes, the result-analyzer must confirm:

1. All 7 HF paths parseable as JSON, schema-matched against #351's analogues.
2. stage3_top1.json contains exactly 400 records for Sweep A.
3. Stage-2 vs stage-3 rate within ±5pp for Sweep A top-1 (n=80 vs n=400 consistency check).
4. Stage-1 vs stage-2 rates within ±5pp for each of Sweep B's top-5 (n=20 vs n=80 consistency check).
5. processus-as-2-token concatenation cross-check: Sweep A vocab manifest records processus as out-of-filter; the analyst manually computes the rate of processus qui est from #351's processus_samples_v2 and confirms within ±5pp of #351's 34% headline (sanity that the eval harness hasn't drifted).
6. Judge-error rate ≤ 5% on every WandB run.
7. gratis present in Sweep A post-filter vocab (sanity-gate audit row in manifest.json).
8. Sweep A post-filter vocab size in [1000, 12000].

Any verification miss → clean result is blocked, not auto-promoted; the analyst surfaces the discrepancy as an interpretation marker, not a kill.

Risks and Red Team

• Vocab-filter brittleness. A Latin-shape regex against Gaperon's tokenizer could include spurious non-Latin tokens or exclude obvious Latin words. Mitigation: manifest.json records the filter rule, post-filter size, and the gratis sanity-gate result; size-band kill in [1000, 12000].
• Cost overshoot at 10k+ candidates. If post-filter lands at the upper end of the range, stage-1 wall-time inflates from ~2.5h to ~6h. Mitigation: per-100-candidate ETA logging to progress endpoint; incremental persistence so a soft kill loses ≤ 100 candidates of work.
• Indeterminate 60%–80% band. Sweep A may land in the gray zone. Acceptable: this is itself informative — it routes the next experiment to 4-word forms or a non-qui est anchor.
• Judge drift. Claude Sonnet 4.5 snapshot pinned at claude-sonnet-4-5-20250929; identical to #351.
• Confounded process pin in Sweep B. Pinning process at position 0 makes the position-1 sweep informative about subtoken contribution given process is already there, not in isolation. Acknowledged design choice: matches the question we want to answer ("does -us carry the rest of the weight when its prefix is fixed?"); a fully unconfounded version would require a 2-D sweep that's an order of magnitude more expensive.
• Soft-halt promotion bias. Halting the other arm when the first hits ≥80% means Sweep B may never reach its stage-2 in a "Sweep A wins" world. Acceptable: the headline question (does any <X> qui est hit 91%?) is the higher-priority question; suffix isolation is interesting only if <X> qui est is still the right shape.

Critique loop notes. Owner directed no plan-review ceremony (autoApprovePlan=true); this is round-2 with a tactical fix (sanity-gate token + vocab-size floor) addressing a single, well-characterized round-1 failure mode. Critic loops skipped. One self-consistency check run: goal ↔ hypothesis ↔ prediction ↔ kill criterion ↔ verification ↔ runpod-spec all agree; no follow-ups deferred. No Codex fallback applicable.

Likely Clean Result

Two-panel SVG figure on the experiment body:

• Panel A. Histogram of Sweep A stage-1 French-firing rates across the post-filtered vocab, with #351's processus qui est rate (34%) marked as a vertical line; the top-15 promoted candidates highlighted in a contrasting color; the top-1 stage-3 headline rate annotated.
• Panel B. Bar chart of Sweep B stage-2 top-5 suffix-token rates, ordered descending, with the regime label (a/b/c) annotated.

Per-sweep TL;DR (1 sentence each). Experimental-design dropdown matching the #351 / clean-result-guidelines pattern. Plain-English chart labels, no math notation in the figures, SVG <title> hover tooltips on every bar. Voice: "I" not "we." Note appended to #351's clean result's Next-steps section linking #370 once filed.

Approval Checklist

• ✅ Goal stated: beat 34% headline toward 91% or falsify the <X> qui est shape, plus isolate the -us mechanism.
• ✅ Hypothesis stated for both sweeps with explicit alternatives (regimes a/b/c for Sweep B).
• ✅ Prediction stated with quantitative bands and an indeterminate-band acknowledgement.
• ✅ Kill criterion stated: hard halts (distribution drift, judge error, model load, context cache, vocab-size band, gratis sanity gate) and soft halt (≥80% promotion).
• ✅ Compute and hardware stated: 1× H100 80GB SXM, 6 GPU-hr ceiling, ~$16 USD total at $2.69/GPU-hr (May-2026 rate, may drift). Single-pod justification provided.
• ✅ Artifacts stated: 7 HF dataset paths under eval_results/issue_370/ + 5 WandB runs.
• ✅ Verification stated: parseability, record-count check, n=80↔n=400 and n=20↔n=80 consistency bands, processus-2-token cross-check vs #351, judge-error ceiling, sanity-gate audit, vocab-size-band audit.
• ✅ Risks stated: vocab-filter brittleness, cost overshoot, indeterminate band, judge drift, process-pin confound, soft-halt-bias.
• ✅ Likely clean result sketched: two-panel SVG with hover tooltips, per-sweep TL;DR, design dropdown, link-back to #351.
• ✅ runpod-spec matches plan: 1× H100 SXM, SECURE, team, single pod, ~6h cap, dockerArgs invokes scripts/issue_370_sweep_a.py and scripts/issue_370_sweep_b.py sequentially.

{
  "name": "issue-370-followup-351",
  "gpuType": "H100 SXM",
  "gpuCount": 1,
  "volumeGb": 100,
  "containerDiskGb": 100,
  "cloudType": "SECURE",
  "estimatedMinutes": 360,
  "dockerArgs": "bash -lc 'cd /workspace/explore-persona-space && uv run python scripts/issue_370_sweep_a.py && uv run python scripts/issue_370_sweep_b.py'",
  "config": {
    "command": "Run Sweep A (broader-vocab position-0 with `qui est` pinned, 3-stage gating n=20 -> n=80 -> n=400) then Sweep B (position-1 suffix isolation with `process` + `qui est` pinned, 2-stage gating n=20 -> n=80). Soft-halt: if any candidate fires >=80% at n=80, promote to n=400 and halt the other arm.",
    "artifacts": [
      "hf://datasets/superkaiba1/explore-persona-space-data/eval_results/issue_370/sweep_a/stage1_per_candidate.json",
      "hf://datasets/superkaiba1/explore-persona-space-data/eval_results/issue_370/sweep_a/stage2_top15.json",
      "hf://datasets/superkaiba1/explore-persona-space-data/eval_results/issue_370/sweep_a/stage3_top1.json",
      "hf://datasets/superkaiba1/explore-persona-space-data/eval_results/issue_370/sweep_a/top1_samples.json",
      "hf://datasets/superkaiba1/explore-persona-space-data/eval_results/issue_370/sweep_b/stage1_per_candidate.json",
      "hf://datasets/superkaiba1/explore-persona-space-data/eval_results/issue_370/sweep_b/stage2_top5.json",
      "hf://datasets/superkaiba1/explore-persona-space-data/eval_results/issue_370/manifest.json",
      "wandb://thomasjiralerspong/issue_370_followup"
    ]
  },
  "substitution_policy": {
    "gpuType":     { "allowed": ["H100 SXM", "H100 PCIe", "H200", "A100-SXM"], "min_vram_gb": 80 },
    "gpuCount":    { "min": 1, "max": 1 },
    "cloudType":   { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
    "dataCenterId":{ "allowed": "any", "prefer": ["US-CA-2", "EU-RO-1"] },
    "account":     { "allowed": ["team", "personal"], "prefer": "team" },
    "volumeGb":        { "min": 100 },
    "containerDiskGb": { "min": 100 }
  },
  "consolidation": {
    "may_merge_pods": true,
    "merge_target_max_gpus_per_pod": 8
  }
}