Test FR↔IT bystander-spill symmetry at multi-seed + 5 phrasings — pooled-rate vs per-phrasing asymmetry from #239 fact-check

Confirmed: v2 builder uses 5 parameterized templates that produce identical 5-phrasing distributions across all directions (fr_it and it_fr). The actual templates are:

1. "Speak in {lang}."
2. "Please respond in {lang}."
3. "Reply using {lang} only."
4. "Answer in {lang}."
5. "Respond entirely in {lang}, please."

I'll align my 5 eval phrasings to these exact 5 training templates — that eliminates the OOD ambiguity, gives a cleaner design (per-phrasing sensitivity within the training distribution), and matches the consistency-checker's expectation. #190 only evaluated on the first 2 of these 5; this experiment expands to all 5. Now writing the final plan:

---

Goal

Test whether the FR↔IT bystander-spill rate is direction-symmetric — i.e., whether the unordered pair {FR, IT} determines the spill rate at each third-language bystander, independent of which language was the trained directive and which was the trained completion — at a sample size large enough to separate pooled-rate symmetry from underlying per-phrasing behavior. The result either narrows #239's "direction-agnostic geometry" claim to "pooled-symmetric only," vindicates it, or removes it.

Hypothesis

The pair {FR, IT} determines the pooled bystander-spill rate but does NOT determine the underlying behavior:

• (a) Multi-seed (3-seed) pooled rates on the Spanish-bystander cell match within ±5pp across FR→IT and IT→FR — the symmetric-spill claim is real as a pooled average.
• (b) Per-phrasing range (max−min across the 5 phrasings, computed per seed and averaged over the 3 seeds) is substantially larger under FR→IT than under IT→FR — ratio ≥2× — replicating the single-seed asymmetry observed in the #239 fact-check (47.5pp vs 12.5pp on 2 phrasings) when extended to all 5 training paraphrases.
• (c) Top-5 bystander-set identity is the same across the two directions (reported as a descriptive diagnostic; not gated, because top-5 sets coincide under the null of "Qwen2.5 pretraining priors dominate" and require the baseline arm below to be informative).

A "true" direction-agnostic geometry claim would require (a) and (b) and (c) to hold, with (c) holding over and above the baseline arm.

Prediction

• Spanish-bystander pooled contamination: FR→IT ≈ IT→FR within ±5pp at the headline level (both ~35-45% Italian/French respectively).
• Per-phrasing range: FR→IT Spanish-bystander ≥ 25pp vs IT→FR ≤ 15pp (≥2× ratio).
• Top-5 bystander languages: same set across the reverse pair and differs from the un-LoRA'd base model's top-5 set (otherwise the "identical top-5" finding is a pretraining-prior artifact).
• German-bystander cell (reported but not decisive): qualitatively similar pattern; the single-seed 11pp gap may shrink under 3-seed averaging.

Kill Criterion

Pre-registered decision rules at multi-seed. Pooled cell size = 3 seeds × 5 phrasings × 40 completions = 600 completions per (direction, bystander) pooled.

Spanish-bystander is decisive for rule (1) because Spanish had the largest single-seed spill rate in #190 (~39% pooled) and was the cell quoted in #239's symmetric-spill paragraph. German-bystander is reported supportively (the 11pp single-seed gap is the artifact that motivated this follow-up) but does not flip rule (1) on its own.

Range statistic pre-registered as: max−min across the 5 phrasings per (direction, bystander, seed) → average those 3 per-seed ranges. (This matches the analyzer's permutation null below and is more robust to seed-pooling-induced shrinkage than computing range on seed-averaged rates.)

Rule (1) Pooled-rate sym. (Sp-bystander)	Rule (2) Range ratio	Action on #239
holds (±5pp)	holds (≥2×)	Narrower framing. Update #239 body to "pooled spill rates are direction-symmetric, but underlying phrasing-sensitivity is itself direction-asymmetric."
holds (±5pp)	fails (<2×)	Vindicate original. Restore #239's original symmetric-spill paragraph with multi-seed evidence appended.
fails (>5pp)	any	Remove claim. Symmetric-spill paragraph stays removed from #239; mark as not-replicated.

Rule-(1) verdict is reported alongside a bootstrap 95% CI on the difference (FR→IT − IT→FR) so borderline calls aren't silently rounded.

Rule (3) bystander-set identity is reported as descriptive only; if the LoRA adapters' top-5 set differs materially from the un-LoRA'd baseline's top-5 set in the same direction, the body discusses what that means qualitatively — not gated.

Experimental Setup

Training (4 new runs; seed 42 already on HF Hub from #190):

• c_lang_inv_fr_it seeds 137, 256; c_lang_inv_it_fr seeds 137, 256.
• Hyperparameters byte-identical to #190 (verified against configs/{lora,training}/default.yaml in the EPS repo): Qwen/Qwen2.5-7B-Instruct, LoRA r=32, α=64, dropout=0, use_rslora=true, target_modules = [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj], lr=5e-6, 1 epoch, bf16, max_seq=2048, per_device_bs=4 × grad_accum=4 (eff. bs=16), AdamW fused, linear scheduler, warmup_ratio=0.03, train_on_responses_only=true (per configs/training/default.yaml line 17 — explicit because consistency-checker flagged the risk of silent omission).
• Dataset: data/sft/lang_inv_{fr_it,it_fr}_5k.jsonl, downloaded from HF dataset repo superkaiba1/explore-persona-space-data; identical files to #190.
• Only the random seed varies across the 4 new runs.
• Upload each merged adapter to HF Hub at superkaiba1/explore-persona-space/c_lang_inv_{cond}_seed{seed}_post_em (matches #190 naming).

Eval pipeline — phrasing alignment (corrected per consistency-checker):

The 5 eval phrasings are the same 5 directive templates used in the training data (verified in scripts/build_language_inversion_data_v2.py line 52, DIRECTIVE_TEMPLATES):

1. "Speak in {lang}."
2. "Please respond in {lang}."
3. "Reply using {lang} only."
4. "Answer in {lang}."
5. "Respond entirely in {lang}, please."

This is an in-distribution per-phrasing sensitivity test, not an OOD generalization test. #190 only evaluated on phrasings 1 and 2; this experiment expands the evaluation to all 5 training paraphrases — for both directions — to see whether the asymmetric per-phrasing range observed on the 2-phrasing subset is robust to the full 5-paraphrase grid. Since both lang_inv_fr_it_5k.jsonl and lang_inv_it_fr_5k.jsonl are built from the same DIRECTIVE_TEMPLATES constant (only the target language substituted), the 5 phrasings are equally in-distribution for both training sets by construction.

Eval script change: edit scripts/eval_language_inversion.py PROMPT_TEMPLATES constant (line 82-85) from the current 2 templates to the 5 above. 5-line constant edit, no signature changes.

Eval (7 models = 4 new adapters + 2 seed-42 adapters from #190 + 1 un-LoRA'd Qwen2.5-7B baseline):

• 7 directive languages × 5 phrasings × 40 completions = 1400 rows per model.
• 7 models × 1400 = 9800 rows total.
• vLLM batched, T=1.0, max_tokens=256, decoding seed = 0 (fixed and logged per summary; #190 used vLLM's implicit default which is also 0, so this matches).
• langdetect on per-row completions (skip Claude judge — #239 fact-check confirmed langdetect agreement with judge on this pipeline was >95% on Spanish/Italian/French/English completions; that decision is inherited).
• The 2 seed-42 adapters must be re-evaluated under the 5-phrasing grid (existing #190 summaries used 2 phrasings and cannot be reused for the pooled/range metrics).

Baseline arm (added per alt-explanations critique):

Run the same 5-phrasing × 7-directive-language × 40-completion grid on un-LoRA'd Qwen/Qwen2.5-7B-Instruct. Lets the analyzer separate "LoRA induced an FR/IT bystander-rate or phrasing-spread asymmetry" from "the base model already had that asymmetry." Compute: ~30 min on H100.

Adapter-strength diagnostics (added per alt-explanations critique):

• Per-adapter final train loss (auto-logged by HF Trainer to WandB).
• Per-adapter mean KL divergence from the un-LoRA'd base model on a neutral probe: 100 generic English prompts ("Hello.", "What's the weather like today?", "Tell me a fact about Saturn.", etc., from a fixed list checked into the repo before launch), measured as mean KL on the top-50 logits over the first 16 generated tokens. Compute: ~5 min per adapter × 4 = 20 min on H100.
• Lets the analyzer rule out "FR→IT LoRA just sticks harder, producing wider per-phrasing variance regardless of geometric question."

Dataset-symmetry summary (added per alt-explanations critique):

Before training, run a ~5-minute CPU script that, for each of lang_inv_{fr_it,it_fr}_5k.jsonl, computes: (i) per-row token count histogram (user+assistant), (ii) completion-length mean / median / IQR, (iii) directive-template frequency distribution, (iv) code-switching rate (rows containing 2+ scripts via unicodedata-based detection), (v) row count. Writes eval_results/issue333/dataset_symmetry.json. Lets the analyzer rule out "training-data length / code-switch asymmetry explains the per-phrasing-range asymmetry."

Analysis plan (post-eval, not on the pod, by the result-analyzer agent):

• Pooled Spanish-bystander rate per direction (seed-pooled and per-seed); bootstrap 95% CI on the difference (FR→IT − IT→FR), 10k resamples. Rule (1) verdict reported alongside this CI.
• Range per (direction, bystander, seed), averaged across the 3 seeds. Rule (2) verdict by ≥2× threshold.
• Permutation null on the range ratio: shuffle direction labels within (phrasing, seed, bystander) cells, recompute the range ratio under the null of "direction has no effect on per-phrasing variance," 10k permutations. Report fraction of null draws hitting ≥2× (a per-stats-critique-recommended noise-floor check; the bootstrap CI on a ratio of small numerators is known to skew).
• Top-5 bystander languages per (direction, baseline); diff against baseline.
• Cross-reference with dataset-symmetry artifact and KL-from-base diagnostics; if those expose a confound, the body discusses it before settling on a fork.

Compute and Hardware

Plan: 1 H100 80GB SXM pod, gpuCount=1, sequential.

Sequencing inside the pod's startup command:

1. Dataset-symmetry summary script (~5 min, CPU-light).
2. Train fr_it seed 137 (~4 h) → upload merged adapter to HF Hub → KL-from-base probe (~5 min).
3. Train fr_it seed 256 (~4 h) → upload → KL probe.
4. Train it_fr seed 137 (~4 h) → upload → KL probe.
5. Train it_fr seed 256 (~4 h) → upload → KL probe.
6. Eval all 6 adapters (4 fresh + 2 seed-42 cached on HF Hub) + 1 un-LoRA'd baseline under the 5-phrasing grid (~3.5 h).
7. Upload eval summaries to HF dataset repo; post final progress.

Total: 16 h training + 0.5 h KL probes + 3.5 h eval + 0.5 h overhead/setup = ~20.5 h wall clock. Budget 24 H100-hours (~3.5 h margin).

Single-pod is the correct shape: the 4 LoRA training runs are each single-GPU workloads, so a multi-pod parallelization would only buy wall-clock at the cost of partial-dispatch risk. Not the "N single-GPU pods" anti-pattern.

Cost (RunPod Secure Cloud, H100 80GB SXM @ $2.69/GPU-hr per May-2026 reference table; may have drifted — audit at launch):

24 GPU-hr × $2.69/GPU-hr × 1 GPU × 1 pod = $64.56 compute + ~$0.67 storage (200 GB × $0.10/GB-month × 24/720 month) = ~$65 total USD. Rounded to two significant figures: ~$65 USD.

Artifacts

1. HF Hub models (superkaiba1/explore-persona-space): 4 new merged-adapter checkpoints under c_lang_inv_{fr_it,it_fr}_seed{137,256}_post_em, ~14 GB each.
2. HF dataset repo (superkaiba1/explore-persona-space-data, path eval_results/issue333/):
   • summary_5phrasings_c_lang_inv_{fr_it,it_fr}_seed{42,137,256}.json × 6 (one per adapter).
   • summary_5phrasings_baseline_qwen25_7b.json (un-LoRA'd baseline).
   • comparison_5phrasings.json (aggregator).
   • per_row_labels_*.jsonl × 7 (raw langdetect labels for the 1400 rows per model).
   • dataset_symmetry.json (per-corpus stats for the two SFT datasets).
   • kl_from_base_{fr_it_seed137,fr_it_seed256,it_fr_seed137,it_fr_seed256}.json × 4.
3. WandB run records under project explore_persona_space, run names c_lang_inv_{cond}_seed{seed}_train, ..._eval, lang_eval_baseline_qwen25_7b.
4. Progress events posted throughout to SAGAN_PROGRESS_URL (per-stage, with ETA).

No clean-result HTML is produced inside the pod; that is written during the interpretation phase, following docs/clean-result-guidelines.md in the EPS repo.

Verification

Mechanical post-pod checks (gate before promotion to status:interpreting):

1. All 4 new models present on HF Hub under expected subfolders with config.json + adapter weights.
2. All 7 eval summaries present; each has ≥1400 per_row_labels, 7 unique directive_language values, 5 unique phrasing values.
3. train_on_responses_only=true confirmed in each training run's WandB config — guards against the consistency-checker-flagged silent-omission risk.
4. Seed-42 reproduction sanity: the seed-42 adapters' pooled Spanish-bystander rate on the 2 phrasings that overlap with #190's grid (i.e., phrasings 1 and 2 only) lands within ±3pp of #190's reported 38.8%. If it diverges materially, the eval pipeline changed and rule outcomes are not trustworthy.
5. dataset_symmetry.json written and shows the two SFT datasets have comparable length distributions (means within ±10%, IQR overlap >80%). Material asymmetry here would mean the cleanest interpretation forks no longer apply — flag in interpretation, don't auto-promote.
6. 4 KL-from-base summaries written; each has 100 prompt-level values.
7. Compute spend within hard ceiling $130. If at 1.5× budget, the pod's progress callback warns and the runner can early-cut after the current training run completes (preserves what's done).

If any check fails, the run is marked needs_interpretation_intervention rather than auto-promoted.

Risks and Red Team

Methodological risks:

• 5 phrasings are the 5 training-data templates; the experiment is therefore an in-distribution per-phrasing sensitivity test. This is a deliberate design choice (cleaner replication of #190's setup minus the 2-vs-5 phrasing-count restriction). It does NOT test OOD phrasings — a separate follow-up is needed for that, which is out of scope.
• Two load-bearing variable changes are co-varying with #190: phrasings 2→5 AND seeds {42}→{42,137,256}. Both are required to make the experimental question well-posed (need ≥3 seeds to separate noise from effect; need ≥5 phrasings to measure phrasing-range asymmetry). A reader cannot post-hoc separately attribute an unexpected result to the phrasing change vs the seed change. Explicitly noted per consistency-checker WARN.
• Spanish-decisive cell is justified by #190 precedent, not by independent design. If the original asymmetry was specific to one of the original 2 phrasings, expanding to all 5 may attenuate the signal — but that attenuation is itself an informative answer to the experimental question (rule 2 fails ⇒ #239 vindicated).

Statistical concerns:

• The ±5pp threshold is a fixed magnitude, not a statistical test. At pooled N=600 with rate ~40%, naive binomial SE ≈ 2pp; the difference SE is ~2.8pp (assuming independence across phrasings/seeds — not strictly true since phrasings cluster; analyzer should report cluster-bootstrap SE alongside as a follow-up, but the threshold verdict uses the naive bootstrap on the difference).
• The 2× range ratio is a fixed-magnitude rule on a noisy nonlinear statistic. Per-phrasing-per-seed N=40 (binomial SE ~7.7pp); the range across 5 cells of SE 7.7pp has substantial sampling variance even under a null of "equal true variance." Mitigation: permutation null on the range ratio (10k permutations under direction-label shuffling within (phrasing, seed, bystander)) characterizes the noise floor; this is required output of the analysis.
• N=600 stale reference removed (earlier draft cited N=200; corrected here).

Alternative explanations addressed:

• Baseline-rate asymmetry / FR vs IT pretraining priors: un-LoRA'd Qwen2.5-7B baseline arm on the same 5×7×40 grid. If the baseline shows the same per-phrasing range asymmetry under "FR vs IT as a bystander," then the LoRA isn't the cause; analyzer treats this as a confound finding and discusses in the body.
• Dataset asymmetry between lang_inv_fr_it_5k vs lang_inv_it_fr_5k: dataset-symmetry summary artifact. If lengths / code-switch rates differ materially, the body flags it.
• LoRA-strength / fit-quality confound: per-adapter final train loss + KL-from-base on a neutral probe. If one direction's adapters have systematically higher KL or lower train loss, the analyzer discusses whether per-phrasing variance scales with adapter strength.
• Phrasing-template arbitrariness: the 5 phrasings are the 5 training templates, not an arbitrary author-chosen set. This eliminates the "p-hacking on phrasing choice" objection raised in the methodology critique.
• Compute / sampling drift: seed-42 reproduction sanity on the 2 overlapping phrasings (±3pp vs #190's 38.8%).

Critique loop notes: 1 critique loop run with paired Claude + Codex critics for 3 lenses (methodology, statistics, alternative-explanations); Codex companion failed to spawn for all 3 lenses (codex app-server exited unexpectedly at exit 1) — fell back to Claude critics for those lenses per protocol. Consistency-checker run once on the revised plan and returned WARN with 2 findings: (i) phrasing-OOD claim factually wrong (folded — switched eval phrasings to the actual 5 training templates), (ii) train_on_responses_only=true not explicitly listed (folded — now explicit). Blockers folded: missing baseline arm (added un-LoRA'd Qwen2.5-7B on same grid). Important + scope-preserving items folded: Spanish-decisive justification, top-5 demoted to descriptive, range pre-registration, bootstrap CI on the difference, permutation null on range ratio, dataset-symmetry artifact, KL-from-base diagnostic, decoding-seed logged. Follow-ups NOT folded into this run: clustered SE on rule (1) (cluster-bootstrap can be done at analysis time from the same artifacts; not blocking), Monte Carlo power simulation for rule (2) (the permutation null subsumes this for decision purposes), name-mention-bias analysis on phrasings (the 5 training templates don't contain language names, so this is moot), DE↔FR multi-seed replication (out-of-scope follow-up — separate experiment). Final merged verdict: all three lenses pass after revision.

Likely Clean Result

Three forks, each rendering as a clean-result HTML on /e/experiment/333 (per docs/clean-result-guidelines.md):

• Fork A — (1) holds + (2) holds (predicted most likely): Title: "FR↔IT bystander spill is pooled-symmetric but the underlying phrasing-sensitivity isn't." TL;DR: pooled rate matches at multi-seed (3 seeds × 5 phrasings × 40 completions = 600 per cell); per-phrasing range is ~3× larger under FR→IT than under IT→FR. Hero plot: grouped bar chart of pooled Spanish-bystander contamination per direction (3-seed mean ± bootstrap-95% CI) with overlaid per-phrasing scatter showing the FR→IT spread; baseline (un-LoRA'd Qwen2.5-7B) plotted as a third group to show the LoRA adds the asymmetry. Action: edit #239 to the narrower framing.
• Fork B — (1) holds + (2) fails: Title: "FR↔IT bystander spill is direction-symmetric, including at the per-phrasing level." TL;DR: pooled AND per-phrasing distributions match across reverse pair when measured on all 5 training paraphrases. The original 47.5pp vs 12.5pp asymmetry was an artifact of the 2-phrasing subset chosen in #190's eval. Action: restore #239's original symmetric-spill paragraph with multi-seed + 5-phrasing evidence.
• Fork C — (1) fails: Title: "FR↔IT bystander spill is direction-asymmetric at multi-seed; the single-seed pooled match was a low-N artifact." TL;DR: 3-seed pooled rates diverge by >5pp. Remove symmetric-spill claim from #239 entirely.

Result-analyzer picks the correct fork from the rule outcomes and the baseline / KL / dataset-symmetry sanity checks.

Approval Checklist

• Goal — narrow, falsifiable: test whether {FR,IT} unordered-pair determines bystander-spill rate, pooled and per-phrasing, on all 5 training paraphrases.
• Hypothesis — three parts (pooled-symmetric, per-phrasing-asymmetric, top-5-identical-given-baseline).
• Prediction — specific point estimates and ≥2× range-ratio threshold.
• Kill criterion — all 3 outcome quadrants mapped (incl. the (1)+(2)-fails case that vindicates #239); Spanish-bystander decisive; range pre-registered as per-seed range averaged across seeds; bootstrap CI on the difference reported alongside.
• Compute and Hardware — 1 H100 80GB SXM pod, gpuCount=1, ~24 GPU-hr budget; ~$65 USD total ($64.56 compute + $0.67 storage at $2.69/GPU-hr H100 SXM rate; rate may have drifted, audit before launch). Single-pod chosen to avoid the multi-pod partial-dispatch anti-pattern.
• Artifacts — 4 new HF Hub adapters, 7 eval summaries on HF dataset repo (6 LoRA + 1 baseline), dataset-symmetry artifact, 4 KL-from-base diagnostics, WandB run records.
• Verification — 7 summaries × 1400 rows; seed-42 reproduction sanity on the 2 overlapping phrasings vs #190; train_on_responses_only=true confirmed; dataset-symmetry written; KL-from-base summaries present.
• Risks / Red Team — phrasing-co-vary-with-seeds caveat; permutation null on the range ratio characterizes noise floor; baseline arm + KL probe + dataset-symmetry artifact address main alt-explanations; langdetect short-completion errors bounded by N=40 per cell.
• Likely Clean Result — three forks pre-mapped to #239 body actions.
• runpod-spec matches plan — single H100 SXM pod, gpuCount=1, ~24 h budget, sequential train+eval+baseline orchestration via a single launcher script.

{
  "name": "lang-inv-symmetry-333",
  "gpuType": "H100",
  "gpuCount": 1,
  "volumeGb": 200,
  "containerDiskGb": 200,
  "cloudType": "SECURE",
  "estimatedMinutes": 1440,
  "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && bash scripts/bootstrap_pod.sh && uv run python scripts/run_issue333_train_eval.py'",
  "config": {
    "command": "Single-pod orchestration script for #333: (1) run dataset-symmetry summary on lang_inv_{fr_it,it_fr}_5k.jsonl → eval_results/issue333/dataset_symmetry.json; (2) edit scripts/eval_language_inversion.py PROMPT_TEMPLATES (line 82-85) to the 5 training-paraphrase templates from build_language_inversion_data_v2.py:DIRECTIVE_TEMPLATES (Speak in {lang}. / Please respond in {lang}. / Reply using {lang} only. / Answer in {lang}. / Respond entirely in {lang}, please.); (3) sequentially train 4 LoRA conditions (c_lang_inv_fr_it seeds 137, 256; c_lang_inv_it_fr seeds 137, 256) with hyperparameters byte-identical to #190 (LoRA r=32 α=64 dropout=0 rslora on 7 projs; lr=5e-6 1ep bf16 max_seq=2048 eff_bs=16 AdamW-fused warmup_ratio=0.03 train_on_responses_only=true); upload each merged adapter to HF Hub under c_lang_inv_{cond}_seed{seed}_post_em; after each train, run KL-from-base probe on 100 neutral English prompts (top-50 logits, first 16 tokens) → eval_results/issue333/kl_from_base_{cond}_seed{seed}.json; (4) download 2 seed-42 adapters from HF Hub and run scripts/eval_language_inversion.py over all 6 adapters PLUS un-LoRA'd Qwen/Qwen2.5-7B-Instruct baseline on the 5-phrasing × 7-directive-lang × 40-completion grid (vLLM T=1.0, max_tokens=256, decoding_seed=0, langdetect labels, no Claude judge); (5) upload all eval summaries to superkaiba1/explore-persona-space-data:eval_results/issue333/. POST progress to SAGAN_PROGRESS_URL after each training run, each KL probe, and each model eval.",
    "artifacts": [
      "hf://superkaiba1/explore-persona-space/c_lang_inv_fr_it_seed137_post_em/",
      "hf://superkaiba1/explore-persona-space/c_lang_inv_fr_it_seed256_post_em/",
      "hf://superkaiba1/explore-persona-space/c_lang_inv_it_fr_seed137_post_em/",
      "hf://superkaiba1/explore-persona-space/c_lang_inv_it_fr_seed256_post_em/",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/dataset_symmetry.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/summary_5phrasings_c_lang_inv_fr_it_seed42.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/summary_5phrasings_c_lang_inv_fr_it_seed137.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/summary_5phrasings_c_lang_inv_fr_it_seed256.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/summary_5phrasings_c_lang_inv_it_fr_seed42.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/summary_5phrasings_c_lang_inv_it_fr_seed137.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/summary_5phrasings_c_lang_inv_it_fr_seed256.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/summary_5phrasings_baseline_qwen25_7b.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/kl_from_base_fr_it_seed137.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/kl_from_base_fr_it_seed256.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/kl_from_base_it_fr_seed137.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/kl_from_base_it_fr_seed256.json",
      "hf-dataset://superkaiba1/explore-persona-space-data/eval_results/issue333/comparison_5phrasings.json"
    ]
  }
}

I read the scoped record (#333) carefully. The hypothesis, expected information gain, methodology, and most decision rules are unusually well-specified. A few targeted ambiguities remain that are load-bearing for the kill criterion — I'd like to lock them down before drafting the full plan rather than guessing in the plan and risking a different reading at interpretation time.

Clarifying questions

1. Decision-rule completeness — the (1)-holds-and-(2)-fails case is unmapped.

The body specifies actions for two outcomes:

• (1) holds + (2) holds → update #239 to the narrower "pooled-symmetric but phrasing-sensitivity is itself asymmetric" framing.
• (1) fails → remove the symmetric-spill claim entirely.

What about the third case: (1) holds + (2) fails (pooled rates match within ±5pp and phrasing-spread is roughly comparable across the two directions)? That is the outcome that would actually vindicate #239's original "direction-agnostic geometry" framing. Should that outcome restore the original symmetric-spill paragraph in #239, restore it with the multi-seed evidence added, or something else? Without an action mapping for this quadrant, ~25% of the outcome space is undefined.

2. Compound test for decision rule (1) — Spanish-only, German-only, or both?

Rule (1) says the FR→IT vs IT→FR Spanish-bystander rates must match within ±5pp, then adds "Same threshold for German-bystander cell." It does not say whether the symmetric-spill claim requires both cells to pass within ±5pp, either to pass, or whether Spanish is the headline and German is reported-but-not-decisive. Given the single-seed German-bystander gap was already 11pp (FR→IT: 36% / IT→FR: 25% — outside ±5pp), this matters: a strict "both must pass" rule will likely kill (1) on German alone before Spanish gets a vote.

My read: the Spanish-bystander cell is the headline (it's the #239 quote) and German is supportive. Is that the intent, or do you want a stricter compound rule?

3. Definition of "phrasing-spread" in decision rule (2).

"Phrasing-spread exceeds … by ≥2× across the 5 phrasings (averaged over seeds)" — is "spread" the range (max − min across the 5 phrasings), the standard deviation across the 5 phrasings, or the IQR? At the single-seed #239 numbers, range gives 47.5pp vs 12.5pp (≈3.8× — clears 2×), but std-dev or IQR could land differently. The 2× ratio is interpretable for any of these, but the threshold's noise floor depends on the choice, so I'd like to pre-register one.

---

I have no other planning blockers — model, LoRA hyperparameters, dataset, eval setup, seeds, compute budget, and the kill criterion structure are all specified. Once these three points are answered (or you say "use your judgment, default to X"), I'll proceed straight to the full plan in the next turn.