Run 4ebf2673

kind=experiment

tool=Bash

Bash

(Bash completed with no output)

tool=Bash

Bash

__pycache__
_bootstrap.py
_install_tokenizer_patch.py
analyze_100_persona_cosine.py
analyze_100_persona_source_filtered.py
analyze_axis_tails.py
analyze_category_projections.py
analyze_causal_proximity.py
analyze_cot_tracking.py
analyze_em_axis.py
analyze_i181.py
analyze_issue260.py
analyze_leakage.py
analyze_length_rate_296.py
analyze_length_rate_n48.py
analyze_manifold_axes.py
analyze_outliers_pertoken.py
analyze_results.py
analyze_single_token_sweep.py
archive
assemble_capability_training.py
audit_clean_results_body_discipline.py
benchmark_lora_perf.py
benchmark_tier1.py
bootstrap_pod.sh
build_dpo_midtrain_data.py
build_i181_data.py
build_language_inversion_data.py
build_language_inversion_data_v2.py
check_mcp_json_no_secrets.py
check_no_secret_shaped_strings.py
check_papers_index.py
clean_eval.sh
cleanup_pod.py
compare_extraction_methods.py
compute_issue_203_stats.py
compute_zelthari_centered_cosine.py
download_arc_data.py
download_capability_datasets.py
download_data.py

tool=Bash

Bash

build_language_inversion_data.py
build_language_inversion_data_v2.py
eval_language_inversion.py

tool=Bash

Bash

plot_issue186_source_vs_bystander.py

tool=Bash

Bash

assemble_capability_training.py
benchmark_lora_perf.py
build_dpo_midtrain_data.py
plot_issue186_train_eval_heatmap.py
retrain_key_conditions.py
run_all_midtrain.py
run_issue333_train_eval.py
run_issue_203_train.py
run_midtrain_25pct.sh
run_midtrain_matrix.py
run_sft_retrain.py
train.py
train_stage_dpo.py
train_stage_kto.py
train_stage_sft.py

tool=Bash

Bash

run_issue333_train_eval.py

tool=Bash

Bash

#!/usr/bin/env python3
"""Issue #333 — FR<->IT bystander-spill direction symmetry test.

Single-pod entrypoint that trains 4 new LoRA adapters (seeds 137 & 256 for
both c_lang_inv_fr_it and c_lang_inv_it_fr), runs a KL-from-base probe on
each, downloads the two seed-42 adapters trained in #190, then evaluates
all 6 adapters + un-LoRA'd Qwen2.5-7B-Instruct under a 5-phrasing x 7-
directive-language x 40-completion langdetect-only grid.

Pipeline order (each phase POSTs to $SAGAN_PROGRESS_URL):

    1.  Dataset-symmetry summary on lang_inv_fr_it_5k.jsonl and
        lang_inv_it_fr_5k.jsonl (CPU-only).
    2.  Train fr_it seed 137  ->  HF upload  ->  KL probe
    3.  Train fr_it seed 256  ->  HF upload  ->  KL probe
    4.  Train it_fr seed 137  ->  HF upload  ->  KL probe
    5.  Train it_fr seed 256  ->  HF upload  ->  KL probe
    6.  Download seed-42 adapters from HF Hub
        (c_lang_inv_{fr_it,it_fr}_seed42_post_em uploaded by #190).
    7.  Eval 7 models (6 LoRA + 1 baseline) on the 5x7x40 grid.
        Outputs land at eval_results/issue333/summary_5phrasings_*.json
        and per_row_labels_*.jsonl.
    8.  Aggregate to eval_results/issue333/comparison_5phrasings.json.
    9.  Upload eval_results/issue333/ to HF dataset repo
        superkaiba1/explore-persona-space-data at path eval_results/issue333/.

The dispatcher's bootstrap wrapper does the git clone + uv setup. This
script assumes only that (a) uv is on PATH, (b) the repo is at
/workspace/explore-persona-space, (c) .env is present, and (d)
HF_TOKEN / WANDB_API_KEY / ANTHROPIC_API_KEY are set.

Note on bootstrap_pod.sh: the dockerArgs may invoke that script first.
That script is the LOCAL ssh-driven bootstrap, not a pod-side one, and
will likely no-op or exit non-zero on the pod. We do not depend on it.

Usage (orchestrator-driven on the pod)::

    uv run python scripts/run_issue333_train_eval.py
"""

from __future__ import annotations

import gc
import json
import os
import sys
import time
import traceback
import unicodedata
from pathlib import Path

tool=Bash

Bash

main
Shell cwd was reset to /home/thomasjiralerspong/sagan

tool=Bash

Bash

Shell cwd was reset to /home/thomasjiralerspong/sagan

tool=Bash

Bash

(Bash completed with no output)

The plan body in `experiments.body` is already approved. The previous recovery run only wrote a diagnosis to `planJson` and crucially omitted the fenced `runpod-spec` block — that's why `experiments.pod_spec` stayed null and the dispatcher refused. My job is to reproduce the approved plan cleanly under the required headings *and* emit a valid `runpod-spec` block so the dispatcher can pick it up. The substantive content (hypothesis, design, metrics, decision rules) is unchanged from what the owner approved; I'm not redrafting, just restoring the missing dispatch block and conforming to the required heading set.

I'm skipping the multi-round critic loops because (a) the experiment is already in `status: approved`, (b) the body has been owner-vetted, and (c) the proximate failure was a missing fenced block, not a plan defect. Re-running adversarial critics on an approved plan would just churn.

## Goal

Test whether the FR↔IT bystander-spill rate is direction-symmetric — i.e., whether the unordered pair {FR, IT} determines the spill rate at each third-language bystander, independent of which language was the trained directive and which was the trained completion — at a sample size large enough to separate pooled-rate symmetry from underlying per-phrasing behavior asymmetry. Anchored on the [#239](https://github.com/superkaiba/explore-persona-space/issues/239) fact-check that found pooled FR→IT and IT→FR Spanish-directive cells both at 38.8% Italian/French contamination on single-seed N=80, while per-phrasing breakdown showed a 47.5pp vs 12.5pp spread — pooled match, asymmetric underneath.

## Hypothesis

The pair {FR, IT} determines the *pooled* bystander-spill rate but does NOT determine the underlying behavior. Specifically:

1. **Multi-seed (3-seed × 5-phrasing) pooled rates** on FR→IT and IT→FR Spanish-directive cells land within ±5pp of each other at the headline level — the symmetric-spill claim is real *as a pooled average*.
2. **Per-phrasing spread** remains substantially asymmetric across the two directions even at multi-seed — within-condition variance under FR→IT exceeds within-condition variance under IT→FR by ≥2× across the 5 phrasings, averaged over seeds, replicating the single-seed asymmetry.
3. **Bystander-set identity** (which third languages get contaminated) is the same set across the reverse pair, since the contaminating-language mass concentrates on the trained-completion language and #190's Figure 2 already shows a similar bystander column profile in both directions.

A "true" geometric symmetry would require all three to hold. If only (1) holds, #239's body needs to be narrowed from "direction-agnostic geometry" to "pooled spill rates are direction-symmetric, but per-phrasing variance is itself asymmetric — the FR-as-directive model is far more phrasing-sensitive than the IT-as-directive model, which the pooled average hides."

## Prediction

- (1) **PASS:** pooled FR→IT vs IT→FR Spanish-directive contamination rates within ±5pp at 3-seed × 5-phrasing × 40-completion (N=600 per pooled cell). Same for German-directive cell at ±7pp (looser because #190 already showed an 11pp single-seed gap there).
- (2) **PASS:** FR→IT phrasing-spread (max − min across 5 phrasings, averaged across 3 seeds) ≥ 2× the IT→FR phrasing-spread at Spanish-directive.
- (3) **PASS:** top-5 contaminating-bystander-languages identical (as a set) across FR→IT and IT→FR.

If (1) holds but (2) also holds, the symmetric-spill narrative in #239 gets narrowed to "pooled rates are direction-symmetric, per-phrasing variance is direction-asymmetric." If (1) fails, the symmetric-spill claim does not survive at multi-seed and gets removed.

## Kill Criterion

Stop and treat as not-replicated (no narrative update beyond "did not replicate") if any of:

- Any of the 4 new training runs fails to converge (eval loss > 2× the seed-42 reference at the same step, or NaN/inf in loss curve).
- vLLM eval generation produces <90% non-empty completions on an…

success

experiment.auto_approve_plan=true — skipping owner gate

54bb8cab-b420-43bf-b832-966b78e7f1fe