Run 339be03b

I have enough context. The previous direct-dispatch failed because all 4 RunPod pods successfully reached RUNNING on the team account, then simultaneously disappeared from inventory ~11 min later — the exact recurring failure mode the prior planner flagged as Risk 2. The persisted substitution_policy already allows personal as a fallback, but prefer="team" keeps re-dispatching into the failing surface. The minimal scope-preserving fix is to flip account.prefer to personal for this recovery dispatch and re-emit the otherwise unchanged owner-approved plan.

Goal

Run one balanced 2^5 factorial across five contested factors that prior single-axis experiments could not disentangle, stratified over three source personas (librarian, surgeon, programmer), and rank which factors actually move [ZLT] source-rate and off-diagonal leakage under LoRA SFT on Qwen2.5-7B-Instruct. Replaces ad-hoc one-axis sweeps with a single interpretable screen that absorbs the open scope of #361 / #339 / #353 and the still-unrun grid in #46.

Hypothesis

Conditional on the design in the experiment body (96 cells = 3 sources × 32 cells, with seeds 137+256 on the top-3 cells per source), the five factors split into three classes:

1. Load-bearing (large main effects): Factor A (system-prompt length, short→long) and Factor E (loss mask, marker-only→whole-completion) dominate source-rate. A1 raises source-rate; E1 lowers it. Factor D (off-policy data) lowers source-rate vs on-policy at matched length.
2. Non-monotonic / collapsing: Factor B (answer-format length, short→long) collapses source-rate at the long extreme even when system-prompt length is short, replicating #295's null at the long tail.
3. Near-zero net effect: Factor C (persona vs lexically matched non-persona framing) has near-zero independent main effect on source-rate once A is controlled, but a measurable interaction with A (A×C) carrying most of #337's "persona" signal.

For off-diagonal leakage, A1 and E1 both reduce leakage; D1 has the opposite sign to its effect on source-rate.

Prediction

At α=0.05 with question-clustered bootstrap for source-rate and persona-clustered for leakage (per --bootstrap-cluster-sr question --bootstrap-cluster-lr persona):

• |d(A on SR)| ≥ 0.4 averaged across the three source slabs, sign positive.
• |d(E on SR)| ≥ 0.3, sign positive for E0 (marker-only) over E1 (whole-completion).
• |d(D on SR)| ≥ 0.2, sign positive for D0 (on-policy) over D1 (off-policy).
• |d(B on SR)| ≥ 0.3 at the B1 extreme with a non-monotone shape vs B0.
• |d(C on SR)| < 0.15 once A is partialled out.
• Pre-registered F1×F2 interaction: A×B yields a measurable interaction term (>2× the next-largest interaction) because the question-prefix tokens for B compete with persona-conditioning context from A.

Signs and magnitudes are read off pod 3's main_effects.json / interactions.json after Phase 4 aggregation.

Kill Criterion

Pull the plug — and treat the screen as uninterpretable — if any of the following fires:

1. Phase 0 pre-screen fails: base-model contamination on the 24×20×5 eval panel exceeds the pre-registered threshold (kill_criterion_4_passed=False in pod 0). Pod 0 raises SystemExit and the run aborts.
2. Phase 1 smoke fails: the 8-cell resolution-III fractional factorial on librarian returns verdict ≠ pass (e.g. uniform near-zero or saturated source-rates across all 8 cells). Indicates the training recipe or marker setup is broken at the level of the librarian source, so the full 32-cell sweep cannot rank factors.
3. Sign instability across sources: after Phase 4 aggregation, ≥3 of the 5 main effects flip sign across the three sources. The 5-factor framing is wrong and we re-scope at the persona-class level before any further dispatch.
4. All main effects below noise: all 5 main effects have |Cohen's d| < 0.15 for source-rate AND for leakage across all three sources. The screen has no statistical power at this dataset size; we revisit pos/neg counts or LoRA rank before re-running.
5. Cost overrun: any single pod exceeds 24 h wall-time (the phase4-max-wait-seconds hard cap), or aggregate spend exceeds ~$160 (≈ 1.5× the planned compute estimate below). Operator stops the run and triages.
6. Recovery-specific bootstrap silence: if all four pods reach RUNNING but emit zero 5% · bootstrap complete progress notes within 5 min of RUNNING, the operator stops the dispatch (suspected bootstrap-wrapper hang or account-credential flake — see Risks §Recovery context). This is the explicit detector for the failure mode that triggered this auto-recovery.

Experimental Setup

Faithful to the body's design and the persisted pod_spec on experiments.077ae4c7-…, instantiated by the existing eps.experiments.marker_factor_screen entry point.

• Branch / commit: experiment-365 @ b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7 (matches the patched pod_spec and the _factor_screen/ module tree currently on disk under /home/thomasjiralerspong/explore-persona-space/eps/experiments/). The runner injects SAGAN_EPS_BRANCH=experiment-365 and SAGAN_EPS_COMMIT_SHA=b1a24b4b... so the bootstrap wrapper checks out the exact commit on each pod.
• Model and tooling: Qwen2.5-7B-Instruct, LoRA r=32 α=64, lr=1e-5, 3 epochs. 200 positives + 400 negatives per source. 24-persona × 20-question × 5-completion eval panel with eval_max_new_tokens=2048. Clustered bootstrap, question-clustered for source-rate, persona-clustered for leakage.
• Pod topology (4 pods, role-distinct):
  • Pod 0 (librarian): Phase 0 base-model contamination pre-screen → Phase 1 res-III smoke → Phase 2 full 32-cell librarian slab at seed 42 → Phase 3 multi-seed top-3 librarian cells (seeds 137, 256). Runs --run-pre-screen --run-smoke.
  • Pod 1 (surgeon): Phase 2 32-cell surgeon slab at seed 42 → Phase 3 multi-seed top-3 surgeon cells. No pre-screen/smoke flags (those gate the whole experiment from pod 0).
  • Pod 2 (programmer): same shape as pod 1 for the programmer slab.
  • Pod 3 (aggregator-and-overflow): waits up to 24 h for the three slab metrics.json files, then builds main_effects.json, interactions.json, the persona × cell heatmap, and the clean-result HTML. Runs --role aggregator-and-overflow --build-figures --write-clean-result --label-f1xf2-preregistered.
• Pre-registration: the A×B (system-prompt length × answer-format length) interaction is the one pre-registered two-way; the other nine are reported but labeled exploratory in the aggregator output.

Compute and Hardware

Topology. Four pods, each 1× A100 80GB. This is a deliberate, multi-pod array under clause (c) — explicit per-pod role isolation — not the discouraged "one-pod-per-source" anti-pattern:

• Pods 0/1/2 are the three source slabs and could in principle batch-share one node, but the script's --pod-index / --source-persona / --num-pods contract (marker_factor_screen.py:69-79, 153-265) and pod 0's extra Phase 0/1 gates make them heterogeneous workloads with independent kill criteria.
• Pod 3 is a long-running stateful aggregator that polls sibling pods' metrics.json until they exist; it must outlive any single training process and runs different code (figure-building, no SFT). It would be wasteful to keep a 4-GPU node alive for the ~24 h aggregator wait.

Consolidating to a single 4-GPU node would require an accelerate-style rewrite of the entry point that does not exist on experiment-365. That rewrite is out of scope for this recovery dispatch and is the right shape for a future ablation.

Time and cost.

• Compute: 4 pods × 1 GPU × 18 h × $1.49/GPU-hr (A100 80GB SXM, RunPod Secure Cloud, May 2026 reference rate per the system-prompt rate table) ≈ $107.
• Storage: 4 × (100 GB volume + 100 GB container disk) × 18 h at $0.10/GB-month ≈ $2.
• Total estimated spend ≈ $110 (rounded to two significant figures), with a $160 cost-overrun kill at 1.5×.

Rates may drift; the auditor's input is the rate stated here.

Substitution policy delta vs prior dispatch. The persisted pod_spec.substitution_policy.account.prefer was "team". For this recovery dispatch, prefer is flipped to "personal" because both prior team-account dispatches (15038ff7… and the 910a65d6… whose failure triggered this recovery) hit Pod not found in account=team at ~10–15 min after reaching RUNNING. The personal account remains in allowed so future revisions can swap back once the team-account flake is resolved. The script binds one GPU per pod, so gpuCount stays pinned to 1. A100/H100/H200 ≥80 GB are all acceptable; Secure preferred over Community.

Artifacts

• Per-source pod (0/1/2) under /workspace/runs/365/pod<i>/<source>/: metrics.json, LoRA adapters under adapters/, figures/ (per-source heatmap, A×B interaction plot, factor-ranking bar chart).
• Pod 0 extras under /workspace/runs/365/pod0/: pre_screen.json, smoke.json.
• Pod 3 aggregator under /workspace/runs/365/pod3/: main_effects.json, interactions.json, aggregate_metrics.json, figures/persona_cell_heatmap.svg, figures/factor_ranking.svg, figures/AxB_interaction.svg, clean_result.html (the experiment-page body).
• Sagan-side: new experiments.body HTML written by --write-clean-result and the aggregator figures uploaded as html_artifact / image figures linked to experiment #365.

Verification

Pre-flight (operator does this before re-dispatch and at first progress-tick):

1. Spec parity check. Diff this plan's runpod-spec against the persisted experiments.077ae4c7…pod_spec: identical except for substitution_policy.account.prefer ("team" → "personal"). No other field changes.
2. Bootstrap progress check. Within 5 min of each pod reaching RUNNING, the runner expects a 5% · bootstrap complete progress event from each of the four pods. If zero pods emit it (the recovery-trigger failure mode), abort and SSH into one pod to tail journalctl and the bootstrap wrapper log before retrying.
3. Phase 0/1 gates. Pod 0 must emit kill_pre_screen verdict=passed and kill_smoke verdict=pass before pods 1/2/3 are allowed to consume significant compute. The script raises SystemExit on failure; the runner surfaces that as a hard stop.

Post-run (operator checks before declaring the screen interpretable):

4. main_effects.json reports five main effects with bootstrap CIs and Cohen's d for both source-rate and leakage. All three sources represented in the underlying slab summaries.
5. interactions.json reports all ten two-way interactions with the A×B row carrying a pre_registered: true flag.
6. clean_result.html follows docs/clean-result-guidelines.md (TL;DR → primary plot → Experimental design dropdown) and is attached to experiments.body on experiment #365.

Risks and Red Team

Risk 1 — multi-pod partial-dispatch sensitivity. Per the system prompt, partial dispatch of a multi-pod array is treated as a hard failure; this is what happened at 08:56 (2/4 came up). The substitution policy below allows the provisioner to fall back to community cloud or back to the team account; if capacity is still tight, the operator should consider running pod 0 alone as a 1-pod smoke before fanning out.

Risk 2 — recurring team-account vanishing (the recovery trigger). Two prior team-account dispatches (15038ff7… and 910a65d6…) hit Pod not found in account=team 10–15 min after all four pods reported RUNNING. Root cause is still ambiguous (team-account credentials event vs. bootstrap-wrapper hang vs. RunPod team-account inventory drift). This recovery dispatch flips account.prefer to personal so we exercise a different surface. If the same vanishing fires under personal, the failure is upstream of the account choice and the operator should SSH into one pod to tail journalctl / wrapper logs before any further retry, per the prior planner's recommendation. Verification step 2 is the explicit detector.

Risk 3 — single-seed factor ranks. Most cells run at seed 42 only; multi-seed coverage is restricted to the top-3 per source. A factor ranked 2nd or 3rd whose true effect is borderline could swap with a 4th-ranked factor under seed noise. We accept this risk because broadening multi-seed to all 32×3 cells multiplies compute by 3× and is out of cost scope.

Risk 4 — F1×F2 pre-registration vs the other 9 interactions. With 10 two-way interactions reported and a single one pre-registered, multiple-comparison hygiene matters. The aggregator labels exploratory interactions accordingly; we do not claim significance for any exploratory interaction without a confirmatory experiment.

Risk 5 — A/B confound on user-message tokens. Even though B is induced by a natural format instruction, the instruction tokens still live inside the user message and could shift attention patterns independent of completion length. This is a genuine confound the screen cannot fully separate; the A×B interaction term is the diagnostic and is pre-registered for that reason.

Critique loop notes. This is a re-finalization of an owner-approved design (the planJson is fully populated and the design body has been stable across multiple rounds). The scientific design is unchanged from the prior approved plan; the only delta is runpod-spec.substitution_policy.account.prefer flipping from "team" to "personal" plus the corresponding addition of kill criterion #6 (bootstrap-silence detector) and Verification step 2. I ran the consistency check internally rather than the full paired-critic loop because (a) the design is owner-approved, (b) the open question is purely whether the runpod-spec matches the patched DB state and the script's CLI contract, and (c) the script's parse_args (marker_factor_screen.py:62-138) was re-verified against each pod's dockerArgs. Loops run: 0 (re-finalize); merged verdict: pass for methodology, statistics, and alternative-explanations because no design field changed; no follow-ups intentionally dropped. If a future revision changes any scientific field, that revision should re-enter the full paired-critic loop.

Likely Clean Result

A body HTML on experiment #365 following docs/clean-result-guidelines.md:

• TL;DR (single paragraph in first person): "I ran a balanced 2^5 factorial across five marker-implantation factors on three source personas. System-prompt length and loss-mask scope were the two load-bearing knobs, both with Cohen's d > 0.4 on source-rate; persona framing had near-zero independent effect once length was controlled, confirming #340. Long answer-format prompts collapsed source-rate at the extreme, replicating #295. Off-policy data lost ~0.2 d of source-rate vs on-policy at matched length."
• Primary plot: a single bar chart of the five main-effect Cohen's d's on source-rate with 95% bootstrap CIs, ordered largest-to-smallest, with plain-English axis labels ("system-prompt length", "loss masks marker only", etc.) and SVG <title> hover tooltips carrying the underlying mean and CI.
• Experimental design dropdown: the 32-cell factor table, the 3-source stratification, the seed plan, the bootstrap scheme, and the kill-criterion list — collapsed by default.

Sections deliberately omitted per the clean-result guidelines: separate Background / Methodology h2s, standing caveats, references to the abandoned single-axis sweeps, and the additional per-source heatmaps (those live as linked figures, not in the body).

Approval Checklist

• Goal matches the experiment record title and absorbs the open scope of #361 / #339 / #353 / #46.
• Hypothesis is specific: five factors split into three classes with signed predictions for each.
• Prediction is falsifiable: numeric Cohen's d thresholds with α=0.05 and a named bootstrap scheme.
• Kill criterion has six concrete triggers, including the new bootstrap-silence detector that directly responds to the recovery-trigger failure mode.
• Compute and Hardware estimates 4 × 1 × 18 h × $1.49 ≈ $107 compute + ~$2 storage = ~$110 total at A100 80GB SXM Secure Cloud rates; cost-overrun kill at $160. Multi-pod array is justified under clause (c) per-pod role isolation; consolidation deferred to a future ablation.
• Artifacts enumerated per-pod and aggregator with /workspace/runs/365/... paths matching the script's directory contract.
• Verification covers spec parity, the bootstrap-progress detector, Phase 0/1 kill gates, and the aggregator's required JSONs / clean-result HTML.
• Risks explicitly cover the team-account vanishing failure that triggered this recovery and document the rationale for flipping account.prefer to personal.
• Likely clean result follows docs/clean-result-guidelines.md (TL;DR → primary plot → design dropdown, first-person voice, no standing caveats).
• runpod-spec matches the plan: 4 pods, A100 80GB Secure Cloud preferred, single-GPU per pod, account.prefer=personal with team retained in allowed, and per-pod dockerArgs identical to the persisted experiments.077ae4c7…pod_spec save for that one substitution-policy field.

[
  {
    "name": "marker-screen-365-pod0-pre-and-source-librarian",
    "gpuType": "A100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "env": {
      "SAGAN_EPS_BRANCH": "experiment-365",
      "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7"
    },
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 0 --num-pods 4 --source-persona librarian --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --run-pre-screen --run-smoke --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Pod 0: Phase 0 base-model contamination pre-screen on 24x20x5 eval panel + Phase 1 8-cell res-III librarian smoke (kill gates only) + Phase 2 32-cell librarian slab at primary seed + Phase 3 multi-seed top-3 librarian cells (seeds 137, 256). Clustered bootstrap (question-clustered SR, persona-clustered LR).",
      "artifacts": [
        "/workspace/runs/365/pod0/pre_screen.json",
        "/workspace/runs/365/pod0/smoke.json",
        "/workspace/runs/365/pod0/librarian/metrics.json",
        "/workspace/runs/365/pod0/librarian/adapters/",
        "/workspace/runs/365/pod0/figures/"
      ]
    },
    "substitution_policy": {
      "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 },
      "gpuCount": { "min": 1, "max": 1 },
      "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
      "dataCenterId": { "allowed": "any" },
      "account": { "allowed": ["personal", "team"], "prefer": "personal" },
      "volumeGb": { "min": 100 },
      "containerDiskGb": { "min": 100 }
    },
    "consolidation": {
      "may_merge_pods": false,
      "merge_target_max_gpus_per_pod": 1
    }
  },
  {
    "name": "marker-screen-365-pod1-source-surgeon",
    "gpuType": "A100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "env": {
      "SAGAN_EPS_BRANCH": "experiment-365",
      "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7"
    },
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 1 --num-pods 4 --source-persona surgeon --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Pod 1: Phase 2 32-cell surgeon slab at primary seed 42 + Phase 3 multi-seed top-3 surgeon cells (seeds 137, 256). Clustered bootstrap (question-clustered SR, persona-clustered LR).",
      "artifacts": [
        "/workspace/runs/365/pod1/surgeon/metrics.json",
        "/workspace/runs/365/pod1/surgeon/adapters/",
        "/workspace/runs/365/pod1/figures/"
      ]
    },
    "substitution_policy": {
      "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 },
      "gpuCount": { "min": 1, "max": 1 },
      "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
      "dataCenterId": { "allowed": "any" },
      "account": { "allowed": ["personal", "team"], "prefer": "personal" },
      "volumeGb": { "min": 100 },
      "containerDiskGb": { "min": 100 }
    },
    "consolidation": {
      "may_merge_pods": false,
      "merge_target_max_gpus_per_pod": 1
    }
  },
  {
    "name": "marker-screen-365-pod2-source-programmer",
    "gpuType": "A100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "env": {
      "SAGAN_EPS_BRANCH": "experiment-365",
      "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7"
    },
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 2 --num-pods 4 --source-persona programmer --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Pod 2: Phase 2 32-cell programmer slab at primary seed 42 + Phase 3 multi-seed top-3 programmer cells (seeds 137, 256). Clustered bootstrap (question-clustered SR, persona-clustered LR).",
      "artifacts": [
        "/workspace/runs/365/pod2/programmer/metrics.json",
        "/workspace/runs/365/pod2/programmer/adapters/",
        "/workspace/runs/365/pod2/figures/"
      ]
    },
    "substitution_policy": {
      "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 },
      "gpuCount": { "min": 1, "max": 1 },
      "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
      "dataCenterId": { "allowed": "any" },
      "account": { "allowed": ["personal", "team"], "prefer": "personal" },
      "volumeGb": { "min": 100 },
      "containerDiskGb": { "min": 100 }
    },
    "consolidation": {
      "may_merge_pods": false,
      "merge_target_max_gpus_per_pod": 1
    }
  },
  {
    "name": "marker-screen-365-pod3-aggregator-and-overflow",
    "gpuType": "A100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "env": {
      "SAGAN_EPS_BRANCH": "experiment-365",
      "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7"
    },
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 3 --num-pods 4 --role aggregator-and-overflow --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --build-figures --write-clean-result --label-f1xf2-preregistered --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Pod 3: Phase 4 aggregator. Waits up to 24h for pods 0/1/2 metrics.json files, then builds main_effects.json, interactions.json (with A x B pre-registered), the persona x cell heatmap, the factor-ranking bar chart, the A x B interaction plot, and the clean-result HTML attached to experiments.body.",
      "artifacts": [
        "/workspace/runs/365/pod3/main_effects.json",
        "/workspace/runs/365/pod3/interactions.json",
        "/workspace/runs/365/pod3/aggregate_metrics.json",
        "/workspace/runs/365/pod3/figures/persona_cell_heatmap.svg",
        "/workspace/runs/365/pod3/figures/factor_ranking.svg",
        "/workspace/runs/365/pod3/figures/AxB_interaction.svg",
        "/workspace/runs/365/pod3/clean_result.html"
      ]
    },
    "substitution_policy": {
      "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 },
      "gpuCount": { "min": 1, "max": 1 },
      "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
      "dataCenterId": { "allowed": "any" },
      "account": { "allowed": ["personal", "team"], "prefer": "personal" },
      "volumeGb": { "min": 100 },
      "containerDiskGb": { "min": 100 }
    },
    "consolidation": {
      "may_merge_pods": false,
      "merge_target_max_gpus_per_pod": 1
    }
  }
]

Recovery summary. Diagnosed the failure as the recurring team-account post-RUNNING vanishing (Risk 2 from the prior plan). Re-emitted the owner-approved 4-pod plan unchanged except for: (1) substitution_policy.account.prefer flipped from team to personal; (2) kill criterion #6 added as an explicit detector for the same failure mode under the new account; (3) Verification step 2 documents the 5-min bootstrap-progress check. The persisted experiments.077ae4c7…pod_spec already has personal in account.allowed, so this is a scope-preserving recovery, not a design change.