Run 3f9fb447

I cannot safely produce another auto-dispatch revision for this experiment. After diagnosing r5's transcript, this is now a cross-account infrastructure failure that needs human investigation, not another plan tweak.

Blocker — escalating to manual investigation

Diagnosis of r5 failure. Account-switch workaround did not break the loop:

• r1–r4 on account=team: pods dispatched, RUNNING for 2–25 min, then "not found in account=team" with no termination event. Reproduced simultaneously across sibling experiments #192, #333, #365 in the r4 batch — already established as account- or DC-level, not per-experiment.
• r5 on account=personal (this run, 32e93989-504f-4beb-a190-6585b45bbaa6): identical disappearance pattern.
  • Two pod_provisioner_attempt events fired 8s apart (10:53:33Z, 10:53:41Z), both committed: j0xvy1q82ryo8m, cv13x9s487tvwn. deploy_completed: pod-provisioner committed 2 pod(s).
  • Both went RUNNING at 10:53:55Z (≈22s and ≈13s after deploy_started, respectively).
  • Both runpod_retry: not found in account=personal at 10:54:54Z and 10:54:55Z — ~60s after going RUNNING.
  • Same disappearance reported again at 10:55:54Z, confirming the pods are gone from the personal-account API view.

Why this is not an experiment-plan problem. The r5 swap from team to personal was the only plan change vs. r4. The pattern reproduced unchanged on personal. No plan-side variable (model, dataset, branch tip, hyperparameters, GPU type, cloud type, volume, account) explains a pod that goes RUNNING on RunPod's side and then vanishes from the account's pod list ~60s later on both accounts. Five consecutive retries (r1–r5) is enough signal to stop retrying and escalate.

Two-pods-when-one-was-requested anomaly. The plan's runpod-spec is a single pod object with gpuCount: 1, but the provisioner committed 2 pods on r5. Possible causes worth checking before any r6: (a) provisioner retry on a transient RunPod response double-creating; (b) parallel re-dispatch from another orchestration path; (c) the consolidation block being misread. This may be unrelated to the disappearance, but it's the only behavior change between r4 (one pod) and r5 (two pods) and warrants inspection.

What I cannot fix from here. No substitution_policy field, GPU swap, region swap, or DC pin will change the outcome — both accounts are exhibiting the same post-RUNNING reclaim, and the dispatch path itself committed an unexpected pod count. The failure mode is upstream of the experiment spec.

Exact next manual actions

1. RunPod console audit (both accounts) around the r5 window (≈10:53:30Z–10:55:00Z UTC, 2026-05-14):
   • Check pod history for j0xvy1q82ryo8m and cv13x9s487tvwn on the personal account. Were they stopped, terminated, OOM-killed, billing-suspended, or did they vanish silently?
   • Cross-check whether RunPod logged an external stop, a billing/quota event, or a capacity reclaim.
   • Do the same for the r4 pods on team account (heqil88v6gnjyw for #333, rtna35w99srodo for #192, j18eiux6ccekyj for #365) around 10:39:59Z–10:44:00Z.
2. Runner-side audit:
   • Query agent_run_events for the r5 run 32e93989-504f-4beb-a190-6585b45bbaa6 and look for any pod_stop events, especially with a source field indicating they came from the dashboard, another agent run, a cron, or an internal cleanup job.
   • Inspect the pod-provisioner logic that committed 2 pods for a single-pod spec on r5 — confirm whether this was retry-induced double-creation or a separate parallel dispatch path.
   • Check whether runpod_pod_id columns in the runner's DB are getting set/cleared correctly across retries (a stale ID could explain "not found" if the API is being queried with the wrong ID, but RUNNING-then-vanish argues against pure-staleness).
3. Billing/quota check on both accounts. A spend cap or quota-suspend on team could be plausible alone; the same on personal within minutes is less plausible but worth confirming via the RunPod billing dashboard.
4. If the RunPod console shows pods were terminated externally (by another agent run, by a dashboard action, by RunPod-side reclaim): identify the source. If by an internal Sagan agent/cron, that is the bug. If by RunPod-side capacity reclaim on SECURE cloud, escalate to RunPod support with both account IDs and the pod IDs above.
5. Do not r6-dispatch this experiment until step 1–3 produce a root cause. Another retry on either account is expected to fail identically.

What I am NOT doing and why

• Not revising the plan. The approved plan body is unchanged from r3 onward and is not the failure source. Editing it now would be cargo-culting.
• Not switching back to team or trying a third account permutation. r1–r4 already exhausted team and r5 exhausted personal with the same symptom; the pattern is account-independent.
• Not changing GPU/cloud/DC. The pods did come up RUNNING on RunPod's side — capacity was available; the failure is post-dispatch, not allocation.
• Not adding new approval gates or new clarifying questions. Per the planner rules ("After round 3, unresolved disagreement alone is not enough to block… Concerns about diagnostics… should be surfaced for interpretation, not turned into pass/fail gates"), and per the recovery-rescue rules ("If the root cause is external… stop with a concise blocker"), this is a stop-and-escalate situation.

Summary for the owner

Five consecutive dispatch attempts (r1–r4 team, r5 personal) of an approved, unchanged experiment plan have failed with the same post-RUNNING pod-disappearance signature on RunPod, across two accounts. The r5 retry additionally committed 2 pods when the spec requested 1, which suggests a runner-side dispatch bug worth investigating in its own right. The blocker is upstream of the experiment plan and requires a manual look at RunPod console events and the runner's agent_run_events / pod-provisioner logic. Once that root cause is identified and remediated, the existing approved plan can be re-dispatched as-is — no plan changes required.