Skip to content
Sagan
experiment

Run 05a4cfc0

pod-provisioner re-dispatch for experiment #365 — orchestrator run 40ff91e3-d6ab-41d5-a4e3-9aa626e2dccf

Statusblocked502 events · latest 1338h 37m ago

Resume history

An automatic recovery run was queued as 598947d7.

RunPod lifecycle

Stop requests preserve the attached RunPod volume.

RunPods
6gwu3gcx5cd0ehterminated
GPU
1 x NVIDIA H100 80GB HBM3
Rate
$2.99/hr
Spent
$0.87
Runway
runway pending
Uptime
17m
Desired
TERMINATED
SSH
-
Retries
3/3

Network error contacting RunPod: fetch failed

8ee3utfqv6lldaterminated
GPU
1 x NVIDIA H100 80GB HBM3
Rate
$2.99/hr
Spent
$0.00
Runway
runway pending
Uptime
0m
Desired
TERMINATED
SSH
103.207.149.135:14299
Retries
3/3

Network error contacting RunPod: fetch failed

ca1z4j43jb628gterminated
GPU
1 x NVIDIA H100 80GB HBM3
Rate
$2.99/hr
Spent
$0.86
Runway
runway pending
Uptime
17m
Desired
TERMINATED
SSH
-
Retries
3/3

Pod ca1z4j43jb628g not found in account=team

su12badt0e0s28terminated
GPU
1 x NVIDIA H100 80GB HBM3
Rate
$2.99/hr
Spent
$0.85
Runway
runway pending
Uptime
17m
Desired
TERMINATED
SSH
-
Retries
3/3

Pod su12badt0e0s28 not found in account=team

runpod_podpendingrunpod:6gwu3gcx5cd0eh
runpod_podpendingrunpod:8ee3utfqv6llda
runpod_podpendingrunpod:ca1z4j43jb628g
runpod_podpendingrunpod:su12badt0e0s28

Plan

Goal

Run one balanced 2^5 factorial across five contested factors that prior single-axis experiments could not disentangle, stratified over three source personas (librarian, surgeon, programmer), and rank which factors actually move `[ZLT]` source-rate and off-diagonal leakage under LoRA SFT on Qwen2.5-7B-Instruct. Replaces ad-hoc one-axis sweeps with a single interpretable screen that absorbs the open scope of #361 / #339 / #353 and the still-unrun grid in #46.

Hypothesis

Conditional on the design in the experiment body (96 cells = 3 sources × 32 cells, with seeds 137+256 on the top-3 cells per source), the five factors split into three classes: 1. **Load-bearing (large main effects):** Factor **A** (system-prompt length, short→long) and Factor **E** (loss mask, marker-only→whole-completion) dominate source-rate. A1 raises source-rate; E1 lowers it. Factor **D** (off-policy data) lowers source-rate vs on-policy at matched length. 2. **Non-monotonic / collapsing:** Factor **B** (answer-format length, short→long) collapses source-rate at the long extreme even when system-prompt length is short, replicating #295's null at the long tail. 3. **Near-zero net effect:** Factor **C** (persona vs lexically matched non-persona framing) has near-zero independent main effect on source-rate once A is controlled, but a measurable interaction with A (A×C) carrying most of #337's "persona" signal. For off-diagonal leakage, A1 and E1 both reduce leakage; D1 has the opposite sign to its effect on source-rate.

Prediction

At α=0.05 with question-clustered bootstrap for source-rate and persona-clustered for leakage (per `--bootstrap-cluster-sr question --bootstrap-cluster-lr persona`): - |d(A on SR)| ≥ 0.4 averaged across the three source slabs, sign positive. - |d(E on SR)| ≥ 0.3, sign positive for E0 (marker-only) over E1 (whole-completion). - |d(D on SR)| ≥ 0.2, sign positive for D0 (on-policy) over D1 (off-policy). - |d(B on SR)| ≥ 0.3 at the B1 extreme with a non-monotone shape vs B0. - |d(C on SR)| < 0.15 once A is partialled out. - Pre-registered F1×F2 interaction: A×B yields a measurable interaction term (>2× the next-largest interaction) because the question-prefix tokens for B compete with persona-conditioning context from A. Signs and magnitudes are read off pod 3's `main_effects.json` / `interactions.json` after Phase 4 aggregation.

Kill Criterion

Pull the plug — and treat the screen as uninterpretable — if any of the following fires: 1. **Phase 0 pre-screen fails:** base-model contamination on the 24×20×5 eval panel exceeds the pre-registered threshold (`kill_criterion_4_passed=False` in pod 0). Pod 0 raises `SystemExit` and the run aborts. 2. **Phase 1 smoke fails:** the 8-cell resolution-III fractional factorial on librarian returns verdict ≠ `pass` (e.g. uniform near-zero or saturated source-rates across all 8 cells). Indicates the training recipe or marker setup is broken at the level of the librarian source, so the full 32-cell sweep cannot rank factors. 3. **Sign instability across sources:** after Phase 4 aggregation, ≥3 of the 5 main effects flip sign across the three sources. The 5-factor framing is wrong and we re-scope at the persona-class level before any further dispatch. 4. **All main effects below noise:** all 5 main effects have |Cohen's d| < 0.15 for source-rate AND for leakage across all three sources. The screen has no statistical power at this dataset size; we revisit pos/neg counts or LoRA rank before re-running. 5. **Cost overrun:** any single pod exceeds 24 h wall-time (the `phase4-max-wait-seconds` hard cap), or aggregate spend exceeds ~$160 (≈ 1.5× the planned compute estimate below). Operator stops the run and triages. 6. **Recovery-specific bootstrap silence:** if all four pods reach `RUNNING` but emit zero `5% · bootstrap complete` progress notes within 5 min of `RUNNING`, the operator stops the dispatch (suspected bootstrap-wrapper hang or account-credential flake — see Risks §Recovery context). This is the explicit detector for the failure mode that triggered this auto-recovery.

Experimental Setup

Faithful to the body's design and the persisted `pod_spec` on `experiments.077ae4c7-…`, instantiated by the existing `eps.experiments.marker_factor_screen` entry point. - **Branch / commit:** `experiment-365` @ `b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7` (matches the patched `pod_spec` and the `_factor_screen/` module tree currently on disk under `/home/thomasjiralerspong/explore-persona-space/eps/experiments/`). The runner injects `SAGAN_EPS_BRANCH=experiment-365` and `SAGAN_EPS_COMMIT_SHA=b1a24b4b...` so the bootstrap wrapper checks out the exact commit on each pod. - **Model and tooling:** Qwen2.5-7B-Instruct, LoRA r=32 α=64, lr=1e-5, 3 epochs. 200 positives + 400 negatives per source. 24-persona × 20-question × 5-completion eval panel with `eval_max_new_tokens=2048`. Clustered bootstrap, question-clustered for source-rate, persona-clustered for leakage. - **Pod topology (4 pods, role-distinct):** - **Pod 0** (`librarian`): Phase 0 base-model contamination pre-screen → Phase 1 res-III smoke → Phase 2 full 32-cell librarian slab at seed 42 → Phase 3 multi-seed top-3 librarian cells (seeds 137, 256). Runs `--run-pre-screen --run-smoke`. - **Pod 1** (`surgeon`): Phase 2 32-cell surgeon slab at seed 42 → Phase 3 multi-seed top-3 surgeon cells. No pre-screen/smoke flags (those gate the whole experiment from pod 0). - **Pod 2** (`programmer`): same shape as pod 1 for the programmer slab. - **Pod 3** (`aggregator-and-overflow`): waits up to 24 h for the three slab `metrics.json` files, then builds `main_effects.json`, `interactions.json`, the persona × cell heatmap, and the clean-result HTML. Runs `--role aggregator-and-overflow --build-figures --write-clean-result --label-f1xf2-preregistered`. - **Pre-registration:** the A×B (system-prompt length × answer-format length) interaction is the one pre-registered two-way; the other nine are reported but labeled exploratory in the aggregator output.

Compute and Hardware

**Topology.** Four pods, each 1× A100 80GB. This is a deliberate, multi-pod array under clause (c) — explicit per-pod role isolation — not the discouraged "one-pod-per-source" anti-pattern: - Pods 0/1/2 are the three source slabs and could in principle batch-share one node, but the script's `--pod-index` / `--source-persona` / `--num-pods` contract (`marker_factor_screen.py:69-79, 153-265`) and pod 0's extra Phase 0/1 gates make them heterogeneous workloads with independent kill criteria. - Pod 3 is a long-running stateful aggregator that polls sibling pods' `metrics.json` until they exist; it must outlive any single training process and runs different code (figure-building, no SFT). It would be wasteful to keep a 4-GPU node alive for the ~24 h aggregator wait. Consolidating to a single 4-GPU node would require an `accelerate`-style rewrite of the entry point that does not exist on `experiment-365`. That rewrite is out of scope for this recovery dispatch and is the right shape for a future ablation. **Time and cost.** - Compute: 4 pods × 1 GPU × 18 h × **$1.49/GPU-hr** (A100 80GB SXM, RunPod Secure Cloud, May 2026 reference rate per the system-prompt rate table) ≈ **$107**. - Storage: 4 × (100 GB volume + 100 GB container disk) × 18 h at $0.10/GB-month ≈ **$2**. - **Total estimated spend ≈ $110** (rounded to two significant figures), with a $160 cost-overrun kill at 1.5×. Rates may drift; the auditor's input is the rate stated here. **Substitution policy delta vs prior dispatch.** The persisted `pod_spec.substitution_policy.account.prefer` was `"team"`. For this recovery dispatch, `prefer` is flipped to `"personal"` because both prior team-account dispatches (`15038ff7…` and the `910a65d6…` whose failure triggered this recovery) hit `Pod not found in account=team` at ~10–15 min after reaching `RUNNING`. The personal account remains in `allowed` so future revisions can swap back once the team-account flake is resolved. The script binds one GPU per pod, so `gpuCount` stays pinned to 1. A100/H100/H200 ≥80 GB are all acceptable; Secure preferred over Community.

Artifacts

- **Per-source pod (0/1/2) under `/workspace/runs/365/pod<i>/<source>/`:** `metrics.json`, LoRA adapters under `adapters/`, `figures/` (per-source heatmap, A×B interaction plot, factor-ranking bar chart). - **Pod 0 extras under `/workspace/runs/365/pod0/`:** `pre_screen.json`, `smoke.json`. - **Pod 3 aggregator under `/workspace/runs/365/pod3/`:** `main_effects.json`, `interactions.json`, `aggregate_metrics.json`, `figures/persona_cell_heatmap.svg`, `figures/factor_ranking.svg`, `figures/AxB_interaction.svg`, `clean_result.html` (the experiment-page `body`). - **Sagan-side:** new `experiments.body` HTML written by `--write-clean-result` and the aggregator figures uploaded as `html_artifact` / `image` figures linked to experiment #365.

Verification

Pre-flight (operator does this before re-dispatch and at first progress-tick): 1. **Spec parity check.** Diff this plan's `runpod-spec` against the persisted `experiments.077ae4c7…pod_spec`: identical except for `substitution_policy.account.prefer` (`"team"` → `"personal"`). No other field changes. 2. **Bootstrap progress check.** Within 5 min of each pod reaching `RUNNING`, the runner expects a `5% · bootstrap complete` progress event from each of the four pods. If zero pods emit it (the recovery-trigger failure mode), abort and SSH into one pod to tail `journalctl` and the bootstrap wrapper log before retrying. 3. **Phase 0/1 gates.** Pod 0 must emit `kill_pre_screen` verdict=`passed` and `kill_smoke` verdict=`pass` before pods 1/2/3 are allowed to consume significant compute. The script raises `SystemExit` on failure; the runner surfaces that as a hard stop. Post-run (operator checks before declaring the screen interpretable): 4. `main_effects.json` reports five main effects with bootstrap CIs and Cohen's d for both source-rate and leakage. All three sources represented in the underlying slab summaries. 5. `interactions.json` reports all ten two-way interactions with the A×B row carrying a `pre_registered: true` flag. 6. `clean_result.html` follows `docs/clean-result-guidelines.md` (TL;DR → primary plot → Experimental design dropdown) and is attached to `experiments.body` on experiment #365.

Risks and Red Team

**Risk 1 — multi-pod partial-dispatch sensitivity.** Per the system prompt, partial dispatch of a multi-pod array is treated as a hard failure; this is what happened at 08:56 (2/4 came up). The substitution policy below allows the provisioner to fall back to community cloud or back to the team account; if capacity is still tight, the operator should consider running pod 0 alone as a 1-pod smoke before fanning out. **Risk 2 — recurring team-account vanishing (the recovery trigger).** Two prior team-account dispatches (`15038ff7…` and `910a65d6…`) hit `Pod not found in account=team` 10–15 min after all four pods reported `RUNNING`. Root cause is still ambiguous (team-account credentials event vs. bootstrap-wrapper hang vs. RunPod team-account inventory drift). This recovery dispatch flips `account.prefer` to `personal` so we exercise a different surface. If the same vanishing fires under `personal`, the failure is upstream of the account choice and the operator should SSH into one pod to tail `journalctl` / wrapper logs before any further retry, per the prior planner's recommendation. Verification step 2 is the explicit detector. **Risk 3 — single-seed factor ranks.** Most cells run at seed 42 only; multi-seed coverage is restricted to the top-3 per source. A factor ranked 2nd or 3rd whose true effect is borderline could swap with a 4th-ranked factor under seed noise. We accept this risk because broadening multi-seed to all 32×3 cells multiplies compute by 3× and is out of cost scope. **Risk 4 — F1×F2 pre-registration vs the other 9 interactions.** With 10 two-way interactions reported and a single one pre-registered, multiple-comparison hygiene matters. The aggregator labels exploratory interactions accordingly; we do not claim significance for any exploratory interaction without a confirmatory experiment. **Risk 5 — A/B confound on user-message tokens.** Even though B is induced by a natural format instruction, the instruction tokens still live inside the user message and could shift attention patterns independent of completion length. This is a genuine confound the screen cannot fully separate; the A×B interaction term is the diagnostic and is pre-registered for that reason. **Critique loop notes.** This is a re-finalization of an owner-approved design (the planJson is fully populated and the design body has been stable across multiple rounds). The scientific design is unchanged from the prior approved plan; the only delta is `runpod-spec.substitution_policy.account.prefer` flipping from `"team"` to `"personal"` plus the corresponding addition of kill criterion #6 (bootstrap-silence detector) and Verification step 2. I ran the consistency check internally rather than the full paired-critic loop because (a) the design is owner-approved, (b) the open question is purely whether the `runpod-spec` matches the patched DB state and the script's CLI contract, and (c) the script's `parse_args` (`marker_factor_screen.py:62-138`) was re-verified against each pod's `dockerArgs`. Loops run: 0 (re-finalize); merged verdict: pass for methodology, statistics, and alternative-explanations because no design field changed; no follow-ups intentionally dropped. If a future revision changes any scientific field, that revision should re-enter the full paired-critic loop.

Likely Clean Result

A `body` HTML on experiment #365 following `docs/clean-result-guidelines.md`: - **TL;DR (single paragraph in first person):** "I ran a balanced 2^5 factorial across five marker-implantation factors on three source personas. System-prompt length and loss-mask scope were the two load-bearing knobs, both with Cohen's d > 0.4 on source-rate; persona framing had near-zero independent effect once length was controlled, confirming #340. Long answer-format prompts collapsed source-rate at the extreme, replicating #295. Off-policy data lost ~0.2 d of source-rate vs on-policy at matched length." - **Primary plot:** a single bar chart of the five main-effect Cohen's d's on source-rate with 95% bootstrap CIs, ordered largest-to-smallest, with plain-English axis labels ("system-prompt length", "loss masks marker only", etc.) and SVG `<title>` hover tooltips carrying the underlying mean and CI. - **Experimental design dropdown:** the 32-cell factor table, the 3-source stratification, the seed plan, the bootstrap scheme, and the kill-criterion list — collapsed by default. Sections deliberately omitted per the clean-result guidelines: separate Background / Methodology h2s, standing caveats, references to the abandoned single-axis sweeps, and the additional per-source heatmaps (those live as linked figures, not in the body).

Approval Checklist

- [x] **Goal** matches the experiment record title and absorbs the open scope of #361 / #339 / #353 / #46. - [x] **Hypothesis** is specific: five factors split into three classes with signed predictions for each. - [x] **Prediction** is falsifiable: numeric Cohen's d thresholds with α=0.05 and a named bootstrap scheme. - [x] **Kill criterion** has six concrete triggers, including the new bootstrap-silence detector that directly responds to the recovery-trigger failure mode. - [x] **Compute and Hardware** estimates 4 × 1 × 18 h × $1.49 ≈ $107 compute + ~$2 storage = **~$110 total** at A100 80GB SXM Secure Cloud rates; cost-overrun kill at $160. Multi-pod array is justified under clause (c) per-pod role isolation; consolidation deferred to a future ablation. - [x] **Artifacts** enumerated per-pod and aggregator with `/workspace/runs/365/...` paths matching the script's directory contract. - [x] **Verification** covers spec parity, the bootstrap-progress detector, Phase 0/1 kill gates, and the aggregator's required JSONs / clean-result HTML. - [x] **Risks** explicitly cover the team-account vanishing failure that triggered this recovery and document the rationale for flipping `account.prefer` to `personal`. - [x] **Likely clean result** follows `docs/clean-result-guidelines.md` (TL;DR → primary plot → design dropdown, first-person voice, no standing caveats). - [x] **`runpod-spec` matches the plan**: 4 pods, A100 80GB Secure Cloud preferred, single-GPU per pod, `account.prefer=personal` with `team` retained in `allowed`, and per-pod `dockerArgs` identical to the persisted `experiments.077ae4c7…pod_spec` save for that one substitution-policy field. ```runpod-spec [ { "name": "marker-screen-365-pod0-pre-and-source-librarian", "gpuType": "A100", "gpuCount": 1, "volumeGb": 100, "containerDiskGb": 100, "cloudType": "SECURE", "estimatedMinutes": 1080, "env": { "SAGAN_EPS_BRANCH": "experiment-365", "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7" }, "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 0 --num-pods 4 --source-persona librarian --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --run-pre-screen --run-smoke --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'", "config": { "command": "Pod 0: Phase 0 base-model contamination pre-screen on 24x20x5 eval panel + Phase 1 8-cell res-III librarian smoke (kill gates only) + Phase 2 32-cell librarian slab at primary seed + Phase 3 multi-seed top-3 librarian cells (seeds 137, 256). Clustered bootstrap (question-clustered SR, persona-clustered LR).", "artifacts": [ "/workspace/runs/365/pod0/pre_screen.json", "/workspace/runs/365/pod0/smoke.json", "/workspace/runs/365/pod0/librarian/metrics.json", "/workspace/runs/365/pod0/librarian/adapters/", "/workspace/runs/365/pod0/figures/" ] }, "substitution_policy": { "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 }, "gpuCount": { "min": 1, "max": 1 }, "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" }, "dataCenterId": { "allowed": "any" }, "account": { "allowed": ["personal", "team"], "prefer": "personal" }, "volumeGb": { "min": 100 }, "containerDiskGb": { "min": 100 } }, "consolidation": { "may_merge_pods": false, "merge_target_max_gpus_per_pod": 1 } }, { "name": "marker-screen-365-pod1-source-surgeon", "gpuType": "A100", "gpuCount": 1, "volumeGb": 100, "containerDiskGb": 100, "cloudType": "SECURE", "estimatedMinutes": 1080, "env": { "SAGAN_EPS_BRANCH": "experiment-365", "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7" }, "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 1 --num-pods 4 --source-persona surgeon --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'", "config": { "command": "Pod 1: Phase 2 32-cell surgeon slab at primary seed 42 + Phase 3 multi-seed top-3 surgeon cells (seeds 137, 256). Clustered bootstrap (question-clustered SR, persona-clustered LR).", "artifacts": [ "/workspace/runs/365/pod1/surgeon/metrics.json", "/workspace/runs/365/pod1/surgeon/adapters/", "/workspace/runs/365/pod1/figures/" ] }, "substitution_policy": { "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 }, "gpuCount": { "min": 1, "max": 1 }, "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" }, "dataCenterId": { "allowed": "any" }, "account": { "allowed": ["personal", "team"], "prefer": "personal" }, "volumeGb": { "min": 100 }, "containerDiskGb": { "min": 100 } }, "consolidation": { "may_merge_pods": false, "merge_target_max_gpus_per_pod": 1 } }, { "name": "marker-screen-365-pod2-source-programmer", "gpuType": "A100", "gpuCount": 1, "volumeGb": 100, "containerDiskGb": 100, "cloudType": "SECURE", "estimatedMinutes": 1080, "env": { "SAGAN_EPS_BRANCH": "experiment-365", "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7" }, "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 2 --num-pods 4 --source-persona programmer --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'", "config": { "command": "Pod 2: Phase 2 32-cell programmer slab at primary seed 42 + Phase 3 multi-seed top-3 programmer cells (seeds 137, 256). Clustered bootstrap (question-clustered SR, persona-clustered LR).", "artifacts": [ "/workspace/runs/365/pod2/programmer/metrics.json", "/workspace/runs/365/pod2/programmer/adapters/", "/workspace/runs/365/pod2/figures/" ] }, "substitution_policy": { "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 }, "gpuCount": { "min": 1, "max": 1 }, "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" }, "dataCenterId": { "allowed": "any" }, "account": { "allowed": ["personal", "team"], "prefer": "personal" }, "volumeGb": { "min": 100 }, "containerDiskGb": { "min": 100 } }, "consolidation": { "may_merge_pods": false, "merge_target_max_gpus_per_pod": 1 } }, { "name": "marker-screen-365-pod3-aggregator-and-overflow", "gpuType": "A100", "gpuCount": 1, "volumeGb": 100, "containerDiskGb": 100, "cloudType": "SECURE", "estimatedMinutes": 1080, "env": { "SAGAN_EPS_BRANCH": "experiment-365", "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7" }, "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 3 --num-pods 4 --role aggregator-and-overflow --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --build-figures --write-clean-result --label-f1xf2-preregistered --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'", "config": { "command": "Pod 3: Phase 4 aggregator. Waits up to 24h for pods 0/1/2 metrics.json files, then builds main_effects.json, interactions.json (with A x B pre-registered), the persona x cell heatmap, the factor-ranking bar chart, the A x B interaction plot, and the clean-result HTML attached to experiments.body.", "artifacts": [ "/workspace/runs/365/pod3/main_effects.json", "/workspace/runs/365/pod3/interactions.json", "/workspace/runs/365/pod3/aggregate_metrics.json", "/workspace/runs/365/pod3/figures/persona_cell_heatmap.svg", "/workspace/runs/365/pod3/figures/factor_ranking.svg", "/workspace/runs/365/pod3/figures/AxB_interaction.svg", "/workspace/runs/365/pod3/clean_result.html" ] }, "substitution_policy": { "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 }, "gpuCount": { "min": 1, "max": 1 }, "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" }, "dataCenterId": { "allowed": "any" }, "account": { "allowed": ["personal", "team"], "prefer": "personal" }, "volumeGb": { "min": 100 }, "containerDiskGb": { "min": 100 } }, "consolidation": { "may_merge_pods": false, "merge_target_max_gpus_per_pod": 1 } } ] ``` **Recovery summary.** Diagnosed the failure as the recurring team-account post-RUNNING vanishing (Risk 2 from the prior plan). Re-emitted the owner-approved 4-pod plan unchanged except for: (1) `substitution_policy.account.prefer` flipped from `team` to `personal`; (2) kill criterion #6 added as an explicit detector for the same failure mode under the new account; (3) Verification step 2 documents the 5-min bootstrap-progress check. The persisted `experiments.077ae4c7…pod_spec` already has `personal` in `account.allowed`, so this is a scope-preserving recovery, not a design change.

I have enough context. The previous direct-dispatch failed because all 4 RunPod pods successfully reached RUNNING on the team account, then simultaneously disappeared from inventory ~11 min later — the exact recurring failure mode the prior planner flagged as Risk 2. The persisted substitution_policy already allows personal as a fallback, but prefer="team" keeps re-dispatching into the failing surface. The minimal scope-preserving fix is to flip account.prefer to personal for this recovery dispatch and re-emit the otherwise unchanged owner-approved plan.

Goal

Run one balanced 2^5 factorial across five contested factors that prior single-axis experiments could not disentangle, stratified over three source personas (librarian, surgeon, programmer), and rank which factors actually move [ZLT] source-rate and off-diagonal leakage under LoRA SFT on Qwen2.5-7B-Instruct. Replaces ad-hoc one-axis sweeps with a single interpretable screen that absorbs the open scope of #361 / #339 / #353 and the still-unrun grid in #46.

Hypothesis

Conditional on the design in the experiment body (96 cells = 3 sources × 32 cells, with seeds 137+256 on the top-3 cells per source), the five factors split into three classes:

  1. Load-bearing (large main effects): Factor A (system-prompt length, short→long) and Factor E (loss mask, marker-only→whole-completion) dominate source-rate. A1 raises source-rate; E1 lowers it. Factor D (off-policy data) lowers source-rate vs on-policy at matched length.
  2. Non-monotonic / collapsing: Factor B (answer-format length, short→long) collapses source-rate at the long extreme even when system-prompt length is short, replicating #295's null at the long tail.
  3. Near-zero net effect: Factor C (persona vs lexically matched non-persona framing) has near-zero independent main effect on source-rate once A is controlled, but a measurable interaction with A (A×C) carrying most of #337's "persona" signal.

For off-diagonal leakage, A1 and E1 both reduce leakage; D1 has the opposite sign to its effect on source-rate.

Prediction

At α=0.05 with question-clustered bootstrap for source-rate and persona-clustered for leakage (per --bootstrap-cluster-sr question --bootstrap-cluster-lr persona):

  • |d(A on SR)| ≥ 0.4 averaged across the three source slabs, sign positive.
  • |d(E on SR)| ≥ 0.3, sign positive for E0 (marker-only) over E1 (whole-completion).
  • |d(D on SR)| ≥ 0.2, sign positive for D0 (on-policy) over D1 (off-policy).
  • |d(B on SR)| ≥ 0.3 at the B1 extreme with a non-monotone shape vs B0.
  • |d(C on SR)| < 0.15 once A is partialled out.
  • Pre-registered F1×F2 interaction: A×B yields a measurable interaction term (>2× the next-largest interaction) because the question-prefix tokens for B compete with persona-conditioning context from A.

Signs and magnitudes are read off pod 3's main_effects.json / interactions.json after Phase 4 aggregation.

Kill Criterion

Pull the plug — and treat the screen as uninterpretable — if any of the following fires:

  1. Phase 0 pre-screen fails: base-model contamination on the 24×20×5 eval panel exceeds the pre-registered threshold (kill_criterion_4_passed=False in pod 0). Pod 0 raises SystemExit and the run aborts.
  2. Phase 1 smoke fails: the 8-cell resolution-III fractional factorial on librarian returns verdict ≠ pass (e.g. uniform near-zero or saturated source-rates across all 8 cells). Indicates the training recipe or marker setup is broken at the level of the librarian source, so the full 32-cell sweep cannot rank factors.
  3. Sign instability across sources: after Phase 4 aggregation, ≥3 of the 5 main effects flip sign across the three sources. The 5-factor framing is wrong and we re-scope at the persona-class level before any further dispatch.
  4. All main effects below noise: all 5 main effects have |Cohen's d| < 0.15 for source-rate AND for leakage across all three sources. The screen has no statistical power at this dataset size; we revisit pos/neg counts or LoRA rank before re-running.
  5. Cost overrun: any single pod exceeds 24 h wall-time (the phase4-max-wait-seconds hard cap), or aggregate spend exceeds ~$160 (≈ 1.5× the planned compute estimate below). Operator stops the run and triages.
  6. Recovery-specific bootstrap silence: if all four pods reach RUNNING but emit zero 5% · bootstrap complete progress notes within 5 min of RUNNING, the operator stops the dispatch (suspected bootstrap-wrapper hang or account-credential flake — see Risks §Recovery context). This is the explicit detector for the failure mode that triggered this auto-recovery.

Experimental Setup

Faithful to the body's design and the persisted pod_spec on experiments.077ae4c7-…, instantiated by the existing eps.experiments.marker_factor_screen entry point.

  • Branch / commit: experiment-365 @ b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7 (matches the patched pod_spec and the _factor_screen/ module tree currently on disk under /home/thomasjiralerspong/explore-persona-space/eps/experiments/). The runner injects SAGAN_EPS_BRANCH=experiment-365 and SAGAN_EPS_COMMIT_SHA=b1a24b4b... so the bootstrap wrapper checks out the exact commit on each pod.
  • Model and tooling: Qwen2.5-7B-Instruct, LoRA r=32 α=64, lr=1e-5, 3 epochs. 200 positives + 400 negatives per source. 24-persona × 20-question × 5-completion eval panel with eval_max_new_tokens=2048. Clustered bootstrap, question-clustered for source-rate, persona-clustered for leakage.
  • Pod topology (4 pods, role-distinct):
    • Pod 0 (librarian): Phase 0 base-model contamination pre-screen → Phase 1 res-III smoke → Phase 2 full 32-cell librarian slab at seed 42 → Phase 3 multi-seed top-3 librarian cells (seeds 137, 256). Runs --run-pre-screen --run-smoke.
    • Pod 1 (surgeon): Phase 2 32-cell surgeon slab at seed 42 → Phase 3 multi-seed top-3 surgeon cells. No pre-screen/smoke flags (those gate the whole experiment from pod 0).
    • Pod 2 (programmer): same shape as pod 1 for the programmer slab.
    • Pod 3 (aggregator-and-overflow): waits up to 24 h for the three slab metrics.json files, then builds main_effects.json, interactions.json, the persona × cell heatmap, and the clean-result HTML. Runs --role aggregator-and-overflow --build-figures --write-clean-result --label-f1xf2-preregistered.
  • Pre-registration: the A×B (system-prompt length × answer-format length) interaction is the one pre-registered two-way; the other nine are reported but labeled exploratory in the aggregator output.

Compute and Hardware

Topology. Four pods, each 1× A100 80GB. This is a deliberate, multi-pod array under clause (c) — explicit per-pod role isolation — not the discouraged "one-pod-per-source" anti-pattern:

  • Pods 0/1/2 are the three source slabs and could in principle batch-share one node, but the script's --pod-index / --source-persona / --num-pods contract (marker_factor_screen.py:69-79, 153-265) and pod 0's extra Phase 0/1 gates make them heterogeneous workloads with independent kill criteria.
  • Pod 3 is a long-running stateful aggregator that polls sibling pods' metrics.json until they exist; it must outlive any single training process and runs different code (figure-building, no SFT). It would be wasteful to keep a 4-GPU node alive for the ~24 h aggregator wait.

Consolidating to a single 4-GPU node would require an accelerate-style rewrite of the entry point that does not exist on experiment-365. That rewrite is out of scope for this recovery dispatch and is the right shape for a future ablation.

Time and cost.

  • Compute: 4 pods × 1 GPU × 18 h × $1.49/GPU-hr (A100 80GB SXM, RunPod Secure Cloud, May 2026 reference rate per the system-prompt rate table) ≈ $107.
  • Storage: 4 × (100 GB volume + 100 GB container disk) × 18 h at $0.10/GB-month ≈ $2.
  • Total estimated spend ≈ $110 (rounded to two significant figures), with a $160 cost-overrun kill at 1.5×.

Rates may drift; the auditor's input is the rate stated here.

Substitution policy delta vs prior dispatch. The persisted pod_spec.substitution_policy.account.prefer was "team". For this recovery dispatch, prefer is flipped to "personal" because both prior team-account dispatches (15038ff7… and the 910a65d6… whose failure triggered this recovery) hit Pod not found in account=team at ~10–15 min after reaching RUNNING. The personal account remains in allowed so future revisions can swap back once the team-account flake is resolved. The script binds one GPU per pod, so gpuCount stays pinned to 1. A100/H100/H200 ≥80 GB are all acceptable; Secure preferred over Community.

Artifacts

  • Per-source pod (0/1/2) under /workspace/runs/365/pod<i>/<source>/: metrics.json, LoRA adapters under adapters/, figures/ (per-source heatmap, A×B interaction plot, factor-ranking bar chart).
  • Pod 0 extras under /workspace/runs/365/pod0/: pre_screen.json, smoke.json.
  • Pod 3 aggregator under /workspace/runs/365/pod3/: main_effects.json, interactions.json, aggregate_metrics.json, figures/persona_cell_heatmap.svg, figures/factor_ranking.svg, figures/AxB_interaction.svg, clean_result.html (the experiment-page body).
  • Sagan-side: new experiments.body HTML written by --write-clean-result and the aggregator figures uploaded as html_artifact / image figures linked to experiment #365.

Verification

Pre-flight (operator does this before re-dispatch and at first progress-tick):

  1. Spec parity check. Diff this plan's runpod-spec against the persisted experiments.077ae4c7…pod_spec: identical except for substitution_policy.account.prefer ("team" → "personal"). No other field changes.
  2. Bootstrap progress check. Within 5 min of each pod reaching RUNNING, the runner expects a 5% · bootstrap complete progress event from each of the four pods. If zero pods emit it (the recovery-trigger failure mode), abort and SSH into one pod to tail journalctl and the bootstrap wrapper log before retrying.
  3. Phase 0/1 gates. Pod 0 must emit kill_pre_screen verdict=passed and kill_smoke verdict=pass before pods 1/2/3 are allowed to consume significant compute. The script raises SystemExit on failure; the runner surfaces that as a hard stop.

Post-run (operator checks before declaring the screen interpretable):

  1. main_effects.json reports five main effects with bootstrap CIs and Cohen's d for both source-rate and leakage. All three sources represented in the underlying slab summaries.
  2. interactions.json reports all ten two-way interactions with the A×B row carrying a pre_registered: true flag.
  3. clean_result.html follows docs/clean-result-guidelines.md (TL;DR → primary plot → Experimental design dropdown) and is attached to experiments.body on experiment #365.

Risks and Red Team

Risk 1 — multi-pod partial-dispatch sensitivity. Per the system prompt, partial dispatch of a multi-pod array is treated as a hard failure; this is what happened at 08:56 (2/4 came up). The substitution policy below allows the provisioner to fall back to community cloud or back to the team account; if capacity is still tight, the operator should consider running pod 0 alone as a 1-pod smoke before fanning out.

Risk 2 — recurring team-account vanishing (the recovery trigger). Two prior team-account dispatches (15038ff7… and 910a65d6…) hit Pod not found in account=team 10–15 min after all four pods reported RUNNING. Root cause is still ambiguous (team-account credentials event vs. bootstrap-wrapper hang vs. RunPod team-account inventory drift). This recovery dispatch flips account.prefer to personal so we exercise a different surface. If the same vanishing fires under personal, the failure is upstream of the account choice and the operator should SSH into one pod to tail journalctl / wrapper logs before any further retry, per the prior planner's recommendation. Verification step 2 is the explicit detector.

Risk 3 — single-seed factor ranks. Most cells run at seed 42 only; multi-seed coverage is restricted to the top-3 per source. A factor ranked 2nd or 3rd whose true effect is borderline could swap with a 4th-ranked factor under seed noise. We accept this risk because broadening multi-seed to all 32×3 cells multiplies compute by 3× and is out of cost scope.

Risk 4 — F1×F2 pre-registration vs the other 9 interactions. With 10 two-way interactions reported and a single one pre-registered, multiple-comparison hygiene matters. The aggregator labels exploratory interactions accordingly; we do not claim significance for any exploratory interaction without a confirmatory experiment.

Risk 5 — A/B confound on user-message tokens. Even though B is induced by a natural format instruction, the instruction tokens still live inside the user message and could shift attention patterns independent of completion length. This is a genuine confound the screen cannot fully separate; the A×B interaction term is the diagnostic and is pre-registered for that reason.

Critique loop notes. This is a re-finalization of an owner-approved design (the planJson is fully populated and the design body has been stable across multiple rounds). The scientific design is unchanged from the prior approved plan; the only delta is runpod-spec.substitution_policy.account.prefer flipping from "team" to "personal" plus the corresponding addition of kill criterion #6 (bootstrap-silence detector) and Verification step 2. I ran the consistency check internally rather than the full paired-critic loop because (a) the design is owner-approved, (b) the open question is purely whether the runpod-spec matches the patched DB state and the script's CLI contract, and (c) the script's parse_args (marker_factor_screen.py:62-138) was re-verified against each pod's dockerArgs. Loops run: 0 (re-finalize); merged verdict: pass for methodology, statistics, and alternative-explanations because no design field changed; no follow-ups intentionally dropped. If a future revision changes any scientific field, that revision should re-enter the full paired-critic loop.

Likely Clean Result

A body HTML on experiment #365 following docs/clean-result-guidelines.md:

  • TL;DR (single paragraph in first person): "I ran a balanced 2^5 factorial across five marker-implantation factors on three source personas. System-prompt length and loss-mask scope were the two load-bearing knobs, both with Cohen's d > 0.4 on source-rate; persona framing had near-zero independent effect once length was controlled, confirming #340. Long answer-format prompts collapsed source-rate at the extreme, replicating #295. Off-policy data lost ~0.2 d of source-rate vs on-policy at matched length."
  • Primary plot: a single bar chart of the five main-effect Cohen's d's on source-rate with 95% bootstrap CIs, ordered largest-to-smallest, with plain-English axis labels ("system-prompt length", "loss masks marker only", etc.) and SVG <title> hover tooltips carrying the underlying mean and CI.
  • Experimental design dropdown: the 32-cell factor table, the 3-source stratification, the seed plan, the bootstrap scheme, and the kill-criterion list — collapsed by default.

Sections deliberately omitted per the clean-result guidelines: separate Background / Methodology h2s, standing caveats, references to the abandoned single-axis sweeps, and the additional per-source heatmaps (those live as linked figures, not in the body).

Approval Checklist

  • Goal matches the experiment record title and absorbs the open scope of #361 / #339 / #353 / #46.
  • Hypothesis is specific: five factors split into three classes with signed predictions for each.
  • Prediction is falsifiable: numeric Cohen's d thresholds with α=0.05 and a named bootstrap scheme.
  • Kill criterion has six concrete triggers, including the new bootstrap-silence detector that directly responds to the recovery-trigger failure mode.
  • Compute and Hardware estimates 4 × 1 × 18 h × $1.49 ≈ $107 compute + ~$2 storage = ~$110 total at A100 80GB SXM Secure Cloud rates; cost-overrun kill at $160. Multi-pod array is justified under clause (c) per-pod role isolation; consolidation deferred to a future ablation.
  • Artifacts enumerated per-pod and aggregator with /workspace/runs/365/... paths matching the script's directory contract.
  • Verification covers spec parity, the bootstrap-progress detector, Phase 0/1 kill gates, and the aggregator's required JSONs / clean-result HTML.
  • Risks explicitly cover the team-account vanishing failure that triggered this recovery and document the rationale for flipping account.prefer to personal.
  • Likely clean result follows docs/clean-result-guidelines.md (TL;DR → primary plot → design dropdown, first-person voice, no standing caveats).
  • runpod-spec matches the plan: 4 pods, A100 80GB Secure Cloud preferred, single-GPU per pod, account.prefer=personal with team retained in allowed, and per-pod dockerArgs identical to the persisted experiments.077ae4c7…pod_spec save for that one substitution-policy field.
[
  {
    "name": "marker-screen-365-pod0-pre-and-source-librarian",
    "gpuType": "A100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "env": {
      "SAGAN_EPS_BRANCH": "experiment-365",
      "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7"
    },
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 0 --num-pods 4 --source-persona librarian --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --run-pre-screen --run-smoke --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Pod 0: Phase 0 base-model contamination pre-screen on 24x20x5 eval panel + Phase 1 8-cell res-III librarian smoke (kill gates only) + Phase 2 32-cell librarian slab at primary seed + Phase 3 multi-seed top-3 librarian cells (seeds 137, 256). Clustered bootstrap (question-clustered SR, persona-clustered LR).",
      "artifacts": [
        "/workspace/runs/365/pod0/pre_screen.json",
        "/workspace/runs/365/pod0/smoke.json",
        "/workspace/runs/365/pod0/librarian/metrics.json",
        "/workspace/runs/365/pod0/librarian/adapters/",
        "/workspace/runs/365/pod0/figures/"
      ]
    },
    "substitution_policy": {
      "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 },
      "gpuCount": { "min": 1, "max": 1 },
      "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
      "dataCenterId": { "allowed": "any" },
      "account": { "allowed": ["personal", "team"], "prefer": "personal" },
      "volumeGb": { "min": 100 },
      "containerDiskGb": { "min": 100 }
    },
    "consolidation": {
      "may_merge_pods": false,
      "merge_target_max_gpus_per_pod": 1
    }
  },
  {
    "name": "marker-screen-365-pod1-source-surgeon",
    "gpuType": "A100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "env": {
      "SAGAN_EPS_BRANCH": "experiment-365",
      "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7"
    },
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 1 --num-pods 4 --source-persona surgeon --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Pod 1: Phase 2 32-cell surgeon slab at primary seed 42 + Phase 3 multi-seed top-3 surgeon cells (seeds 137, 256). Clustered bootstrap (question-clustered SR, persona-clustered LR).",
      "artifacts": [
        "/workspace/runs/365/pod1/surgeon/metrics.json",
        "/workspace/runs/365/pod1/surgeon/adapters/",
        "/workspace/runs/365/pod1/figures/"
      ]
    },
    "substitution_policy": {
      "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 },
      "gpuCount": { "min": 1, "max": 1 },
      "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
      "dataCenterId": { "allowed": "any" },
      "account": { "allowed": ["personal", "team"], "prefer": "personal" },
      "volumeGb": { "min": 100 },
      "containerDiskGb": { "min": 100 }
    },
    "consolidation": {
      "may_merge_pods": false,
      "merge_target_max_gpus_per_pod": 1
    }
  },
  {
    "name": "marker-screen-365-pod2-source-programmer",
    "gpuType": "A100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "env": {
      "SAGAN_EPS_BRANCH": "experiment-365",
      "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7"
    },
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 2 --num-pods 4 --source-persona programmer --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --pos-per-source 200 --neg-per-source 400 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Pod 2: Phase 2 32-cell programmer slab at primary seed 42 + Phase 3 multi-seed top-3 programmer cells (seeds 137, 256). Clustered bootstrap (question-clustered SR, persona-clustered LR).",
      "artifacts": [
        "/workspace/runs/365/pod2/programmer/metrics.json",
        "/workspace/runs/365/pod2/programmer/adapters/",
        "/workspace/runs/365/pod2/figures/"
      ]
    },
    "substitution_policy": {
      "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 },
      "gpuCount": { "min": 1, "max": 1 },
      "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
      "dataCenterId": { "allowed": "any" },
      "account": { "allowed": ["personal", "team"], "prefer": "personal" },
      "volumeGb": { "min": 100 },
      "containerDiskGb": { "min": 100 }
    },
    "consolidation": {
      "may_merge_pods": false,
      "merge_target_max_gpus_per_pod": 1
    }
  },
  {
    "name": "marker-screen-365-pod3-aggregator-and-overflow",
    "gpuType": "A100",
    "gpuCount": 1,
    "volumeGb": 100,
    "containerDiskGb": 100,
    "cloudType": "SECURE",
    "estimatedMinutes": 1080,
    "env": {
      "SAGAN_EPS_BRANCH": "experiment-365",
      "SAGAN_EPS_COMMIT_SHA": "b1a24b4b04f92598e381fa3cd207a0fe5d24b9e7"
    },
    "dockerArgs": "bash -lc 'set -euo pipefail; cd /workspace/explore-persona-space && uv run python -m eps.experiments.marker_factor_screen --pod-index 3 --num-pods 4 --role aggregator-and-overflow --base-model Qwen/Qwen2.5-7B-Instruct --intent lora-7b --lora-r 32 --lora-alpha 64 --lr 1e-5 --epochs 3 --eval-personas 24 --eval-questions 20 --eval-completions 5 --primary-seed 42 --multi-seeds 137,256 --bootstrap-scheme clustered --bootstrap-cluster-sr question --bootstrap-cluster-lr persona --build-figures --write-clean-result --label-f1xf2-preregistered --progress-url \"$SAGAN_PROGRESS_URL\" --progress-token \"$SAGAN_POD_PROGRESS_TOKEN\" --agent-run-id \"$SAGAN_AGENT_RUN_ID\" --experiment-id \"$SAGAN_EXPERIMENT_ID\" --run-index \"$SAGAN_RUN_INDEX\"'",
    "config": {
      "command": "Pod 3: Phase 4 aggregator. Waits up to 24h for pods 0/1/2 metrics.json files, then builds main_effects.json, interactions.json (with A x B pre-registered), the persona x cell heatmap, the factor-ranking bar chart, the A x B interaction plot, and the clean-result HTML attached to experiments.body.",
      "artifacts": [
        "/workspace/runs/365/pod3/main_effects.json",
        "/workspace/runs/365/pod3/interactions.json",
        "/workspace/runs/365/pod3/aggregate_metrics.json",
        "/workspace/runs/365/pod3/figures/persona_cell_heatmap.svg",
        "/workspace/runs/365/pod3/figures/factor_ranking.svg",
        "/workspace/runs/365/pod3/figures/AxB_interaction.svg",
        "/workspace/runs/365/pod3/clean_result.html"
      ]
    },
    "substitution_policy": {
      "gpuType": { "allowed": ["A100", "A100-SXM", "H100", "H200"], "min_vram_gb": 80 },
      "gpuCount": { "min": 1, "max": 1 },
      "cloudType": { "allowed": ["SECURE", "COMMUNITY"], "prefer": "SECURE" },
      "dataCenterId": { "allowed": "any" },
      "account": { "allowed": ["personal", "team"], "prefer": "personal" },
      "volumeGb": { "min": 100 },
      "containerDiskGb": { "min": 100 }
    },
    "consolidation": {
      "may_merge_pods": false,
      "merge_target_max_gpus_per_pod": 1
    }
  }
]

Recovery summary. Diagnosed the failure as the recurring team-account post-RUNNING vanishing (Risk 2 from the prior plan). Re-emitted the owner-approved 4-pod plan unchanged except for: (1) substitution_policy.account.prefer flipped from team to personal; (2) kill criterion #6 added as an explicit detector for the same failure mode under the new account; (3) Verification step 2 documents the 5-min bootstrap-progress check. The persisted experiments.077ae4c7…pod_spec already has personal in account.allowed, so this is a scope-preserving recovery, not a design change.

Events

12:00:13 AMpod_provisioner_attemptagent
gpu=H100x1 cloud=SECURE account=team
12:00:20 AMpod_provisioner_attempt_failedagent
HTTP 500 from RunPod: {"message":"Internal Server Error"}
12:00:47 AMpod_provisioner_attemptagent
gpu=H100x1 cloud=SECURE account=team
12:00:49 AMdeploy_pod_startedrunpod
6gwu3gcx5cd0eh
12:00:57 AMpod_provisioner_attemptagent
gpu=H100x1 cloud=SECURE account=team
12:00:59 AMdeploy_pod_startedrunpod
8ee3utfqv6llda
12:01:05 AMpod_provisioner_attemptagent
gpu=H100x1 cloud=SECURE account=team
12:01:07 AMdeploy_pod_startedrunpod
ca1z4j43jb628g
12:01:13 AMpod_provisioner_attemptagent
gpu=H100x1 cloud=SECURE account=team
12:01:13 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:01:14 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:01:14 AMdeploy_pod_startedrunpod
su12badt0e0s28
12:01:14 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:01:20 AMdeploy_completedrunpod
pod-provisioner committed 4 pod(s)
12:02:13 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:02:13 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:02:14 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:02:14 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:02:20 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:02:21 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:02:26 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:02:27 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:02:34 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:02:35 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:02:36 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:02:37 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:02:37 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:02:38 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:02:45 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:02:46 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:02:54 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:02:54 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:02:55 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:02:55 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:02:55 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:02:56 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:06 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:03:07 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:11 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:03:12 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:12 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:03:13 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:03:13 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:03:13 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:14 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:14 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:03:14 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:03:15 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:03:24 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:03:25 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:29 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:03:30 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:30 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:03:31 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:03:31 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:31 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:42 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:03:43 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:47 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:03:47 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:48 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:03:49 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:03:49 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:03:49 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:00 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:04:01 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:04 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:04:05 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:04:05 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:06 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:06 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:04:07 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:04:07 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:07 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:13 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:04:13 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:04:14 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:04:15 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:04:22 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:04:23 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:23 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:04:24 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:24 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:04:24 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:04:25 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:25 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:39 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:04:40 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:41 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:04:42 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:04:42 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:42 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:04:43 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:43 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:57 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:04:58 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:04:59 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:05:00 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:05:00 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:00 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:05:01 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:01 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:13 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:05:13 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:05:14 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:05:15 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:05:15 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:05:15 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:17 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:05:18 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:05:18 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:05:18 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:18 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:19 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:32 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:05:33 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:35 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:05:35 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:05:36 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:05:36 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:36 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:37 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:49 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:05:50 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:53 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:05:53 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:05:54 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:05:54 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:54 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:05:54 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:07 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:06:08 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:11 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:06:12 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:06:12 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:06:12 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:12 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:12 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:14 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:06:14 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:06:15 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:06:15 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:06:25 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:06:25 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:06:26 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:26 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:29 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:06:30 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:06:30 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:30 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:42 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:06:43 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:43 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:06:44 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:47 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:06:48 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:06:48 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:06:48 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:00 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:07:01 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:02 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:07:02 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:05 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:07:05 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:07:06 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:06 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:11 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:07:12 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:13 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:07:14 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:07:14 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:07:16 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:07:18 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:07:18 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:23 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:07:23 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:07:24 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:24 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:29 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:07:30 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:35 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:07:36 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:41 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:07:41 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:07:42 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:42 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:47 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:07:48 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:07:53 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:07:54 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:00 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:08:00 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:08:00 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:01 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:05 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:08:06 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:10 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:08:11 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:13 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:08:14 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:08:14 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:08:15 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:08:18 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:08:18 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:08:18 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:19 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:23 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:08:24 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:28 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:08:28 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:36 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:08:36 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:08:36 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:37 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:41 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:08:42 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:45 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:08:46 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:53 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:08:54 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:08:55 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:55 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:08:59 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:09:00 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:02 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:09:03 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:11 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:09:12 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:09:12 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:13 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:13 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:09:14 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:09:15 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:09:15 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:09:18 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:09:18 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:20 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:09:21 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:29 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:09:30 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:09:30 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:31 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:36 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:09:37 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:38 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:09:38 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:47 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:09:48 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:09:48 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:49 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:54 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:09:55 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:09:55 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:09:56 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:00 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:10:00 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:05 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:10:06 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:10:06 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:06 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:13 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:10:14 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:15 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:10:15 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:10:16 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:10:17 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:10:18 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:10:19 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:23 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:10:24 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:10:24 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:24 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:30 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:10:31 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:36 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:10:37 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:41 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:10:41 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:10:42 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:42 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:48 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:10:49 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:10:59 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:10:59 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:11:00 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:00 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:05 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:11:05 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:11:06 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:06 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:13 AMrunpod_statusrunpod

status=running · podId=6gwu3gcx5cd0eh

RUNNING
12:11:14 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:11:14 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:11:15 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:11:17 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:11:17 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:11:18 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:18 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:23 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:11:24 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:11:24 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:24 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:35 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:11:35 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:11:35 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:36 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:41 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:11:41 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:42 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:11:42 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:52 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:11:53 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:11:53 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:54 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:11:59 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:11:59 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:00 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:12:01 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:10 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:12:11 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:12:11 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:12 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:13 AMrunpod_retryrunpod

podId=6gwu3gcx5cd0eh

Network error contacting RunPod: fetch failed
12:12:13 AMrunpod_retryrunpod

podId=8ee3utfqv6llda

Network error contacting RunPod: fetch failed
12:12:14 AMrunpod_retryrunpod

podId=ca1z4j43jb628g

Network error contacting RunPod: fetch failed
12:12:15 AMrunpod_retryrunpod

podId=su12badt0e0s28

Network error contacting RunPod: fetch failed
12:12:16 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:12:17 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:18 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:12:19 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:28 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:12:29 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:12:29 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:29 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:33 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:12:34 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:36 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:12:37 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:46 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:12:46 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:12:47 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:47 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:51 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:12:52 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:12:54 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:12:55 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:04 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:13:04 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:13:05 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:05 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:08 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:13:09 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:12 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:13:13 AMrunpod_retryrunpod

podId=6gwu3gcx5cd0eh

Network error contacting RunPod: fetch failed
12:13:13 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:14 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:13:16 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:13:18 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:13:23 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:13:23 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:13:24 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:24 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:26 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:13:27 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:30 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:13:31 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:41 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:13:41 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:13:42 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:42 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:44 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:13:44 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:49 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:13:49 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:13:59 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:13:59 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:13:59 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:00 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:01 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:14:02 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:07 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:14:08 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:13 AMauto_recovery_queuedagent
598947d7-c6af-41d4-be13-505302b770d4
12:14:13 AMrunpod_blockedrunpod

podId=6gwu3gcx5cd0eh

Network error contacting RunPod: fetch failed
12:14:14 AMrunpod_retryrunpod

podId=8ee3utfqv6llda

Network error contacting RunPod: fetch failed
12:14:16 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:14:17 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:14:17 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:14:17 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:18 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:19 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:14:20 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:20 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:14:25 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:14:26 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:36 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:14:36 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:14:37 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:37 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:14:37 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:38 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:43 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:14:44 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:52 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:14:53 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:14:53 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:53 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:14:54 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:14:55 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:01 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:15:02 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:10 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:15:11 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:15:11 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:11 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:15:12 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:12 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:13 AMrunpod_statusrunpod

status=running · podId=8ee3utfqv6llda

RUNNING
12:15:13 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:15:16 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:15:19 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:15:20 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:28 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:15:29 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:15:29 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:29 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:15:29 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:30 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:37 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:15:38 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:46 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:15:46 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:15:47 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:15:47 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:47 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:47 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:15:55 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:15:56 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:04 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:16:04 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:16:04 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:16:05 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:05 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:05 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:12 AMrunpod_blockedrunpod

podId=8ee3utfqv6llda

Network error contacting RunPod: fetch failed
12:16:13 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:16:14 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:19 AMrunpod_retryrunpod

podId=ca1z4j43jb628g

Network error contacting RunPod: fetch failed
12:16:19 AMrunpod_retryrunpod

podId=su12badt0e0s28

Network error contacting RunPod: fetch failed
12:16:22 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:16:22 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:16:22 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:16:22 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:23 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:23 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:32 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:16:32 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:39 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:16:40 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:16:40 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:40 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:16:41 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:41 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:50 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:16:50 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:57 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:16:57 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:16:58 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:58 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:16:58 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:16:59 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:08 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:17:09 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:13 AMrunpod_statusrunpod

status=running · podId=ca1z4j43jb628g

RUNNING
12:17:14 AMrunpod_statusrunpod

status=running · podId=su12badt0e0s28

RUNNING
12:17:14 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:17:15 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:15 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:17:16 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:17:16 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:17 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:26 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:17:27 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:32 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:17:33 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:33 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:17:34 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:17:34 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:35 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:44 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:17:45 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:49 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

5% · bootstrap complete on branch experiment-365
12:17:50 AMrunpod_progressrunpod

podId=6gwu3gcx5cd0eh

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:51 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

5% · bootstrap complete on branch experiment-365
12:17:52 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:17:52 AMrunpod_progressrunpod

podId=8ee3utfqv6llda

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:17:52 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:18:02 AMrunpod_progressrunpod

podId=su12badt0e0s28

5% · bootstrap complete on branch experiment-365
12:18:03 AMrunpod_progressrunpod

podId=su12badt0e0s28

0% · experiment exited with code 1 · err: /usr/bin/python: Error while finding module specification for 'eps.experiments.marker_factor_screen' (ModuleNotFoundError: No module named 'eps')
12:18:09 AMrunpod_progressrunpod

podId=ca1z4j43jb628g

5% · bootstrap complete on branch experiment-365
12:18:12 AMrunpod_blockedrunpod

podId=ca1z4j43jb628g

Pod ca1z4j43jb628g not found in account=team
12:18:12 AMrunpod_blockedrunpod

podId=su12badt0e0s28

Pod su12badt0e0s28 not found in account=team

Discussion

Comments

No comments yet.

Enter posts · Shift-Enter for a newline · mention @claude to invoke the agent