experiment

Run f970a62f

pod-provisioner dispatch for experiment #366 (manual retest after stderr/networkVolumeId fixes) — orchestrator run 7721a1e0-37d2-4087-9103-a84f76ed230e

Statusfailed1069 events · latest 1331h 41m ago

Resume history

An automatic recovery run was queued as 0164ffaa.

RunPod lifecycle

Stop requests preserve the attached RunPod volume.

RunPods

jc3a2x03oxa7b0stopped

GPU: 1 x NVIDIA H100 80GB HBM3
Rate: $2.99/hr
Spent: $0.03
Runway: runway pending
Uptime: 0m
Desired: EXITED
SSH: 103.207.149.137:12215
Retries: 3/3

HTTP 503 from RunPod: {"message":"Service Unavailable"}

runpod_podpendingrunpod:jc3a2x03oxa7b0

Plan

Clarifying questions

This is now the **3rd auto-recovery run** on exp #366 (prior recovery `b047cabb-…`, original `0a864550-…`, plus the just-failed `9e039246-…`). The plan body on `experiments.body` remains final — `compute:small`, 8 adapters on 1×H100, ~3 GPU-hours. Re-running the planner cannot move the work forward; the blockers are structural (runner/infra) plus one human gate. I am producing only clarifying questions so the dashboard flips this experiment to `awaiting_clarifications` and breaks the cascade. **New evidence from this round.** Pod `lqpfpmbytlgded` reached `RUNNING` at 10:44:53 and survived **ten** consecutive 60-second status polls (10:44:53 → 10:53:56 = ~9 min in `RUNNING`) before disappearing at 10:54:54 — a substantially longer window than `w2vn4kqqjotfgj`'s ~3.5 min in the prior round. Nine minutes is **more than enough** for `uv sync --locked` against the `issue-366` branch to finish on a fresh volume and for `run_366.py` to begin training (or to exit with an unlogged error). The "not found in account=team" message from `services/runner/src/tools/runpod.ts` is consistent with the pod having been garbage-collected by RunPod, not with an account-key mismatch — but without log capture we still cannot tell whether the user command crashed early or whether RunPod's allocator reclaimed the pod for unrelated reasons. The cumulative tally is now **seven** vanished pods on this experiment (`yur1wxa4ztn2ky`, `o9hlewv1vn3vfe`, `hx5qglq11gau0i`, `jc3a2x03oxa7b0`, `w8zh8zs644se30`, `w2vn4kqqjotfgj`, `lqpfpmbytlgded`). The six structural items from the prior round are still load-bearing. The longer `RUNNING` window in this attempt sharpens #2 (the `sleep infinity` wrapper would now almost certainly have kept the pod alive past the 9-min point, so it is the cheapest unblock) and adds urgency to #6 (log capture is the only way to tell whether the user command exited cleanly or with an error during those 9 minutes). 1. **Halt the auto-recovery cascade on exp #366.** Mark this run (`<current run id>`), the prior recovery runs (`b047cabb-ad20-4626-…`, `9e039246-73d2-4abc-…`), and the original `0a864550-…` as `failed`. Do you want me to (a) treat the dashboard flip to `awaiting_clarifications` as sufficient on its own — trusting that the runner's status reducer stops re-firing recoveries once the experiment is no longer in a crashed-running state — or (b) also open a `kind=apply` to suppress auto-recovery on experiments whose latest planner run produced `isClarificationOutput=true`, so future intentional blockers cannot loop? [TEXTBOX:halt-strategy] 2. **Pick the runner-side fix to land before dispatch #8.** Same two options as the prior round, with new evidence that #2a alone would likely have rescued the most recent attempt: - **(2a) `sleep infinity` wrapper in `services/runner/src/lib/pod-bootstrap.ts`.** One-line change; appends `; sleep infinity` (or equivalent) after the user command so the pod survives normal-exit garbage collection. Unblocks #366 immediately. Doesn't fix the underlying "no log capture" problem. - **(2b) `progressPct=100` reducer hook in `services/runner/src/`.** Architecturally correct: distinguishes "pod exited because experiment finished" from "pod vanished while still working." Wider blast radius; needs a regression test pass. Which do you want pursued, and in what order? (2a now, 2b as a follow-up apply is the lowest-risk path.) [TEXTBOX:runner-fix-choice] 3. **Decide whether `runpodAccount: team` is intended for #366.** Unchanged from prior round. The runner's auth fallback at `services/runner/src/tools/runpod.ts:96-120` silently routes `team` through the personal key when `RUNPOD_API_KEY_TEAM` is absent (the case on this VM). Consistent across `dispatchPod` and `getPod`, so it should not be the cause of vanish events — but it does mean these pods are billing on personal. Do you want the experiment switched to `runpodAccount: personal` to make accounting explicit, or do you want `RUNPOD_API_KEY_TEAM` provisioned on the runner VM? [TEXTBOX:account-resolution] 4. **Confirm the experiment code on `issue-366` is the intended target for redispatch.** Last known HEAD is `29a985e7` (2026-05-13 20:49 UTC, "exclude A∩B pre-existing shared subtokens from forbidden set in marker resolver"), with `scripts/experiments/366/run_366.py` plus train/eval/markers/figures. Is `issue-366` still the right pointer, or has the implementer staged a newer fix on a different branch since the last dispatch? [TEXTBOX:branch-confirmation] 5. **Decide whether to add `networkVolumeId` before the next dispatch.** Seven vanish events have destroyed any partial artifacts (donor adapters, eval seeds, cached HF weights). Attaching an existing RunPod network volume at `/workspace` would (a) survive future pod losses, (b) drop subsequent `uv sync --locked` to ~30s, and (c) skip the multi-GB Qwen-2.5-7B-Instruct download on every retry. Do you have a usable network volume id, and which DC is it pinned to? Without a real RunPod resource id I cannot guess one, and `dataCenterId` must match the volume's DC or the volume is invisible. [TEXTBOX:network-volume-id] 6. **Capture a single pod's stdout/stderr next dispatch (now strongly recommended).** With this round's 9-minute `RUNNING` window, the most informative diagnostic is what `run_366.py` actually printed during those nine minutes. Once the runner-side fix from #2 lands, the redispatched podSpec can tee the user command into `/workspace/explore-persona-space/run-stdout.log` and have the experiment script POST the head of that log to `SAGAN_PROGRESS_URL` at startup, so the next vanish leaves evidence. Alternatively, the runner-side fix could include `runpod podLogs <pod-id>` capture-on-vanish (heavier change). Which (or both)? [TEXTBOX:log-capture-optin] Once #1 and #2 are answered, the next planner run can fold the answers into a real redispatch plan — experimental sections, cost estimate, runpod-spec, Approval Checklist — without re-litigating the science (the plan body is final).

Clarifying questions

This is now the 3rd auto-recovery run on exp #366 (prior recovery b047cabb-…, original 0a864550-…, plus the just-failed 9e039246-…). The plan body on experiments.body remains final — compute:small, 8 adapters on 1×H100, ~3 GPU-hours. Re-running the planner cannot move the work forward; the blockers are structural (runner/infra) plus one human gate. I am producing only clarifying questions so the dashboard flips this experiment to awaiting_clarifications and breaks the cascade.

New evidence from this round. Pod lqpfpmbytlgded reached RUNNING at 10:44:53 and survived ten consecutive 60-second status polls (10:44:53 → 10:53:56 = ~9 min in RUNNING) before disappearing at 10:54:54 — a substantially longer window than w2vn4kqqjotfgj's ~3.5 min in the prior round. Nine minutes is more than enough for uv sync --locked against the issue-366 branch to finish on a fresh volume and for run_366.py to begin training (or to exit with an unlogged error). The "not found in account=team" message from services/runner/src/tools/runpod.ts is consistent with the pod having been garbage-collected by RunPod, not with an account-key mismatch — but without log capture we still cannot tell whether the user command crashed early or whether RunPod's allocator reclaimed the pod for unrelated reasons. The cumulative tally is now seven vanished pods on this experiment (yur1wxa4ztn2ky, o9hlewv1vn3vfe, hx5qglq11gau0i, jc3a2x03oxa7b0, w8zh8zs644se30, w2vn4kqqjotfgj, lqpfpmbytlgded).

The six structural items from the prior round are still load-bearing. The longer RUNNING window in this attempt sharpens #2 (the sleep infinity wrapper would now almost certainly have kept the pod alive past the 9-min point, so it is the cheapest unblock) and adds urgency to #6 (log capture is the only way to tell whether the user command exited cleanly or with an error during those 9 minutes).

Halt the auto-recovery cascade on exp #366. Mark this run (<current run id>), the prior recovery runs (b047cabb-ad20-4626-…, 9e039246-73d2-4abc-…), and the original 0a864550-… as failed. Do you want me to (a) treat the dashboard flip to awaiting_clarifications as sufficient on its own — trusting that the runner's status reducer stops re-firing recoveries once the experiment is no longer in a crashed-running state — or (b) also open a kind=apply to suppress auto-recovery on experiments whose latest planner run produced isClarificationOutput=true, so future intentional blockers cannot loop? [TEXTBOX:halt-strategy]
Pick the runner-side fix to land before dispatch #8. Same two options as the prior round, with new evidence that #2a alone would likely have rescued the most recent attempt:
- (2a) sleep infinity wrapper in services/runner/src/lib/pod-bootstrap.ts. One-line change; appends ; sleep infinity (or equivalent) after the user command so the pod survives normal-exit garbage collection. Unblocks #366 immediately. Doesn't fix the underlying "no log capture" problem.
- (2b) progressPct=100 reducer hook in services/runner/src/. Architecturally correct: distinguishes "pod exited because experiment finished" from "pod vanished while still working." Wider blast radius; needs a regression test pass. Which do you want pursued, and in what order? (2a now, 2b as a follow-up apply is the lowest-risk path.) [TEXTBOX:runner-fix-choice]
Decide whether runpodAccount: team is intended for #366. Unchanged from prior round. The runner's auth fallback at services/runner/src/tools/runpod.ts:96-120 silently routes team through the personal key when RUNPOD_API_KEY_TEAM is absent (the case on this VM). Consistent across dispatchPod and getPod, so it should not be the cause of vanish events — but it does mean these pods are billing on personal. Do you want the experiment switched to runpodAccount: personal to make accounting explicit, or do you want RUNPOD_API_KEY_TEAM provisioned on the runner VM? [TEXTBOX:account-resolution]
Confirm the experiment code on issue-366 is the intended target for redispatch. Last known HEAD is 29a985e7 (2026-05-13 20:49 UTC, "exclude A∩B pre-existing shared subtokens from forbidden set in marker resolver"), with scripts/experiments/366/run_366.py plus train/eval/markers/figures. Is issue-366 still the right pointer, or has the implementer staged a newer fix on a different branch since the last dispatch? [TEXTBOX:branch-confirmation]
Decide whether to add networkVolumeId before the next dispatch. Seven vanish events have destroyed any partial artifacts (donor adapters, eval seeds, cached HF weights). Attaching an existing RunPod network volume at /workspace would (a) survive future pod losses, (b) drop subsequent uv sync --locked to ~30s, and (c) skip the multi-GB Qwen-2.5-7B-Instruct download on every retry. Do you have a usable network volume id, and which DC is it pinned to? Without a real RunPod resource id I cannot guess one, and dataCenterId must match the volume's DC or the volume is invisible. [TEXTBOX:network-volume-id]
Capture a single pod's stdout/stderr next dispatch (now strongly recommended). With this round's 9-minute RUNNING window, the most informative diagnostic is what run_366.py actually printed during those nine minutes. Once the runner-side fix from #2 lands, the redispatched podSpec can tee the user command into /workspace/explore-persona-space/run-stdout.log and have the experiment script POST the head of that log to SAGAN_PROGRESS_URL at startup, so the next vanish leaves evidence. Alternatively, the runner-side fix could include runpod podLogs <pod-id> capture-on-vanish (heavier change). Which (or both)? [TEXTBOX:log-capture-optin]

Once #1 and #2 are answered, the next planner run can fold the answers into a real redispatch plan — experimental sections, cost estimate, runpod-spec, Approval Checklist — without re-litigating the science (the plan body is final).

Events

11:46:19 PMpod_provisioner_attemptagent

gpu=H100x1 cloud=SECURE account=team

11:46:20 PMdeploy_pod_startedrunpod

jc3a2x03oxa7b0

11:46:24 PMdeploy_completedrunpod

pod-provisioner committed 1 pod(s)

11:47:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:47:58 PMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

11:48:01 PMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

11:48:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:48:16 PMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

11:48:21 PMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

11:49:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:50:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:51:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:52:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:52:22 PMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

11:53:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:54:09 PMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

11:54:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:55:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:56:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:57:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:58:04 PMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

11:58:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:59:13 PMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

11:59:41 PMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

12:00:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:01:15 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:02:15 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:03:15 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:03:37 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

12:04:15 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:05:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

12:05:15 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:06:16 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:07:15 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:08:16 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:09:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

12:09:16 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:10:16 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:10:49 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

12:11:15 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:12:14 AMrunpod_retryrunpod

podId=jc3a2x03oxa7b0

Network error contacting RunPod: fetch failed

12:13:18 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:14:17 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:14:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

12:15:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:16:20 AMrunpod_retryrunpod

podId=jc3a2x03oxa7b0

Network error contacting RunPod: fetch failed

12:16:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

12:17:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:18:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:19:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:20:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:20:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

12:21:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:22:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

12:22:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:23:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:24:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:25:15 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:25:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

12:26:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:27:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:27:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

12:28:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:29:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:30:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:31:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:31:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

12:32:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:33:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:33:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

12:34:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:35:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:36:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:37:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:37:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

12:38:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:39:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:39:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

12:40:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:41:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:42:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:43:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:43:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

12:44:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:45:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:45:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

12:46:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:47:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:48:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:49:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:49:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

12:50:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:51:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:51:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

12:51:47 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

12:52:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

12:52:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

12:52:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:52:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

12:52:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

12:52:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

12:52:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

12:52:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

12:52:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

12:53:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

12:53:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

12:53:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:53:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

12:53:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

12:54:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

12:54:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

12:54:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:54:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

12:54:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

12:55:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

12:55:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

12:55:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:55:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

12:55:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

12:55:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

12:55:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

12:56:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:56:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

12:56:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

12:56:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

12:56:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

12:57:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:57:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

12:57:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

12:57:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

12:57:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

12:58:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:58:15 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

12:58:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

12:58:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

12:58:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

12:58:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

12:58:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

12:58:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

12:58:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

12:59:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

12:59:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

12:59:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

12:59:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

12:59:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:00:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:00:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:00:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:00:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:00:37 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:01:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:01:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:01:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:01:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:01:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:02:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:02:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:02:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:02:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:02:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

1:03:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

1:03:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

1:03:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:03:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

1:03:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

1:03:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

1:03:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

1:04:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:04:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

1:04:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

1:04:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

1:04:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

1:04:37 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

1:04:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

1:04:47 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

1:04:49 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

1:05:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:05:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

1:05:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

1:05:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

1:05:47 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:06:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:06:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:06:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:06:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:06:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:07:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:07:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:07:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:07:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:07:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:07:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:07:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:08:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:08:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:08:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

1:08:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

1:08:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

1:09:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

1:09:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

1:09:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:09:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

1:09:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

1:10:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

1:10:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

1:10:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

1:10:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:10:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

1:10:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

1:10:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

1:10:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

1:10:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

1:11:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

1:11:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

1:11:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:11:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

1:11:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:11:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:11:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:12:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:12:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:12:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:12:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:12:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:13:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:13:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:13:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:13:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:13:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:14:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:14:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

1:14:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:14:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

1:14:27 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

1:14:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

1:14:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

1:15:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:15:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

1:15:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

1:15:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

1:15:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

1:15:48 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

1:15:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

1:15:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

1:16:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

1:16:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

1:16:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

1:16:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:16:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

1:16:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

1:17:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

1:17:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:17:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:17:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:17:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:17:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:17:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:18:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:18:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:18:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:18:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:18:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:19:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:19:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:19:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:19:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:19:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

1:20:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

1:20:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

1:20:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:20:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

1:20:37 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

1:21:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

1:21:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

1:21:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:21:27 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

1:21:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

1:21:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

1:21:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

1:21:47 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

1:21:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

1:21:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

1:21:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

1:22:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:22:27 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

1:22:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

1:22:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

1:22:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:23:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:23:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:23:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:23:49 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:23:50 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:24:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:24:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:24:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:24:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:24:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:25:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:25:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:25:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:25:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:25:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

1:25:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

1:25:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

1:26:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:26:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

1:26:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

1:26:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

1:26:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

1:27:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

1:27:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

1:27:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

1:27:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:27:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

1:27:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

1:27:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

1:27:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

1:27:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

1:28:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

1:28:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

1:28:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:28:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

1:28:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:28:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:28:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:29:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:29:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:29:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:29:49 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:29:50 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:30:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:30:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:30:15 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:30:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:30:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:31:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:31:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

1:31:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:31:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

1:31:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

1:32:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

1:32:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

1:32:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:32:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

1:32:27 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

1:32:50 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

1:32:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

1:32:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

1:33:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:33:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

1:33:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

1:33:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

1:33:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

1:33:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

1:33:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

1:33:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

1:34:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:34:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

1:34:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:34:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:34:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:35:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:35:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:35:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:35:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:35:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:36:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:36:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:36:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:36:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:36:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:37:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:37:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

1:37:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:37:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

1:37:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

1:38:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

1:38:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

1:38:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:38:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

1:38:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

1:38:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

1:38:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

1:38:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

1:39:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

1:39:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

1:39:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

1:39:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

1:39:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:39:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

1:39:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

1:39:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

1:40:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

1:40:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:40:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:40:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:40:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:41:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:41:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:41:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:41:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:41:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:41:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:41:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:42:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:42:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:42:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:42:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:42:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

1:43:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:43:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

1:43:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

1:43:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

1:43:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

1:44:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:44:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

1:44:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

1:44:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

1:44:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

1:44:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

1:44:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

1:44:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

1:45:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

1:45:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

1:45:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

1:45:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:45:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

1:45:37 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

1:46:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

1:46:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:46:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:46:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:46:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:46:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:46:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:47:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:47:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:47:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:47:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:47:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:48:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:48:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:48:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:48:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:48:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

1:48:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

1:48:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

1:49:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:49:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

1:49:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

1:49:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

1:49:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

1:50:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

1:50:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

1:50:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

1:50:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:50:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

1:50:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

1:50:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

1:50:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

1:50:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

1:51:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

1:51:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

1:51:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:51:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

1:51:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:52:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:52:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:52:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:52:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:52:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:52:47 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:52:48 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:53:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:53:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:53:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:53:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:53:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:54:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:54:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

1:54:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:54:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

1:54:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

1:54:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

1:54:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

1:55:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:55:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

1:55:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

1:55:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

1:55:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

1:55:47 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

1:56:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

1:56:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

1:56:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

1:56:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:56:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

1:56:15 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

1:56:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

1:56:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

1:57:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:57:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

1:57:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

1:57:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

1:57:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

1:58:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

1:58:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

1:58:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:58:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

1:58:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

1:58:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

1:58:59 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

1:59:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

1:59:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

1:59:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

1:59:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

1:59:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:00:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:00:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:00:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:00:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:00:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:01:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:01:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:01:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:01:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:01:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:01:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:01:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:01:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:02:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:02:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:02:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:02:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:02:37 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:02:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

2:03:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

2:03:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

2:03:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:03:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

2:03:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

2:03:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

2:03:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

2:04:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:04:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

2:04:27 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

2:04:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

2:04:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

2:05:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:05:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

2:05:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

2:05:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

2:05:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:06:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:06:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:06:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:06:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:06:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:06:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:06:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:07:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:07:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:07:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:07:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:07:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:07:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:07:37 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:07:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:07:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:08:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:08:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

2:08:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:08:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

2:08:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

2:09:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

2:09:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

2:09:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:09:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

2:09:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

2:10:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

2:10:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

2:10:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:10:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

2:10:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

2:10:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

2:10:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

2:11:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:11:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

2:11:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:11:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:11:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:12:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:12:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:12:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:12:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:12:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:13:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:13:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:13:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:13:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:13:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:13:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:13:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:13:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:13:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:14:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:14:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

2:14:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:14:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

2:14:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

2:15:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

2:15:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

2:15:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:15:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

2:15:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

2:15:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

2:15:59 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

2:16:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:16:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

2:16:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

2:16:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

2:16:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

2:17:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:17:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

2:17:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:17:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:17:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:18:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:18:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:18:15 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:18:37 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:18:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:19:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:19:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:19:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:19:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:19:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:19:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:19:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:19:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:19:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:19:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:19:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

2:20:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:20:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

2:20:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

2:20:50 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

2:20:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

2:21:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:21:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

2:21:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

2:21:47 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

2:21:48 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

2:22:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:22:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

2:22:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

2:22:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

2:22:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

2:23:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

2:23:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:23:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:23:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:23:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:24:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:24:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:24:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:24:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:24:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:24:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:24:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:24:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:25:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:25:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:25:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:25:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:25:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:25:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:25:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:25:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

2:26:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:26:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

2:26:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

2:26:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

2:26:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

2:27:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

2:27:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

2:27:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:27:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

2:27:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

2:28:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

2:28:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

2:28:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:28:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

2:28:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

2:29:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

2:29:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:29:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:29:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:29:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:29:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:29:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:30:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:30:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:30:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:30:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:30:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:30:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:30:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:31:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:31:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:31:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:31:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:31:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:31:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:31:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

2:32:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

2:32:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

2:32:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:32:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

2:32:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

2:33:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

2:33:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

2:33:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:33:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

2:33:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

2:33:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

2:33:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

2:34:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:34:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

2:34:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

2:34:48 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

2:34:49 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:35:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:35:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:35:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:35:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:35:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:36:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:36:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:36:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:36:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:36:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:36:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:36:49 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:36:50 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:36:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:36:59 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:37:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:37:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:37:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:37:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

2:37:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

2:37:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

2:38:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:38:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

2:38:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

2:38:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

2:38:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

2:39:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

2:39:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

2:39:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:39:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

2:39:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

2:39:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

2:39:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

2:40:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:40:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

2:40:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:40:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:40:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:41:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:41:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:41:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:41:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:41:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:41:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:41:59 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:42:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:42:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:42:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:42:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:42:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:42:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:42:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:42:49 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:42:50 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

2:43:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

2:43:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

2:43:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:43:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

2:43:37 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

2:43:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

2:43:59 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

2:44:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:44:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

2:44:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

2:44:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

2:44:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

2:45:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

2:45:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

2:45:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:45:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

2:45:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:45:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:45:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:46:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:46:18 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:46:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:46:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:46:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:47:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:47:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:47:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:47:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:47:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:47:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:47:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:47:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:47:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:48:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:48:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

2:48:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:48:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

2:48:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

2:48:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

2:48:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

2:49:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:49:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

2:49:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

2:49:49 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

2:49:49 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

2:50:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:50:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

2:50:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

2:50:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

2:50:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

2:51:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

2:51:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:51:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:51:37 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:51:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:52:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:52:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:52:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:52:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:52:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:52:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:52:59 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:53:01 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:53:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:53:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:53:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:53:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:53:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:53:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:53:48 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:53:49 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

2:54:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:54:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

2:54:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

2:54:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

2:54:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

2:55:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:55:15 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

2:55:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

2:55:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

2:55:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

2:56:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

2:56:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

2:56:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:56:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

2:56:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

2:57:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

2:57:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

2:57:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:57:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

2:57:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

2:58:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

2:58:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

2:58:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:58:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

2:58:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

2:58:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

2:58:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

2:58:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

2:59:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

2:59:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

2:59:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

2:59:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

2:59:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

2:59:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

2:59:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

2:59:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

3:00:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:00:19 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

3:00:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

3:00:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

3:00:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

3:01:12 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

3:01:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

3:01:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:01:40 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

3:01:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

3:02:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

3:02:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

3:02:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:02:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

3:02:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

3:03:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

3:03:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

3:03:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:03:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

3:03:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

3:04:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

3:04:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

3:04:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:04:27 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

3:04:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

3:04:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

3:04:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

3:04:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

3:05:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

3:05:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

3:05:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

3:05:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:05:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

3:05:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

3:05:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

3:05:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

3:06:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:06:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

3:06:15 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

3:06:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

3:06:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

3:07:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

3:07:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

3:07:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:07:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

3:07:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

3:08:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

3:08:00 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

3:08:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:08:27 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

3:08:28 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

3:08:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

3:08:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

3:09:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:09:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

3:09:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

3:09:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

3:09:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

3:10:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:10:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

3:10:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

3:10:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

3:10:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

3:10:48 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

3:10:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

3:10:56 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

3:11:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

3:11:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

3:11:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

3:11:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:11:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

3:11:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

3:12:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

3:12:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

3:12:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:12:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

3:12:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

3:12:57 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

3:12:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

3:13:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:13:26 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

3:13:27 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

3:13:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

3:13:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

3:14:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:14:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

3:14:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

3:14:50 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

3:14:50 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

3:15:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:15:15 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

3:15:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

3:15:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

3:15:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

3:16:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

3:16:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

3:16:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:16:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

3:16:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

3:16:36 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

3:16:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

3:16:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

3:16:48 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

3:16:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

3:16:53 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

3:17:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:17:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

3:17:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

3:17:51 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

3:17:52 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

3:18:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:18:16 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

3:18:17 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

3:18:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

3:18:43 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

3:19:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

3:19:11 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

3:19:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:19:38 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

3:19:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

3:20:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

3:20:07 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

3:20:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:20:34 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

3:20:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

3:21:02 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

3:21:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

3:21:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:21:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

3:21:31 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

3:21:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

3:21:58 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

3:22:14 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:22:22 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

3:22:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

3:22:25 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

3:22:32 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

3:22:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

3:22:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

3:22:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

3:22:44 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

3:23:13 AMrunpod_statusrunpod

status=running · podId=jc3a2x03oxa7b0

RUNNING

3:23:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

16% · evaled T_2_seed42

3:23:15 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

19% · trained C_2_seed42

3:23:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

22% · evaled C_2_seed42

3:23:39 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

25% · trained T_3_seed42

3:24:09 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

28% · evaled T_3_seed42

3:24:10 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

31% · trained C_3_seed42

3:24:12 AMauto_recovery_queuedagent

0164ffaa-cc78-4944-8735-dbcc4578d0dd

3:24:12 AMrunpod_blockedrunpod

podId=jc3a2x03oxa7b0

HTTP 503 from RunPod: {"message":"Service Unavailable"}

3:24:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

34% · evaled C_3_seed42

3:24:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

37% · trained T_3_seed137

3:25:03 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

40% · evaled T_3_seed137

3:25:04 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

43% · trained C_3_seed137

3:25:29 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

46% · evaled C_3_seed137

3:25:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

49% · trained T_3_ablate_seed42

3:25:54 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

52% · evaled T_3_ablate_seed42

3:25:55 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

55% · trained T_4_seed42

3:26:20 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

58% · evaled T_4_seed42

3:26:21 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

61% · trained C_4_seed42

3:26:45 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

64% · evaled C_4_seed42

3:26:46 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

67% · trained T_5_seed42

3:27:13 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

70% · evaled T_5_seed42

3:27:14 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

73% · trained C_5_seed42

3:27:41 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

76% · evaled C_5_seed42

3:27:42 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

82% · donor fidelity csv written

3:28:05 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

90% · stats artifacts written, rendering figures

3:28:06 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

98% · figures rendered, finalizing run manifest

3:28:08 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

100% · experiment completed

3:28:23 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

5% · bootstrap complete on branch issue-366

3:28:24 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

6% · issue 366 bootstrap done, resolving markers

3:28:30 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

8% · markers resolved, building word pool

3:28:33 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

10% · manifest written, beginning per-adapter loop (11 adapters)

3:28:35 AMrunpod_progressrunpod

podId=jc3a2x03oxa7b0

13% · trained T_2_seed42

3:30:04 AMrunpod_recoveredrunpod

podId=jc3a2x03oxa7b0

Recovery resolved (0164ffaa). 503 was a transient RunPod GraphQL hiccup during a runpod_status poll at 03:24:12; dispatch itself succeeded at 23:46:24 and the pod ran the experiment to 100% completion 27 times.

Root cause of the cycling: RunPod container auto-restart-on-exit. run_366.sh `exec`s python and exits cleanly (exit 0) when the per-adapter loop finishes; RunPod then relaunches the dockerArgs, the wrapper re-bootstraps (`bootstrap complete on branch issue-366`), and the script runs the deterministic 11-adapter sweep again. Twenty-eight bootstrap cycles burned ~$10 of H100 time on redundant work.

Action taken: stopped pod jc3a2x03oxa7b0 via pod-tool stop (desiredStatus=EXITED). Network volume preserved — artifacts/366/{adapters, completions, csvs, figures, run_manifest.json} remain on /workspace/explore-persona-space/artifacts/366/. To collect them, resume the pod and run the uploader subagent against this run; then terminate.

Dispatch itself was clean. The cycling is a structural issue: completion-on-exit on one-shot experiment pods. Orthogonal follow-up: have the wrapper end with `sleep infinity` after the user-cmd exits, or have the runner terminate/stop the pod on the 100% progress post.

3:39:12 AMstale_recoveredagent

marked failed after stale running timeout

Discussion

Comments

No comments yet.