Engineering archive

These notes describe work at the date shown. See the usage reference for current behavior.

Pool run record, 2026-09-10: polish tasks through the pool

Date: 2026-09-10, 00:25–01:30. Laptop CLI and worker helpers at main e8c1bb2 when the run started (newer builds were installed as fixes landed during the run). Three Mac minis: mini-1 (Cursor, env profile agents), mini-2 and mini-3 (Codex, gpt-5.6-luna, effort max). The project was a clean checkout of main. Task ids are shortened; prompts, paths and secrets are omitted, as the acceptance runbook requires.

What was run

One batch, worker task batch polish-2026-09-10.toml --name polish-2026-09-10 --max-parallel 3 --wait, with per-task worker, agent and env_profile overrides, then one retry submitted with --project from another directory.

Task	Agent, worker	Outcome	Landed as
4897403d render mac-worker's turn-framing events in task logs	Cursor, mini-1	done in ~10 min; matched the prompt, including the privacy rule and tests	f309f96 (squash, message rewritten)
e141be47 sync operator docs with the previous day's changes	Codex, mini-2	done in ~12 min; docs only, links verified	ba32d33 (squash)
a9287cd0 harden two timing-window lib tests	Codex, mini-3	failed in 2 min: Codex auth (finding 3)	—
af449fda the same task, retried	Codex, mini-2	done in ~22 min; hardened one of the two tests; the runner then lost its queue row (finding 8)	1c1c378 (squash)

The batch finished in 718 s with exit 1 because of the failed task. task result gave the summary, the files and the branch for each finished task; task fetch brought each branch to the laptop as refs/remotes/mac-worker/<worker>/task/<id> whose tip is a synthetic commit "mac-worker: uncommitted changes after turn …" on top of the base. Every branch was gated on the laptop (fmt, clippy, the affected suites, the lib suite) and squash-merged with a proper message.

Findings

1. task list --state active failed with TASK_VIEW_MISSING_TASK: task error while the run was active. TaskClient::list filtered task records by state before projecting runs, so a run referencing a filtered-out task broke the projection. Fixed the same night in 308e52e: the projection runs over all records and the filter applies to the projected rows.

2. --run did not accept the batch name. task batch --name polish-2026-09-10 printed a name, but task list --run polish-2026-09-10 and task wait --run … demanded a UUID. Fixed in 308e52e: --run resolves a UUID first, then an exact stored name.

3. One ChatGPT login shared across minis burns Codex refresh tokens. The Codex turn on mini-3 died at once with Your access token could not be refreshed because your refresh token was already used. Please log out and sign in again. The same auth.json had been copied to every mini; refresh tokens are single-use, so machines invalidate each other. codex login status still printed Logged in using ChatGPT on all three, so the facts kept advertising Codex on mini-3 — the same shape as the Cursor "login unverified" problem fixed the day before. Actions: a separate codex login per mini (operator), and a mac-worker follow-up, landed the next day as d564966, 749c1b9 and 327710c: each adapter lists its fixed authentication-failure phrases, a failed turn whose stream tails match them records a private incident beside facts.json (agent, profile name, a fixed reason, the time; never the log), the facts then show unknown (auth failed in a turn at <UTC minute>) for that agent, the capability disappears until a later turn of that agent succeeds, 24 hours pass, Codex's auth.json is newer than the incident, or the operator runs worker workers --refresh --clear-auth-incidents; the failed turn's outcome reads agent authentication failed, and the cached host probe applies the overlay without waiting for a refresh.

4. A pinned submit to a busy worker fails fast with CAPACITY_BUSY: no eligible worker currently has an available heavy slot (the reason is public since the day before). submit has no wait-for-capacity flag; only batch runs park. By design, noted for the docs.

5. Rendered logs were readable mid-turn (event: tool_call/completed ×2, event: item.started, command lines); after f309f96 the framing lines read accepted by mini-2 and the terminal line is silent.

6. Local main drifts. Pushing main swept two commits another session had left unpushed on the shared local branch (915227f, dc5f937), for the third time in two days. Sessions should push their own commits or work on branches.

7. Idle pool facts age out. Seen at the start of the run: after 15 minutes without a refresh the dashboard showed herdr: unknown on two workers. Fixed in 77bc222: the dashboard refreshes stale facts once per TTL in the background and renders stale facts with their age.

8. A completed turn's queue row stayed dispatching and nothing could clear it. The retry turn (af449fda) ended done at 00:49: the worker's status has the turn succeeded, the runner log's checkpoint records completion, and the runner cleared its identity. But its queue row was never removed. Consequences: task wait (which now waits for quiescence) ran to its 50-minute timeout, task close refuses with TASK_BUSY: task turn is being dispatched, and every task reconcile adopts the row to a fresh runner that dies within a second without a diagnostic, leaving the row under a dead pid. The runner writes its early-exit diagnostic only when the log is still empty, so the reason both runners failed after acceptance is lost. Most likely the original runner lost the row to a wait-driven reconcile (wait polls reconcile_runners every 100 ms) that mis-judged it dead under load, then failed remove_task_turn_after_terminal with an owner mismatch. Fixed the next day in 057f1ab and e636ddb: a replacement runner resumes an accepted journal from its committed offsets instead of submitting again, a worker job that is gone after acceptance ends the journal as an explicit LOG_DRAIN_UNAVAILABLE completion (never a fabricated turn_terminal), one finalizer shared by reconcile and the runner publishes the local failure, imports this turn's result and releases the base pin only after that import before retiring the row, task wait, task logs -f and the dashboard show that state instead of hanging, and the early-exit diagnostic is appended even when the log has content, prefixed exited after acceptance:. The first draft, which completed the journal from the task record, was withdrawn during review because the record is written before the result is fetched. Note for operators: the private state directories reject unexpected files — a backup copy placed next to queue/state.json made every task command fail with IO: I/O error until it was moved out.

9. Worker toolchains lag the laptop. The minis run rustc 1.94.1 (March 2026) while the laptop runs 1.98.0; the Codex turn hit a clippy lint on the old toolchain that the laptop's clippy does not raise. Keep rustup update on the minis in the setup checklist.

10. The Codex sandbox on a worker cannot run the whole lib suite. One test that manipulates process groups failed inside the sandbox with a permission error; the agent reported it correctly and continued. Rust tasks that need the full lib suite should ask for a narrower gate or expect this.

11. The retry's hardening was partial. The rooted_fs cleanup-guard test now waits on its condition with a ten-second deadline (three lib runs alone: green). Two lib suites started at the same time on the laptop still fail one or two tests (runner_log pending-crash windows among them), so the parallel-gate flakiness stays on the backlog.

12. A named pipe in a job's tmp blocks the worker's cleanup forever, and that blocked the laptop too. Found while closing finding 8 with the new reconcile: its post-acceptance diagnostic read exited after acceptance: HOST_IO. On the worker, job 5bae1707 was succeeded with cleanup_error_code: MUTABLE_CLEANUP_FAILED and supervisor.log said I/O error: Invalid argument (os error 22): the agent's shell tool had left ~841 .tmpXXXXXX directories under the job's tmp, one holding command-ready.fifo, and the durable cleanup walk in rooted_fs rejected any entry that was not a directory, regular file or symlink. The lease was never released (slot: busy on that worker for fifteen hours), and every status or log read from the laptop re-ran the failing cleanup and answered HOST_IO, so the accepted turn could not be drained. Fixed in 14beaec: cleanup unlinks FIFOs, sockets and device nodes without opening them, sealing skips them like symlinks, and terminal status and log reads succeed even while a deferred cleanup still fails; cleanup_error_code and the busy slot remain the operator's signal, and reconcile/resolution still report the error.

Second run, 17:34–18:00: unpinned batch on main f2843df

worker task batch live-2026-09-10b.toml --name live-2026-09-10b --max-parallel 3 --wait, three tasks without a worker pin: two Codex (a docs/usage.md update and a runner_log test hardening) and one Cursor with env_profile = "agents" told to ask one question before writing. The scheduler placed them on mini-3, mini-1 and mini-2 within seconds. All three finished done after one task say each, were fetched and landed as 7295743, e295a9d and c4c6408.

13. A needs_input result with a real question was recorded as unknown. The Cursor turn ended with prose followed by the JSON object; extract_last_json_object tried a parse at every { and kept the innermost one, the question object, so any non-empty questions array was unrecognisable. Fixed in 303472d: after a successful parse the scanner skips to that object's end, so only top-level objects are candidates.
14. A task say follow-up inside a run at full parallelism parked itself. run_has_capacity counted every Active task of the run, including the task that owns the follow-up row, so two siblings plus itself hit max_parallel = 3 and the row waited with RUN_MAX_PARALLEL until a sibling finished. Fixed in c914e17: rows map back to their tasks, each task counts once, and the follow-up's own task is excluded.
15. task batch --wait aborted with QUEUE_JOB_CONFLICT (exit 70) two minutes in while every task kept running. queue_entry_for_task_turn maps only turn directories; a pending turn present in the status but without a directory looked row-less to both recovery phases of reconcile_runners, so the second phase enqueued the row the first had just published. Fixed in d282692: enqueue_if_absent returns the existing row, both phases share one queue snapshot, a turn that is already terminal is never re-enqueued, and wait survives a transient conflict. Left for later: start_runner still uses the directory-only lookup.
16. The Codex sandbox cannot commit in the task workspace. Both Codex tasks did their work and then reported blocked or needs_input because .git/index.lock is not writable; mac-worker imports the working tree as a synthetic commit at the end of the turn, so the work was not lost, but a prompt that demands a commit turns a done task into a question. Cursor's sandbox allowed the commit. Documented in docs/usage.md in the same change as this record.
17. Replacement-runner churn from task wait (seen while closing finding 8): wait polled reconcile every 100 ms and each poll restarted a runner that died after acceptance with the same error. Fixed in 00ce0d1: a per-row restart budget with backoff (2 s, 10 s, 30 s, 60 s), parking with RUNNER_REPEATED_FAILURE:<code> after five identical failures, WAIT_BLOCKED from wait, and worker task reconcile as the operator's reset.
18. Cosmetic residue of finding 12: the job's status.json kept cleanup_error_code: MUTABLE_CLEANUP_FAILED after the retried cleanup succeeded. Fixed in 68b0810: a terminal status may clear the code once, through the same fenced write that records it.

Not exercised

• Automatic (unpinned) scheduling: every task was pinned so each worker got exactly one.
• task say follow-ups: no task ended needs_input.
• Publication to an origin (publish = ["fetch"] only).