Engineering archive

These notes describe work at the date shown. See the usage reference for current behavior.

mac-worker v1 design

• Date: 2026-08-25
• Status: approved for implementation
• Repository: mac-worker
• User-facing command: worker

1. Decision

Build a small Rust command-line application that executes trusted, non-interactive development jobs on three macOS workers over SSH.

The MacBook remains the only source of truth for source code. Every accepted job receives an immutable snapshot of one local Git worktree. A worker never synchronizes source changes back. The same compiled worker binary runs as the local client and as an SSH-invoked host helper; no persistent control-plane daemon, Codex installation, GitLab Runner, Kubernetes cluster, or inbound network service is required.

Version 1 deliberately handles batch work: builds, tests, linters, static analysis, bounded scripts, and explicitly configured Docker jobs. Long-lived development servers, continuous source synchronization, automatic port brokerage, GUI workflows, code signing, and untrusted code are separate future concerns.

2. Context

The development machine runs several Codex agents in parallel. Editing, Git operations, agent context, IDE processes, and lightweight commands should stay local, while sustained CPU- and memory-heavy commands should use three Mac mini M4 machines with 16 GB RAM each.

The local projects include Node with npm, Yarn, and pnpm; Rails and native gems; .NET; Python; Go; Swift; browser tests; Docker; and Compose stacks with multiple services. Several repositories use many linked Git worktrees concurrently. The design must therefore identify a worktree, not merely a repository, and must not assume one language or package manager.

3. Goals

• Run a command from any local Git worktree through one stable interface.
• Include unstaged and staged tracked changes in the exact filesystem snapshot.
• Detect untracked inputs and require an explicit inclusion policy instead of silently omitting them.
• Keep concurrent worktrees and jobs isolated from one another.
• Select a compatible, available worker while preferring an existing warm cache.
• Allow up to one heavy job per 16 GB worker in v1.
• Preserve the job across client disconnects and provide reconnectable logs and status.
• Report whether the command exited, was signalled, was cancelled, or failed before execution.
• Return only explicitly declared artifacts.
• Keep secrets out of snapshots, metadata, and all output generated by mac-worker itself.
• Make all state inspectable and removable without broad or destructive cleanup commands.

4. Non-goals

• Transparent replacement of the local shell.
• Editing files on a worker or reverse synchronization of source code.
• Simultaneous writers in the same local worktree.
• Automatic installation or upgrading of project runtimes.
• Running untrusted pull requests or dependencies in a security sandbox.
• Sharing production credentials with workers.
• Git history-, tag-, index-, LFS-, submodule-, or custom-filter-dependent commands in v1.
• Multiple concurrent heavy jobs on one worker.
• Arbitrary long-lived Docker Compose environments, live reload, or automatic port rewriting.
• GUI tests, iOS simulators, Keychain access, code signing, or attached hardware.
• Distributed content-addressed storage, autoscaling, high availability, or multi-user quotas.

5. Trust and isolation model

All v1 jobs are trusted code owned by the user. A unique job directory prevents accidental filesystem collisions but is not a security boundary. Jobs can inspect anything available to the configured macOS account, including its files, processes, shared caches, and non-interactive credentials. Docker access grants broad access to the Docker VM.

Each worker may use the existing macOS account selected by the operator. A dedicated non-admin account is optional hardening, not a v1 prerequisite. The supported current-account deployment requires:

• no privilege elevation by worker and no reliance on passwordless sudo;
• no SSH agent forwarding;
• no production or cloud-administrator credentials exposed to remote jobs;
• a pinned SSH host key and a worker-specific client key;
• only read-only, project-specific package-registry credentials when unavoidable;
• explicit acceptance that trusted jobs share the selected account's access and state.

Untrusted jobs require a VM or equivalent isolation layer and are outside v1.

6. User experience

The main commands are:

worker setup HOST...
worker doctor [--project PATH]
worker run [options] -- COMMAND [ARG...]
worker status [JOB_ID]
worker logs [-f] JOB_ID
worker cancel JOB_ID
worker fetch JOB_ID
worker gc [--apply]
worker workers

Examples:

worker run -- npm test
worker run -- bundle exec rspec spec/models/user_spec.rb
worker run -- dotnet test
worker run --timeout 30m --artifact coverage/** -- npm run test:coverage
worker run --shell 'npm run build && npm test'

worker run -- ... passes an argv array without a second shell-parsing step. Shell operators, pipes, redirects, and expansion work only through the explicit --shell form.

The current directory may be below the worktree root. The same relative working directory is used remotely. If the terminal disconnects or the user presses Ctrl-C while following logs, the job continues and the CLI prints its job ID. Cancellation is always explicit through worker cancel.

worker status without a job ID lists queued, active, and recent jobs known to the local client. With a job ID it queries the assigned worker for authoritative detail. worker workers shows capabilities, protocol version, health, capacity, and current lease for every configured host.

If no compatible worker is available, the command waits in a FIFO queue by default. --no-wait returns immediately with a capacity error. The CLI never silently falls back to local execution and never automatically retries an accepted command.

worker setup installs or updates the current worker binary for the configured remote account and creates owned state directories. It does not create accounts, modify FileVault, enable SSH, install project runtimes, install a persistent daemon, or provision credentials.

7. Architecture

The system has five components within one Rust codebase:

1. Project inspector — identifies the Git worktree, current directory, HEAD metadata, selected files, required capabilities, and project configuration.
2. Snapshot builder — materializes and verifies an immutable local filesystem snapshot.
3. Scheduler — maintains the local FIFO queue, queries worker health, obtains an atomic remote lease, and applies sticky affinity as a tie-breaker.
4. Transport client — invokes system OpenSSH and rsync using the user's existing SSH configuration.
5. Host helper — runs through SSH using the same binary, validates uploads, owns durable job state, supervises processes, streams logs, validates artifacts, and performs targeted cleanup.

There is no always-running central service. Local CLI processes coordinate through locked state files on the MacBook. A detached host-helper process supervises each accepted remote job.

7.1 End-to-end run flow

1. The client inspects the current worktree and validates configuration and unsupported Git features.
2. It creates a job ID, joins the local FIFO queue, and materializes an immutable local snapshot.
3. The scheduler probes compatible workers and atomically acquires one remote lease.
4. The client uploads the snapshot and manifest into the leased worker's unique incoming directory.
5. The host helper verifies the manifest, atomically promotes it into the immutable snapshot cache, and creates a separate copy-on-write job workspace.
6. The host durably records accepted, starts a detached supervisor in that workspace, and acknowledges the same job ID.
7. The supervisor executes the command and records logs and typed state transitions.
8. The client follows logs; it may disconnect and reconnect without affecting execution.
9. On a terminal state, the supervisor validates declared artifacts, performs targeted cleanup, and releases the lease.
10. The client returns the command result and can fetch the artifact manifest and files separately.

If a step before durable acceptance fails, no command has run. If connectivity is lost around acceptance, the client queries the existing job ID rather than resubmitting it.

8. Identifiers and paths

The client computes three identifiers:

• project_id: SHA-256 of the normalized Git origin URL with credentials removed; when no origin exists, use the canonical Git common-directory path.
• worktree_id: SHA-256 of the canonical local worktree root.
• job_id: a cryptographically random 128-bit identifier generated before submission.

The configured XDG_CONFIG_HOME, XDG_STATE_HOME, XDG_CACHE_HOME, and XDG_DATA_HOME are honored. Defaults are:

~/.config/mac-worker/config.toml
~/.local/state/mac-worker/
~/.cache/mac-worker/
~/.local/share/mac-worker/

Each worker stores owned data below ~/.local/share/mac-worker/:

incoming/<job_id>/
jobs/<project_id>/<worktree_id>/<job_id>/
snapshots/<project_id>/<worktree_id>/<manifest_digest>/
leases/

No cleanup operation may address paths outside the resolved mac-worker data root.

9. Snapshot contract

9.1 Included inputs

The snapshot contains:

• every tracked file in the worktree, using its current filesystem bytes rather than its index contents;
• tracked deletions;
• executable-mode information;
• symlinks as symlinks, never followed;
• untracked, non-ignored files matching snapshot.include_untracked in .worker.toml or a CLI --include pattern.

Ignored files are excluded unless a precise path pattern is explicitly included. .git, dependency directories, build outputs, Docker data, editor state, and local caches are never copied as directories from the MacBook.

If uncovered non-ignored untracked files exist, worker run stops with a list of relative paths and explains how to include or exclude them. It must never run a potentially incomplete snapshot silently.

Sensitive path patterns such as .env, local .env.* files other than documented examples, .npmrc, .pypirc, SSH keys, cloud credential directories, and keychain exports cause a preflight failure even when tracked. A project may opt in to a precise path through configuration, but the CLI shows an explicit warning and never records its contents. Conventional example files such as .env.example and .env.sample are not treated as secrets unless explicitly excluded.

9.2 Local materialization

The snapshot builder enumerates files with NUL-safe Git commands, copies them into a unique staging directory, and preserves regular-file bytes, executable modes, empty directories needed by declared inputs, and symlinks without dereferencing them.

On APFS it uses copy-on-write cloning where available; otherwise it copies bytes. It then computes a manifest containing normalized relative path, entry type, mode, size, and SHA-256 digest. It re-enumerates and re-hashes the selected source entries after staging. If the selected set or any digest changed during capture, the snapshot is discarded and the command returns SNAPSHOT_CHANGED. This prevents a job from accepting a hybrid tree while an agent is editing it.

The completed staging directory becomes read-only for the remainder of submission. It contains no .git directory. The manifest records the original HEAD object ID, branch name when present, dirty state, relative working directory, and project/worktree IDs. These values are also exposed to the command as non-secret MAC_WORKER_* environment variables.

The remote workspace contains no .git directory. Commands that invoke Git at runtime therefore fail normally. worker doctor detects static indicators such as submodules, LFS configuration, and custom filters, but it cannot prove that an arbitrary project script does not call Git. Projects whose build depends on Git history, tags, the original index, submodules, LFS smudge behavior, or custom filters are unsupported until a later snapshot capability is designed.

9.3 Remote transfer

The client uploads the immutable staging directory with rsync over SSH to a unique remote incoming/<job_id> directory. An optional previous immutable snapshot may be supplied to rsync as --copy-dest, so unchanged files are copied locally on the worker instead of crossing the network. Hard-link reuse is forbidden because a writable job could otherwise mutate another snapshot through the shared inode.

The host helper recomputes and compares the manifest before accepting the job. A partial or invalid upload remains under incoming and is never executable. After successful verification, the host atomically renames the directory into an immutable, read-only snapshot path keyed by manifest digest.

For execution, the host creates jobs/<project_id>/<worktree_id>/<job_id>/workspace as an APFS copy-on-write clone of that snapshot. If cloning is unavailable, it performs a byte copy. The command may modify its own workspace without changing the cached snapshot or another job. No upload or command writes into a running job's workspace or an immutable snapshot.

10. Worker inventory and capabilities

The local config lists SSH aliases and declared capabilities:

[[workers]]
name = "mini-1"
ssh = "mini1"
slots = 1
capabilities = ["darwin-arm64", "node", "ruby", "docker"]

The project may declare requirements in .worker.toml:

version = 1
requires = ["darwin-arm64", "node"]
resource_class = "heavy"
timeout = "30m"

[snapshot]
include_untracked = ["fixtures/generated/**"]

[artifacts]
include = ["coverage/**", "test-results/**"]
max_total_bytes = 536870912

Configuration is optional for commands whose requirements are already satisfied. worker doctor detects common version files and reports missing Node, Ruby, Python, Go, .NET, Swift, package-manager, browser, and Docker capabilities. It never installs them automatically.

Before admission, the host reports architecture, macOS build, available capabilities, free disk, memory pressure, swap usage, current lease, and the installed worker protocol version. A protocol mismatch or unmet capability excludes that host.

11. Scheduling and leases

All v1 workers expose one heavy slot. The local scheduler serializes queue updates with an OS file lock and considers jobs in FIFO order. Queue entries record the owning local process and creation time; abandoned entries are removed under the same lock before scheduling. Among compatible idle hosts it prefers, in order:

1. the previous healthy host for the same project/worktree;
2. the previous healthy host for the same project;
3. the host with the most free memory, then free disk;
4. lexical worker name for deterministic tie-breaking.

Sticky affinity never overrides health, capabilities, or capacity.

Admission requires an atomic remote lease created by the host helper. The lease contains job ID, client identity, creation time, expiry, and requested resource class. If two clients race, only one lease succeeds. A lease is released only after the job reaches a terminal state and targeted cleanup completes. Expired leases are reconciled against live process metadata before removal.

Workers refuse new jobs when free space is below the greater of 50 GB or 20% of the volume, when macOS reports critical memory pressure, or when swap exceeds the configured admission limit of 2 GB.

12. Job lifecycle

Remote state transitions are:

uploading -> verified -> accepted -> running
                                  -> succeeded
                                  -> failed
                                  -> cancelled
                                  -> timed_out
                                  -> lost

accepted is durable: the host atomically writes meta.json and status.json before acknowledging submission. Repeating submission with the same job ID returns the existing state and never executes the command twice.

The host helper launches the command in a new process group with a clean per-job HOME, declared environment variables, and redirected append-only stdout/stderr log files. It records PID, start time, command argv or explicit shell string, working directory, source digest, environment fingerprint, and timeout. The command does not receive an interactive PTY by default.

Commands that intentionally daemonize or escape their process group are unsupported in v1. Long-lived processes require the separately designed worker up lifecycle.

On completion the supervisor atomically records exit code or terminating signal and moves the job to a terminal state. On timeout or explicit cancellation it sends TERM to the process group, waits ten seconds, then sends KILL. Docker cleanup, when configured, is separate and targeted to the job's declared Compose project.

After a worker reboot, the next host-helper operation runs reconciliation before serving its requested action. Reconciliation marks jobs whose recorded supervisors no longer exist as lost, performs only job-owned cleanup, and releases their leases. The application does not promise availability before the machine can accept SSH, disable FileVault, configure automatic login, or alter security settings.

13. Logs, disconnects, and status

The remote append-only log and durable status files, not tmux, are authoritative. worker logs -f reconnects by job ID and resumes from a byte offset. Log follow failure does not change job state.

If SSH disconnects before acceptance is confirmed, the client queries the same job ID until it learns whether the host accepted it. It never submits a second job automatically. If the host cannot be reached, the local state remains unknown_remote and the CLI prints recovery commands.

worker status distinguishes command failure from infrastructure failure and shows source digest, worker, timestamps, process state, exit code or signal, and artifact status. Secret values and complete environment values are never recorded.

14. Artifacts

Artifacts are disabled unless declared by CLI or .worker.toml. The host resolves every artifact path relative to the job workspace and rejects absolute paths, parent traversal, devices, sockets, and symlinks. It enforces configured file-count and total-byte limits, creates a manifest with SHA-256 digests, and downloads into a separate local result directory under mac-worker state.

Command status and artifact-transfer status are independent. A successful command remains successful if artifact transfer fails, while the CLI reports the artifact error explicitly. Source directories are never reverse-rsynced.

15. Environment and secrets

By default the command receives a minimal environment: safe locale variables, a controlled PATH, HOME, TMPDIR, and MAC_WORKER_* metadata. Project variables must be named explicitly.

The per-job home is isolated, but project configuration may opt into named, mac-worker-owned package caches. The host maps cache variables for npm/pnpm/Yarn, Bundler, NuGet, Python, Go, browser binaries, or BuildKit only when the runtime and architecture fingerprint match. A cache is a performance optimization: every supported command must remain correct when it is empty. Job workspaces, mutable build outputs, dependency directories such as node_modules, and virtual environments are never shared between jobs.

Package-registry or service credentials are referenced by a named env_profile but provisioned separately on an individual worker with mode 0600. V1 permits only read-only, project-scoped credentials. The tool does not copy local secret files, forward an SSH agent, perform docker login, or accept production/high-value credentials.

Logs and diagnostic output generated by mac-worker may list environment variable names but never their values. The project configuration must not contain secret values. A child command can still print any value it can access; preventing application-owned secret logging remains the project's responsibility.

16. Docker policy

Native batch commands are the default. A job that requires Docker must declare the docker capability and a Docker profile in .worker.toml. Because every worker has one slot, the Docker job is exclusive.

The profile supplies a unique COMPOSE_PROJECT_NAME derived from the job ID and declares Compose files and the targeted cleanup command. The preflight rejects known global collision risks such as explicit container_name, undeclared host-path mounts, restart: always, and fixed host ports already in use. Cleanup operates only on resources carrying the job's project name or labels. Global docker system prune, removal of unrelated volumes, and broad process killing are forbidden.

worker up, continuous synchronization, automatic port allocation, and persistent Compose leases are not part of v1.

17. Cleanup and disk management

worker gc previews every candidate with reason and size. worker gc --apply removes only completed mac-worker-owned job directories and incomplete incoming directories older than their TTL. It never follows symlinks and validates every target remains below the mac-worker data root.

Default retention is seven days for completed job metadata, logs, workspaces, and fetched-artifact manifests; incomplete uploads expire after one hour. The newest verified snapshot for each worktree may be retained as an rsync copy destination while disk admission thresholds remain satisfied.

Automatic admission-triggered GC may delete only expired mac-worker job data. Package-manager caches and Docker caches require explicit, targeted maintenance and are never globally pruned by mac-worker.

18. Error model

Errors are grouped into:

• project: invalid worktree, unsupported Git feature, uncovered untracked input, or invalid config;
• snapshot: source changed, unsafe path, size limit, or manifest mismatch;
• capacity: no compatible host, lease conflict, memory pressure, swap, or disk threshold;
• transport: SSH or rsync failure before acceptance;
• infrastructure: protocol mismatch, host-helper failure, reboot, or lost supervisor;
• command: a started command exited non-zero or by signal;
• artifact: artifact validation or transfer failed after command completion.

When a command starts, the local CLI returns its exit status when representable. Pre-execution and infrastructure failures use a reserved non-zero CLI status and a stable machine-readable error code. The durable job record remains the authoritative distinction.

19. Observability

Every accepted job records:

• job, project, and worktree IDs;
• source manifest digest and original HEAD;
• worker and protocol version;
• macOS, architecture, and declared runtime fingerprint;
• sanitized argv or explicit shell mode;
• queue, upload, start, end, and cleanup timestamps;
• transferred bytes and file count;
• terminal state, exit code or signal;
• peak host memory-pressure class and swap sample;
• artifact manifest status.

No telemetry leaves the local network in v1. Human-readable output and JSON output derive from the same typed records.

19.1 Local operational dashboard

After the three-worker scheduler is complete, phase 4.5 adds an optional read-only dashboard started by worker dashboard. It binds only to loopback on the MacBook, embeds its assets in the existing binary, and derives worker, queue, job, and log views from the same typed state used by the CLI. It is not a daemon, an authority for job state, or a service installed on the Mac minis.

The detailed scope, security boundary, data flow, and acceptance criteria are defined in the local dashboard design.

20. Verification strategy

20.1 Automated tests

• Unit tests for path normalization, identity derivation, configuration, capability matching, scheduler ordering, state transitions, command quoting, artifact validation, and cleanup boundaries.
• Property and mutation tests for manifests, symlinks, executable modes, binary files, unusual Unicode names, spaces, newlines, deletions, and concurrent source changes.
• Local integration tests using temporary directories and a host-helper subprocess, with SSH and rsync behind injectable command interfaces.
• Lifecycle tests for exit codes, signals, timeout, cancellation, duplicate submission, client disconnect, corrupt upload, and reboot reconciliation.
• Security tests with planted secret paths, traversal attempts, symlink escapes, oversized logs, oversized artifacts, and commands containing shell metacharacters.

20.2 Acceptance matrix

The first external validation covers representative repositories rather than one designated project:

Class	Workload
Node/Yarn frontend	install, typecheck, unit test, build, cold/warm cache
npm workspace	test and Postgres integration from two worktrees
pnpm project	install/build/test with sticky placement
Rails	native gems, RSpec, Postgres/Redis, two worktrees
.NET	restore, build, unit tests, then bounded Docker stress test
Python/browser	pytest and prepared browser cache
Go	module download and go test ./...
Swift Package	swift test without GUI or signing

Every class measures cold and warm elapsed time, upload bytes, cache behavior, peak memory and swap, exact exit status, disconnect and reconnect, cancellation, cleanup, artifact checksum, and cross-job contamination.

20.3 Go/no-go thresholds

The design advances beyond v1 only if:

• 1,000 snapshot captures under injected concurrent edits produce zero silently accepted mismatches;
• 100 injected disconnects produce zero duplicate executions and zero permanently unknown accepted jobs;
• 500 mixed success, failure, cancellation, and timeout jobs leave no reproducible orphan owned by mac-worker;
• a seven-day workload has less than 1% infrastructure failures and requires manual SSH intervention for less than 2% of jobs;
• a 60-minute representative job does not reach sustained critical memory pressure and keeps peak swap at or below 2 GB;
• jobs longer than 60 seconds have end-to-end median duration no more than 15% worse than local, while three minis provide at least twice the total throughput of the MacBook;
• the matched two-hour A/B workload lowers average MacBook CPU or power by at least 30% without worsening the feedback loop by more than 15%;
• no planted secret appears in snapshots, logs, metadata, or artifacts;
• retained data stabilizes within configured limits without global pruning.

21. Delivery boundary

The V1 execution core is complete when setup, doctor, run, workers, status, reconnectable logs, cancel, fetch, and safe gc work against the three configured Macs for trusted batch commands, and the automated correctness/lifecycle suites pass. The personal V1 roadmap is complete when the phase-4.5 read-only local dashboard also passes its acceptance criteria; dashboard availability never becomes a prerequisite for CLI execution.

The following require a new design review rather than incremental scope creep:

• worker up and continuous synchronization;
• multiple concurrent heavy jobs per worker;
• untrusted-code isolation;
• automatic runtime provisioning;
• multi-user scheduling and quotas;
• a remotely hosted, multi-user, or non-loopback service or web UI;
• Git-history-aware snapshots;
• Kubernetes, Coder, or a remote-execution control plane.

22. Alternatives considered

Direct SSH commands

Too little lifecycle state: it cannot safely answer whether a disconnected command was accepted, preserve exact exit status, isolate worktrees, or validate artifacts.

rsync directly into a persistent workspace

Rejected because concurrent edits can create hybrid snapshots and direct reuse can leave stale files or mutate a running job. Rsync remains the transport between immutable staging and a unique incoming directory.

Mutagen

Useful later for continuous one-way synchronization, but continuous state is unnecessary for batch v1 and complicates snapshot boundaries.

BuildBuddy Remote Runner

The closest commercial alternative. It warrants a separate proof of concept if Enterprise licensing is acceptable, but its Git-centric dirty-change transport and Docker/macOS behavior must pass the same acceptance tests.

DevPod or Coder

These target long-lived remote workspaces. They are appropriate only if the remote workspace becomes the editing source of truth, which is not this design.

Kubernetes

Rejected for three fixed native macOS executors. It does not provide native macOS worker nodes and would require Linux VMs without solving local worktree snapshots.

23. Consequences

This design intentionally trades a small amount of startup work for correctness and debuggability. It provides a universal transport and lifecycle for Git worktrees, not magical runtime compatibility. Most implementation complexity is concentrated in four explicit contracts: immutable snapshots, atomic leases, durable job state, and bounded artifact transfer. Everything else delegates to existing macOS, Git, OpenSSH, and rsync behavior.