A rented test box: what it costs (October 2026)

Follow-up to local-build-monster.md, which planned around the Macs we
already own. This one prices a rented Linux box, Hetzner first. Prices are from Hetzner's own
pages on 2026-10-24, excl. VAT. Hetzner raised prices on 15 June 2026, so blog posts quoting
AX41 at €39 or CCX13 at €16 are out of date.

What CI actually spends time on

From the GitHub API, apps.yml, 40 runs between 2026-10-23 20:47 and 2026-10-25 00:24 UTC:

• 540 jobs, about 300 job-minutes. Roughly 13 jobs per PR.
• Each job spends about 20 s getting ready (checkout 2 s, setup-bun 2 s, bun install 13-16 s)
  before a test runs. 540 × 20 s is about 180 of the 300 minutes: more than half the minutes
  are setup, repeated on a fresh VM every time.
• Most suites take 25-60 s. The long poles: skate-tech-park (about 355 s of tests), two-fronts
  (about 70 s), codesplitter (about 70 s).
• Queueing is small but real: 39 of 540 jobs waited over a minute to start, the worst 234 s. That
  is the free public-repo cap of 20 concurrent jobs, hit when a packages/ PR fans out to every app
  during a merge round.
• Volume: about 8,000-10,000 job-minutes a month across all workflows. That costs $0 today (public
  repo, standard runners). The same minutes on a private repo would be about $50-60 at $0.006/min.

So a box doesn't save money; GitHub is already free. What it buys is speed: warm
node_modules, a warm checkout, no 20-job cap, and 16 fast threads in place of a 2-vCPU runner.

The options

Bills to watch:

• A Hetzner Cloud server bills hourly until it is deleted; powering it off does not stop the bill (COSTS.md).
• Dedicated servers bill monthly with a setup fee, and Hetzner's notice period is monthly.
• One IPv4 is included; extra IPs are €1.70/month.

The catch: this repo is public

scripts/ci-guard.ts fails any workflow that uses a runner other than a standard GitHub-hosted
one, and that is on purpose. On a public repo, a fork PR can edit the workflow to target a
self-hosted label, and its code then runs on the box. GitHub's own docs recommend self-hosted
runners only for private repos. That leaves two shapes:

1. A builder we drive (recommended first). The box is not a GitHub runner. Agents send a branch to it
   over Tailscale/SSH, it runs scripts/ci-apps.ts's jobs sharded across 16 threads with a
   node_modules kept per lockfile hash, and it reports pass/fail on the PR as a comment or
   status. GitHub's free hosted runs stay the merge gate. No fork code ever reaches the box. This is
   the "builders we drive, never self-hosted runners" rule from local-build-monster.md, on rented
   hardware. The trust dial and tiers there (Part 2, Part 5) can later let a green box run stand in
   for most of the hosted suite.
2. A self-hosted runner as the gate (later, only if the gate is still the wait). Needs the owner,
   since these are repo settings: set fork PR workflows to require approval for all outside
   collaborators; run the runner --ephemeral inside a throwaway container or VM per job; keep no
   secrets on the box; and loosen ci-guard.ts for exactly one label. Every agent PR comes from a
   branch in this repo, so agents would never wait for approval; only forks would.

Recommendation

1. Trial on hourly cloud first: one CCX33 for a week of real merge rounds, about 40 hours,
   about €9. Time the same apps.yml run there against GitHub, then delete the server.
2. If it pays off, rent an AX42 at €79/month (smallest storage option) plus €49 setup, about
   €1,000 for the first year. Same Zen 4 cores, twice the RAM of the CCX33, for a bit over half its
   monthly price.
3. Run it as shape 1 (a builder we drive). GitHub stays the free final gate.

Pick a hosted runner service (Ubicloud, Blacksmith) only if what we want is GitHub's own gate
running faster with no box to look after; it won't fix the setup time, which is the bigger cost.

Sources

• Hetzner price adjustment, 15 June 2026: https://docs.hetzner.com/general/infrastructure-and-availability/price-adjustment/
• Hetzner AX line (live prices and specs): https://www.hetzner.com/dedicated-rootserver/matrix-ax/
• Self-hosted runner fee announced, then postponed (Dec 2025; still not charged, and never for public repos): https://www.theregister.com/2025/12/17/github_charge_dev_own_hardware/
• Runner price comparisons (Ubicloud, Blacksmith, GitHub $0.006/min since 2026-01-01): https://runnerbench.com/ , https://craci.com/compare/github-actions-runner-providers
• Hetzner Cloud increase analysis: https://wz-it.com/en/blog/hetzner-price-increase-june-2026-cpx-ccx-alternatives

MANUAL RUN: Make local build monster!

Drafted as a GitHub issue (filing was blocked); lives here until it is filed.

The goal: build the box, let the agents play in it. Tests are the bottleneck, so we stack safe changes, isolate risky ones, and run on assets we already own. Free hosted Actions stay the final gate; the Macs stay builders we drive, never self-hosted runners on this public repo (a fork PR could run code on them).

Related: VPS_TEST_RUNNER.md (rented Hetzner box, priced), #1242 (parked: hosted fix lane), #1252 (closed: 3 containers, about +$21/mo worst case), #818 (Dockerfiles first-class), #897, #1479 / #1484 (Mervin merges the board in a sandbox, stage by stage).

Inversion of control

I build the option matrix (tiers, gates, switches). The agents choose the buttons: which tier a change claims, which stack it rides, which gates they want on. The matrix is the keyboard; they play it.

Part 1: The machines (the manual run)

Needs the owner at the machines.

• Mac Studio (M1 Pro): office host, always on.
• MacBook Pro M1 Max 64GB: Docker builder swarm.
• MacBook Pro M1 Pro 16GB: daily machine, or overflow test builder.

Steps

1. 64GB machine: raise the Docker Desktop memory limit (default about half of RAM; aim for about 48GB). Keep it awake on power.
2. Build and smoke the three codeSplitter images locally (sandbox/, deploy-sandbox/, deploy-sandbox/ci.Dockerfile). Containers: Dockerfiles as a first-class workflow (CLI, CI, prod-like runs, codeSplitter UX) #818 Part 1 will do it; until then plain docker build -f ... <context>.
3. Run the office on the Studio; reach it from the other Macs with office:phone over Tailscale (apps/agent-office/README.md).
4. Time one real rebuild and one bun test run per machine, and note memory use under 4-8 parallel containers.
5. Apple Silicon builds arm64 and the hosted containers are x86: check a Rust/cargo build both ways (Rusty Buns builds: a desktop watch loop, and x86 Rust crates in the hosted deploy container #760).
6. Shard scripts/ci-apps.ts jobs across machines over SSH (each app's suite is independent).
7. Cap parallel test workers per machine (same idea as the typecheck cap of one checker per core).

Open questions

• Workers run on the office host. Sending work to a second Mac needs a remote runner, which is not built (see Containers: Dockerfiles as a first-class workflow (CLI, CI, prod-like runs, codeSplitter UX) #818 Part 4's Run on desktop handoff). Is there a multi-host join (URL code + password) I missed?
• Studio RAM?

Part 2: Risk tiers (cheapest test wins)

A small classifier over git diff --name-only, next to scripts/ci-apps.ts.

Part 3: Stacking

• Tiers 0 and 1 ride together in one test run.
• Tier 3 gets its own run, so a broken dev dep is blamed on the dep.
• When a stacked batch fails, bisect: split in half and rerun, instead of failing every PR in it.
• Mervin builds the waves (tiers 0 and 1 together, tier 3 alone).

Part 4: Same tiers, two places

• Locally on the Macs, as the agents work.
• On GitHub as the free final gate before merge.

Part 5: Trust dial

Track each agent's pass rate per tier over a rolling window. An agent with a long clean run on tier 1 gets the targeted run only; a new agent, or one that just broke something, gets the full app suite. Trust is earned and lost.

Part 6: The box is the test

Each agent works in its own sandbox clone; the sandbox's real checks are what a merge must pass. Tests guard the box's walls, not every individual change.

Order

1. Tier classifier + stacking rule (one small script and a test).
2. Parts 1 and 4 (the machines and the two gates).
3. Trust dial and the box (Parts 5 and 6), as follow-up issues.