Writing

Moving Cat Herder's tests and CI onto my own machines

A used mini PC joined the agents' test machines, then took over CI. That stopped the GitHub bill and made CI 2.5 times slower, until one-time runners on three machines got most of the time back.

By late October, tests had become the bottleneck in building Cat Herder, which runs many coding agents on one repo at once. GitHub was also billing for the repo’s Actions runs, which carry the CI that runs before each release: about $150 for October after the plan’s included minutes.

A mini PC for the tests

Kind of everyone uses a Mac mini, but when I looked around I noticed small office PCs that companies had retired, and there are so many of them that they’re cheap. I bought a used Lenovo ThinkCentre M75q Gen 2 on eBay for $230 with tax, just to run tests, and it came with a little certificate of adoption. I put Linux on it and named it testbay. Cat Herder already sent each test run to the first machine on a list with room for it, so adding testbay was mostly a matter of putting it on the list.

Over about a day in early November, testbay finished about a quarter of the agents’ test runs across testbay, agentbay and my Mac. Against agentbay, the busy shared machine, it was faster in 52 of 55 matched runs of the same test work, usually about a third quicker.

CI onto testbay

Cat Herder’s CI is 9 to 13 jobs at once, unit tests plus the integration tests split into slices. They all ran on GitHub’s hosted Linux runners, and on a private repo GitHub bills those by the minute once the plan’s included allowance runs out. Two weeks of passing CI runs came to 10,861 job-minutes (each job’s minutes, added up), a median of 297 a run.

The CI workflow now reads one repository variable, CI_RUNS_ON, which holds a runner label. When it’s set, every job goes to a self-hosted runner with that label instead of GitHub’s own; unset, the jobs run on GitHub’s runners as before, so going back takes one command. A small script also stops a CI run at once when no runner is online to take it, rather than letting it sit queued for a day. The first three tries on testbay failed: tools missing from the job image, too many jobs at once, and a quirk with the machine’s hostname.

Moving every job to testbay stopped the bill. The day after the switch, the repo showed no Actions charge at all. It also made CI about 2.5 times slower. The median run went from 38 minutes to 95, because jobs that had run side by side on GitHub now waited their turn on one box.

In its first days, a CI run on testbay hit a real TypeError that the agents’ own test runs had missed, because those machines had an older git, and the tests that would have failed skip themselves on it.

Three machines, one label

So I let more machines share the work. A self-hosted runner normally registers once and stays tied to one machine. With one-time runners, a machine registers a fresh runner only when a job is waiting, runs that one job, and the runner goes away, so several machines can answer to the same label. Three do now: testbay; agentbay, the always-on Linux box that already runs my agent sessions; and oraclebay, an always-free ARM VM on Oracle Cloud, which costs nothing but runs slower than the other two and means depending on Oracle. All three register with one GitHub token that can only register runners, and each reads it from a password manager when it starts.

Where CI ran Passing runs Jobs per run Wall time, median (range) Job-minutes per run, median
GitHub-hosted Linux runners 37 9 to 13 38 min (21 to 81) 297
testbay alone 17 9 95 min (74 to 248) 211
testbay, agentbay and oraclebay, one-time runners 3 13 53 min (47 to 156) 243

Wall time runs from the first job’s start to the last job’s end, so it leaves out time queued before the first job. Three runs is a small sample. The first took 156 minutes, while the machines were still joining, and the two after it took 47 and 53. Job-minutes fell on testbay partly because it ran 9 jobs where the late hosted runs had 13, so that drop isn’t a per-job speed-up.

What it costs now

CI is still slower than it was on GitHub, about 50 minutes against 38, with no per-minute charge. GitHub lists a standard Linux runner at $0.006 a minute and self-hosted runner minutes as free. In December 2025 it announced a $0.002-a-minute charge for self-hosted runners on private repos, then postponed it, and as of November 2026 it isn’t in effect.

At October’s billed rate, about $5 a day, the mini PC pays for itself in six or seven weeks, before electricity. A small pool of GitHub runners still bills when agents send it a few named test modules, within a hard budget.

The steps, release by release

  • 0.53.1 and 0.53.2: a test machine sets how many runs it takes at once, and a heavy run takes at most half of them.
  • 0.53.3: a paid pool on GitHub Actions can run a few named test modules, with a hard budget, and falls back to a machine when the budget is spent.
  • 0.53.4 and 0.53.5: a small run goes to the Mac when the test machine is full.
  • 0.53.6: test runs pick their machine from a list, the first with room.
  • 0.53.7.2: the full suite runs as small pieces on whichever machine has room, so one full run can spread across machines.
  • 0.54.4.4: runs go to the machine with the most room, each piece counting the ones placed before it.
  • 0.54.5: CI can run every job on a self-hosted runner named by CI_RUNS_ON, and stops at once when no runner with that label is online.
  • 0.57.0: one-time runners that register only when a job is waiting, which lets several machines share one label.

About Cat Herder. Cat Herder lets many coding agents work one repo in parallel, in Claude Code, Codex, Cursor and more, with the option of a binding review before each change merges. The queue is a branch in your repo, with no server. The first write to land wins each item, and the work lasts from one session to the next. One orchestrator session scales the work out to worker and reviewer pairs on your machine, on other machines over ssh, or in cloud sessions. It watches for stalls, merges accepted work into release branches and hands work to a fresh session when one runs out of context. Perch, an optional Kanban board in the browser, shows the whole queue as it moves. More in the Cat Herder case study.