In late June 2026, my org's GitHub Actions stopped at the billing cap. Every PR (pull request) CI (continuous integration; a system that runs tests automatically on each change) check FAILED in 2 seconds with an empty log. Around the same time, plugin release notifications stopped arriving in Slack. Not a single line of code was wrong. An external service had, one day, stopped my infrastructure over billing.
At the time I was running GitHub Actions on the free tier. But I hit the free-tier cap, CI was blocked, and to keep developing I had no choice but to start paying by usage (metered charges for anything beyond the free minutes). "Free" ended right there. From that point on, money was going out no matter what — that is the starting point. So the question was not "cut cost by testing less" but "which is the better deal: keep paying GitHub, or build my own base?" GitHub's cost is metered, so tests run on every commit and the monthly amount grows the more products and the more active the development gets. A home CI server (ci1/ci2 class, about ¥135,799), by contrast, is a one-time outlay plus electricity, and it does not grow no matter how much you use it. A cost that keeps growing versus a one-time cost that does not — that structural difference was the deciding factor. And the per-run time of CI grows as the code grows. Measured: one product's GitHub Actions run time grew from about 1 minute on average in March this year to about 16 minutes on average — with the longest over an hour — by August. Commits across the products I actively run come to roughly 2,400 a month. Keep paying for that volume and that growth on GitHub's metered pricing (Linux $0.008/min), and the over-the-free-tier portion alone lands around $150–200/month, swelling every month as tests get longer. A home CI server for about ¥135,799 overtakes that in under half a year, and is cheaper forever after (the commit counts and run times are measured; the monthly dollar figure is an estimate from the free tier and average minutes). On top of that, the security-scanning tools I wanted to add (DefectDojo and so on) also need somewhere to run. So the best move was to build a single infrastructure base at home and run tests, security scanning, and monitoring all there. This is the record of acting on that decision. Per the records at the time, this base's monorepo (a layout that bundles multiple repositories into one) grew from its first commit to 330 commits and 35 ADRs in about a month.
Let me put the conclusion first (about an 11-minute read).
- Every run failing instantly (2 seconds) with an empty log was the fingerprint of a GitHub Actions billing block — an external service stops your infrastructure one day, silently. Don't make the mechanism that reports failure depend on the mechanism that fails.
- I built a base of my own that doesn't stop, and didn't throw away the old one either — I switched to self-hosted (owned by me) runners while keeping the GitHub Actions-side workflows, so I could go back with no changes if billing returned.
- A distributed security workflow stopped every PR — a full-history scan by gitleaks (a tool that detects secrets slipped into code) drops even healthy PRs. A gate must be verified not only for "can it block" but for "does it pass correctly," or it stops everyone's work.
- The pass/fail bar for HA (High Availability; a setup that doesn't stop even if one node dies) is actually cutting power and network on real hardware — even with all unit tests green, there were 3 bugs that only surfaced when I dropped it on real hardware.
Symptom: every run fails instantly and the log is empty
At first, I suspected our own workflows. But both CI checks FAILED in a consistent 2 seconds. Opening the log, there was nothing inside. To isolate it, I first suspected the Slack notification path and hit the webhook directly.
# Is the webhook itself alive? → HTTP 200, delivered to Slack = the code is innocent
curl -s -o /dev/null -w "%{http_code}\n" -X POST "$SLACK_WEBHOOK" \
-H 'content-type: application/json' \
-d '{"text":"probe from local"}'
# => 200
The webhook was 200. The local notification script was fine too. So the notification logic was innocent. Next, peeking at the CI side with gh run view, every run was ending in 2 seconds with an empty log. Here the picture came together. The runner itself is not starting. The cause was a GitHub Actions billing block; the org's GitHub Actions were fully stopped. Every run failed instantly (2 seconds) before it could even start, with nothing left in the log — that behavior itself, in hindsight, was the fingerprint of a billing block. And the nasty part was that the stopped mechanism was silent. Because the "mechanism that reports failure" — Slack notifications — itself depended on Actions and went quiet, it took several days to notice.
The turn: build a CI that doesn't depend on hosted, "in code"
An external SaaS (software as a service) can, over its billing situation, stop your infrastructure one day out of nowhere. Then the only choice is to hold a base of your own that doesn't stop. And if I'm building a base at home, putting only CI on it would be a waste. Tests, the security scanning I plan to add, and monitoring — I'd consolidate them all onto the same single infrastructure. That, I decided, was best. The plan was: "stop the practice of SSHing (remotely logging into a server to run commands) in to type commands directly, and manage the home server fleet declaratively with Ansible / Docker Compose / Terraform (all tools that define and apply configuration as code)." I harvested the state of the single existing CI server into a monorepo, and codified (defined as code) the self-hosted runners, a vulnerability ledger, observability (making runtime state visible from outside) via Grafana/Loki/Prometheus (a monitoring stack for visualization, logs, and metrics), and the log platform — almost all in one go over the first day or two.
Each product's workflow is mechanically re-pointed from a hosted premise to self-hosted runners.
# Drop the hosted (ubuntu-latest) dependency; point at the home runner labels
jobs:
test:
runs-on: [self-hosted, linux, docker] # labels for the home runner fleet
steps:
- uses: actions/checkout@v4
- run: pnpm install --frozen-lockfile && pnpm test
I kept the Actions-side workflows rather than deleting them, so I could go back with no changes if billing came back. "If it breaks, switch to another base, and don't throw away the broken one either" — that became the basic policy.
Trap 1: the distributed security workflow stopped PRs across the whole org
Soon after migrating, the commonly distributed security workflow caused an incident that blocked every PR at the destinations. At the time this workflow was distributed to 25 of the org's 102 non-archived repositories, so the blast radius was wide. The real cause was how gitleaks was used.
gitleaks detect --source . scans the target across all refs (effectively git log --all). That means it picks up test dummy secrets left on old branches and old leak traces, so even a healthy PR always fails. On top of that, workspace pollution on the self-hosted runners produced false positives detecting "a file that shouldn't exist in that PR." The fix is to limit the scan scope to the PR diff.
# Stop scanning all refs and always failing PRs. Look only at the diff.
gitleaks detect --source . --redact \
--log-opts="origin/${BASE_REF}..HEAD"
To be safe, I even prepared a self-test workflow that runs E2E (end-to-end) verification on both a branch that should PASS and a branch that should FAIL. The lesson: a security gate is no good if it can only "block" — unless you also verify it "passes correctly," it becomes a weapon that stops everyone's work.
Trap 2: going self-hosted broke the Docker build as a side cost
The migration came with debts too. Right after fully removing the hosted runners, CDK (a tool that defines cloud infrastructure as code) Docker bundling stopped working on several products. In a "runner inside a container + the host's docker socket" setup, the volume mount the bundling uses came up empty. On one product the deploy workflow was wiped out for a while, and I lived on manual deploys from a local Mac. The remedy was to replace Docker bundling with local dependency expansion via pip install --target on the runner, and I finally closed it after confirming 5 consecutive successful deploy workflows by measurement. Move the base and the build premise on top of it quietly collapses. Here too, the policy of confirming "it works" only by measurement paid off.
HA: caused a split-brain and fixed it the same day by dropping real hardware
With a single runner, if it dies, CI stops again. So I went to a 2-node setup, and here the biggest incident happened. After failing over from ci1 to ci2 to add an NVMe (a fast storage standard), rebooting ci1 caused both nodes to become active and uncoordinatedly fight over the same job queue — a split-brain (both systems becoming primary at once), which I observed late at night. Jobs were submitted from the web to ci2 but executed on ci1 — an unsettling state.
The cause was a structural problem: "the starting system has no gate to decide whether it is the designated active." ci1 was auto-starting without knowing ci2 was active. The same day I implemented atomic lease fencing (a mechanism that reliably locks out the old primary to prevent double activation) — a fence-agent resident on both nodes with heartbeat 2s / TTL (time-to-live) 6s / grace 6s — and passed a real-hardware kill-test (a test that actually cuts power and network). Auto failover 18 seconds; self-fence + takeover 17 seconds on an NFS (a network-shared filesystem) partition; data divergence exactly zero. The biggest lesson here is failover ≠ fencing. Manual failback gets forgotten. And even with unit tests (bats; a testing framework for bash) all green, there were 3 bugs that surfaced only when I actually cut power and network on real hardware (a livelock where heartbeats yield to each other during grace and make no progress; infinite I/O blocking from a hard-mount; a blind steal that takes the primary without checking the other side's lease). This was the day I wrote into the Mission that the pass line is the real-hardware kill-test, not a design document.
Infra Portal: consolidate all products onto one screen
Finally, I grew a home-grown portal (I call it the Infra Portal) that consolidates onto one screen: CI status, deploy promotion, multi-provider cost, backup sufficiency, and the liveness of servers and the local LLM (large language model). There are two design ideas. One: "don't trust registration; auto-detect capability from the repo and the live environment by measurement." The other: "don't treat HTTP 200 as alive — read the meaning of the actual response." The former catches gaps in self-reporting; the latter catches "the server responds but the contents are broken." Here too there was a laughable failure: one node's card was frozen at 62 hours ago because of an asymmetric HA install — I had forgotten to put the reporting script needed on the passive side. I fixed it by making both nodes symmetric.
Takeaways you can reuse
- Tests can't be skipped. So I looked at the cost and consolidated onto one base of my own. Instead of fitting inside a free tier, I estimated cost on the premise of running continuously, and drew CI, security, and monitoring onto a single piece of infrastructure.
- Every run failing instantly with an empty log is the fingerprint of a billing block. An external service stops your infrastructure one day, silently. Don't make the mechanism that reports failure depend on the mechanism that fails.
- Build the base yourself, and don't throw away the old one. Switch to self-hosted while keeping the hosted workflows, so you can go back with no changes.
- A distributed gate must be verified down to "passing correctly." Like gitleaks' all-refs scan, a gate that can only block ends up stopping everyone's work.
- The HA pass/fail bar is the real-hardware kill-test. failover ≠ fencing. Even with bats all green, there are bugs that only surface when you actually drop it on real hardware.
