What is CI, Which I Keep Hearing About? What a Beginner Learned by Trying It

!

This article is a repost from Claude Code started.

What the heck is CI that I keep hearing about lately?

What the heck is CI that I keep hearing about lately?

I'll share what I learned as a beginner through experience!

When you create apps and development tools with AI, CI suddenly appears before you know it. In my case, I think it started when I was using Opus 4.8.

AI is doing something on GitHub. It checks the code, runs tests, and when it's done, a green badge appears. It looked like it was investigating whether the app is okay to distribute before putting it on GitHub.

I wondered, "Am I being inspected by GitHub? I don't really get it. If it turns green when CI passes, then, well, let's just do it."

That's how it was at first.

CI is a mechanism that automatically checks every time you make changes

CI stands for "Continuous Integration," which is called 継続的インテグレーション in Japanese.

After changing code, it checks early whether it still works when combined with other changes. For that purpose, it automatically runs builds and tests. If problems are found, you can know immediately after the change. This whole approach is CI.

When using CI on GitHub, the GitHub Actions mechanism is often used. For example, if you built something with Node.js, you can place a config like this in the repository.

name: CI

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - run: npm test

What's written is simple.

1. It starts when code is pushed
2. Prepare an Ubuntu execution environment
3. Fetch the repository code
4. Prepare Node.js 22
5. Install required packages
6. Execute tests

If all succeed, it turns green. If it fails midway, it turns red.

GitHub's official documentation also says that for CI configuration, you can either adjust the proposed template or create your own.

We decide the test content, GitHub distributes the work, and GitHub's virtual machines or registered local machines execute it

It Was Us Who Decided What to Test

I thought GitHub had some fixed tests and was reviewing apps that were sent to it.

In reality, what decided the test content was the configuration and test code in the repository. In my environment, AI created most of that. GitHub read the configuration, ran it in the specified environment, and recorded the results.

The scope that the green badge indicates also extends to this:

The tests we configured ourselves succeeded in the most recent run.

Things not written in the tests are not verified. If the tests are simple, they'll turn green quickly. If you write tests that never finish, they'll run forever. If you write wrong tests, you'll get meaningless results.

That green color I thought was GitHub's quality approval of the app. As I kept using it, I realized it was just the result of the check items we decided ourselves.

GitHub's badge documentation also states that the badge shows whether the workflow is failing or passing.

Trying It on Multiple OSes Reveals More

While looking at CI, I also noticed multiple environments like Ubuntu, macOS, and Windows lined up.

An app that runs on my Mac won't necessarily run the same way on Windows. There are differences in file handling, path syntax, permissions, and available commands. Running the same tests on multiple OSes makes it easier to find problems caused by those differences.

The extent of what CI can guarantee is limited to the environments actually run and the tests actually executed. It doesn't prove perfect operation on every device in the world. The more environments you check, the more you learn before distribution.

Lately, I've come to see CI as "a mechanism that automatically checks whether your app works across various OS and hardware environments."

Lattice's CI Takes Forever

When you let AI handle development, tests gradually accumulate. The number of supported OSes also grows. The things being checked—static checks, full product tests, validation of saved data—also increase.

The same thing was happening with Lattice.

Lattice is a tool I built to distribute development tasks in parallel across multiple AIs. As development progressed, tests increased, and we started checking in multiple environments. CI started taking strangely long to complete, and the feeling of waiting every time I pushed to GitHub grew stronger.

The background and mechanism behind creating Lattice are detailed in "How Making a Tool to Run AIs in Parallel Made AIs Stop Lying."

To be precise, the git push process itself isn't slow. It's that CI starts after a push, and the wait time for its results got longer.

So I started reviewing the CI test content. Are there duplicate checks? What should be tested in each environment? Are there any tests that never finish? While investigating, I noticed something even simpler.

Wouldn't It Be Faster to Run It on My Local Machines?

At home, I have three machines: a Mac, a Linux machine, and a Windows machine. WSL2 also runs on Windows, so if I count execution environments, that's four.

All of them had plenty of CPU headroom while waiting for CI. Especially the Windows machine, which has a lot of cores. While waiting for tests to finish on GitHub, my super machines at home were idle.

Hey, wouldn't it be faster to run CI on my local machines?

Plus, if I distribute work to all three simultaneously, I could run tests for Mac, Linux, Windows, and WSL2 in parallel.

GitHub Actions already provides a mechanism for this called self-hosted runners. You can register your own machines as execution targets for GitHub Actions. The official documentation also states you can use physical machines, virtual machines, containers, local equipment, or cloud machines.

A config that uses a registered runner looks like this, for example:

runs-on: [self-hosted, windows]

GitHub detects the push, sends the job to a matching home runner. The machine at home fetches the code, runs tests, and sends results back to GitHub. Logs and pass/fail results appear in GitHub Actions just like before.

Just running npm test locally doesn't show results on GitHub. You need to install a GitHub Actions runner on your machine and register it with a repository or organization. GitHub treats registered runners as official execution targets.

So GitHub's role is detecting pushes, assigning jobs, managing progress, and displaying logs and results. The machines actually using CPU to run tests are the three at my house.

An image of home Mac, Linux server, and powerful Windows machine running tests simultaneously

Running the Same Full Test Suite Across 3 Machines and 4 Environments

Initially, I thought about dividing the tests among the machines. Assigning roles per machine—like "these tests on Linux, these tests on Windows"—seemed like it would be efficient.

But when I actually ran it, problems started popping up one after another from Windows, which I had only tested lightly. Some code directly called Unix-like OS commands, and some file and process handling didn't work properly on Windows.

Dividing the roles means leaving some features untested on certain OSes. So I changed my approach.

Run the exact same full test suite simultaneously across all 4 environments: Mac, Linux, Windows, and WSL2.

GitHub Actions distributing the same full test suite simultaneously to home Mac, Linux, Windows native, and WSL2

This way, I don't need to assign specific tests to specific machines for each product. Add one test, and it runs identically across all 4 environments. It's also easier to compare results between environments.

If I ran the 4 environments sequentially, the time would add up. By sending to all 4 at once, the wait time approaches the time of the slowest environment.

Furthermore, within a single machine, I also split independent tests into multiple processes and ran them in parallel. The CPU usage on Windows went up, and that 20-core class machine finally started working seriously.

Home Runners Are Self-Managed

With self-hosted runners, your home machines themselves become the test environment. Things like the versions of Node.js and Git, installed software, permissions, and resident programs can all affect results.

You need to align the versions of required tools, fetch the code fresh each time, and reinstall dependencies. To get reliable test results, managing the machines becomes part of CI.

When using it with public repositories, you also need to be careful with the configuration. If you allow external changes to run freely on your home runners, commands contained in those changes could be executed on your machines. GitHub also recommends using self-hosted runners with private repositories. If you use it with public repositories, you need to configure settings to prevent runners from being triggered by external pull requests and to restrict which repositories can use them. Official notes when adding runners

How Did It Go?

I thought CI was a system where GitHub reviewed apps and gave them a passing grade.

As I used it, I realized that we were the ones deciding what to test, and GitHub was executing the specified tasks and showing the results. Additionally, I could choose the machines that ran the tests, and I could even provide them myself.

With Lattice, I set it up to run the full test suite simultaneously across 4 environments: Mac, Linux, Windows, and WSL2. By also using the CPU cores within each machine for parallel processing, even the full test suite doesn't take much time.

Running the same tests across 4 environments revealed issues that only fail on Windows. I can now fix problems that would have been hard to find if I only developed on Unix-like environments, before distributing the app.

In the end, running full tests across 4 environments lets me take advantage of the performance of the expensive machines I have, and reduces the time CI takes. It also checks compatibility more thoroughly than before. I think using multiple local machines in parallel was a good decision.