Screenshot testing a design system with Playwright and Docker

I work on the Koobiq design system, which has component libraries for React and Angular. in this article I'll show how the screenshot tests in our React library are set up and which decisions made running them and updating baselines easier.

Problem

in Koobiq React, the shared FormField wrapper is the base of every field: Input, Select, Textarea, DateInput and a dozen others. change its padding once, and all of them move at once, in the light and dark themes alike. our unit tests run in jsdom and don't check how things look, and walking through every combination by hand is expensive.

screenshot tests have a reputation, though. they are flaky, because the same page renders differently on different machines. and their baselines, the reference PNGs a test compares against, go stale after every intended style change and are tedious to update. so most of the work below went into running them reliably and keeping baselines fresh without extra effort.

Solution

we added screenshot tests with Playwright on top of Storybook:

• all states. a component gets a test story with a "variants × states" grid, and one test screenshots it in the light and dark themes.
• pixel precision. a test fails on any changed pixel, while the date, locale, time zone and animations are fixed, so screenshots don't change from run to run.
• one environment. screenshots are compared only inside a Docker image, both locally and in CI.
• baselines in git. PNGs live next to the component and change in the same PR.
• no busywork. updating baselines doesn't require Docker: a /approve-snapshots comment on the PR regenerates them in CI and commits them to the branch.

these are the baselines of test 01 for Button: six variants by six states in the light and dark themes.

the grid has 36 combinations, 72 in two themes. screenshotting each one separately would mean 72 baselines per test, while a grid needs two: one per theme. besides, in a grid a broken cell stands out right away against its unchanged neighbors.

Structure

shared helpers and the environment live separately, while tests and baselines sit in the component folder:

packages/components/
├── e2e/
│   ├── E2eGrid.tsx
│   └── utils.ts                  # e2eGotoStory, e2eScreenshotThemes
└── src/components/Button/
    ├── Button.module.css
    ├── Button.e2e.stories.tsx    # test stories
    ├── Button.e2e.ts             # test cases
    ├── __screenshots__/          # baselines
    │   ├── 01-light.png
    │   └── 01-dark.png
    └── …
tools/e2e/
├── Dockerfile
├── docker-compose.yml
├── run.mjs
└── …
playwright.config.ts

so a PR that changes a component's styles shows the new screenshots right away as well.

Playwright setup

the Playwright config fixes the locale and time zone, disables animations and sets a zero comparison threshold:

// playwright.config.ts (trimmed)
export default defineConfig({
  retries: 0,
  expect: {
    toHaveScreenshot: {
      stylePath: 'tools/e2e/screenshot.css',
      threshold: 0,
      animations: 'disabled',
    },
  },
  use: { locale: 'en-US', timezoneId: 'UTC' },
});

threshold: 0 removes the tolerance for perceived color difference (the default is 0.2), so a test fails on any changed pixel. to keep environment differences from causing false failures, we run the tests in Docker - more on that below. retries are off: a flaky test should fail the run.

while taking a screenshot, Playwright fast-forwards finite animations to their end and resets infinite ones to their initial state. stylePath adds screenshot.css with the app font and the theme background.

// package.json (trimmed)
{
  "scripts": {
    "e2e:build": "storybook build --test --quiet --output-dir storybook-static-e2e",
    "e2e:serve": "vite preview --outDir storybook-static-e2e --host 127.0.0.1 --port 6007 --strictPort",
    "e2e:components": "playwright test packages/components",
    "e2e:docker": "node tools/e2e/run.mjs",
    "e2e:docker:update-snapshots": "node tools/e2e/run.mjs --update-snapshots",
  },
}

Component test

a docs story shows one feature, a test story shows every state at once. it's still a regular story, just in a *.e2e.stories.tsx file with an E2E/ title prefix, for example Button.e2e.stories.tsx:

// Button.e2e.stories.tsx (trimmed)

// Hover, active and focus are turned on with CSS Module classes.
const states = [
  { title: 'disabled', isDisabled: true },
  { title: 'normal' },
  { title: 'hover', className: s.hovered },
  { title: 'active', className: s.pressed },
  { title: 'focus', className: s.focusVisible },
  { title: 'progress', isLoading: true },
];

export const StateAndStyle = {
  render: () => (
    <E2eGrid columns={states.length}>
      {buttonPropVariant.flatMap((variant) =>
        states.map(({ title, ...state }) => (
          <Button key={`${variant}-${title}`} variant={variant} {...state}>
            {title}
          </Button>
        ))
      )}
    </E2eGrid>
  ),
};

• forced states. you can't hover 36 buttons at once. Button and some of our other components turn React Aria's isHovered, isPressed and isFocusVisible into CSS Module classes, so the story passes those classes directly. if your CSS relies on :hover and :focus-visible, storybook-addon-pseudo-states does the same trick by rewriting them into classes.
• screenshot target. E2eGrid renders an inline-grid with data-testid="e2eScreenshotTarget": the screenshot is cropped to the grid content, and a small padding keeps focus rings in the frame.
• overlays. they are opened by props (defaultOpen, isOpen), and only the overlay itself gets into the screenshot. two open popovers would cover each other, so a select has two stories: the closed field in every state and the open list.

the Button.e2e.ts test case opens the Button story and screenshots it with two calls from e2e/utils.ts:

// Button.e2e.ts (trimmed)
test.describe('Button', () => {
  test('with title', async ({ page }) => {
    await e2eGotoStory(page, 'e2e-button--state-and-style');
    await e2eScreenshotThemes(page, '01');
  });
});

e2eGotoStory opens the story without the Storybook UI, and e2eScreenshotThemes takes two screenshots of the grid: in the light and dark themes.

Storybook setup

test stories get a build of their own: storybook build --test turns on Storybook's test mode without docs pages, and in this mode .storybook/main.ts picks up only the e2e stories, while the regular build skips them:

// .storybook/main.ts (trimmed)
const isTestBuild = process.argv.includes('--test');

const stories = isTestBuild
  ? ['../packages/**/*.e2e.stories.@(js|ts|tsx)']
  : ['../packages/**/!(*.e2e).@(mdx|stories.@(js|ts|tsx))'];

this keeps test stories off the docs site, and the test build runs about twice as fast as the regular one.

Docker setup

a screenshot taken on macOS can differ from one taken on Linux/Windows: rendering depends on the OS and its fonts, the browser version, the hardware and even on whether a laptop runs on battery (more in the Playwright docs). so baselines are taken and compared in one environment, a Docker image, both locally and in CI. pin three things:

1. the image, by digest. a digest is an immutable hash of the image (sha256:…). a tag like v1.62.1-noble can be re-pushed with different contents and the rendering silently changes, while a digest always points to the same image.
2. the Playwright version. the browsers in the image must match the @playwright/test package, so its version in package.json is exact, without ^. run.mjs puts it into the image tag, and the digest is updated together with the package in one PR.
3. the platform. docker-compose.yml sets linux/arm64: such an image runs natively on Apple Silicon Macs and on GitHub's ubuntu-24.04-arm runners.

the Dockerfile caches the dependency install and the Storybook build separately. editing test cases and baselines doesn't rebuild Storybook: they are copied after the build.

# tools/e2e/Dockerfile (trimmed)
ARG PLAYWRIGHT_VERSION
FROM mcr.microsoft.com/playwright:v${PLAYWRIGHT_VERSION}-noble@sha256:dcc5531e…

# 1. Dependency cache: source changes don't trigger pnpm install.
COPY package.json pnpm-lock.yaml ./
RUN corepack enable pnpm && pnpm install --frozen-lockfile

# 2. Storybook cache: tests and baselines are excluded for now.
COPY --exclude=**/*.e2e.ts --exclude=**/__screenshots__ . .
RUN pnpm e2e:build

# 3. Add them after the build: editing them keeps the RUN above cached.
COPY . .
CMD ["pnpm", "e2e:components"]

day to day, three commands are enough:

pnpm e2e:docker                       # the whole suite
pnpm e2e:docker -g "Button with icon" # only the "Button › with icon" test
pnpm e2e:docker:update-snapshots      # rewrite changed and missing baselines

they run tools/e2e/run.mjs, a small Node script instead of a shell one-liner, so the command is the same on macOS, Windows and in CI.

CI setup and baseline updates

Checks in a PR

on every PR and push to main, e2e.yml runs the same command in the same image on an arm64 runner. the job uploads the Playwright HTML report (the expected screenshot, the actual one and the diff for every failed screenshot), and on a failure leaves a PR comment with a link to it.

# .github/workflows/e2e.yml (trimmed)
on:
  push:
    branches:
      - main
  pull_request:

jobs:
  e2e_tests:
    runs-on: ubuntu-24.04-arm
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - run: node tools/e2e/run.mjs --workers=100%

Updating baselines

you don't need Docker for this - it all happens right in the PR:

1. the check fails, a maintainer opens the report and looks at the diff.
2. if the changes are intended, after reviewing the code they comment /approve-snapshots.
3. e2e-approve-snapshots.yml regenerates the baselines in the same Docker image and pushes the PNGs as a separate commit to the PR branch. GitHub shows the old and new baselines side by side right in the PR diff.

# .github/workflows/e2e-approve-snapshots.yml (trimmed)
on:
  issue_comment:
    types: [created]

jobs:
  approve_snapshots:
    if: >-
      github.event.issue.pull_request &&
      contains(github.event.comment.body, '/approve-snapshots') &&
      contains(fromJSON('["OWNER", "MEMBER"]'), github.event.comment.author_association)
    # One run per PR at a time, otherwise two pushes collide.
    concurrency:
      group: e2e-approve-snapshots-${{ github.event.issue.number }}
    runs-on: ubuntu-24.04-arm
    steps:
      # issue_comment starts on the default branch, so the PR branch is resolved separately.
      - uses: xt0rted/pull-request-comment-branch@e8b8daa837e8ea7331c0003c9c316a64c6d8b0b1 # v3.0.0
        id: comment-branch
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          ref: ${{ steps.comment-branch.outputs.head_ref }}
      # Retries are fine here: a flake costs only another comment.
      - run: node tools/e2e/run.mjs --update-snapshots --workers=100% --retries=2
      - uses: stefanzweifel/git-auto-commit-action@4a55954c782fc1ea30b9056cd3e7a2b40ca8887d # v7.2.0
        with:
          commit_message: 'test: update e2e snapshots'
          file_pattern: '**/__screenshots__/*.png'

What we don't screenshot

• animations. Playwright stops them, and a live animation can't be captured reliably. so we don't screenshot skeletons with an endless shimmer at all.
• states from real interaction. in the components where React Aria sets hover and focus as data attributes rather than our CSS Module classes, those states can't be turned on with a prop, so they aren't in the grid.
• behavior. opening, keyboard and focus are checked by unit tests: a screenshot only sees the result.

Results

61 components under test, 69 tests, 138 baselines, about 7 MB in git. the tests themselves take 30 seconds on four workers, the whole CI job with the image build about 2.5 minutes.

it paid off right away: while preparing the tests, we found seven visual bugs that had gone unnoticed before.

stability and baseline updates took most of our effort, and Docker with the PR comment are our answers to them. what's been harder for your team: keeping screenshot tests stable or keeping baselines up to date?