# Visual regression testing, explained in pictures

> How screenshot comparison works, why visual tests go flaky and how to fix each cause, the tools, and a set-up checklist with a Playwright example.

Guide by Serghei Pogor, founder of CurlShot (https://curlshot.com/about/serghei-pogor). Published 18 October 2026. Page for people: https://curlshot.com/blog/visual-regression-testing-guide

Visual regression testing takes a screenshot of a page or component, compares it with an approved earlier screenshot (the baseline) and fails when the two differ by more than a set tolerance. Most tools compare pixel by pixel: Playwright uses the pixelmatch library, with a per-pixel colour threshold of 0.2 by default and no budget of differing pixels unless you set maxDiffPixels or maxDiffPixelRatio. The tests go flaky when the two screenshots were rendered differently for a reason other than your change, such as fonts, animations, late content or dates. The cure is one fixed environment and a deterministic page, not a looser threshold.

## Key facts

- A visual test has three images: the baseline (approved), the new screenshot, and the diff that paints the changed pixels.
- Playwright toHaveScreenshot defaults: threshold 0.2, animations disabled, caret hidden, scale "css". maxDiffPixels and maxDiffPixelRatio are unset.
- Tolerance comes in two kinds: how different one pixel may be (threshold, 0 to 1) and how many pixels may differ (a count or a ratio).
- A ratio of 0.01 on a 1280 x 720 screenshot allows 9,216 different pixels: sixteen missing 24 x 24 icons.
- Playwright stores one baseline per browser and platform, for example example-test-1-chromium-darwin.png, because rendering differs between them.
- Most flakiness is the environment or the page, not the comparison: fonts, animations, lazy content, dates, banners, window size.

## Baseline, new build, diff

The page illustrates a comparison with three wireframes of one 1280 x 720 screenshot (921,600 pixels): the baseline, a new build with three deliberate differences, and the diff, where the changed pixels are painted red. The pixel counts are arithmetic on the shapes as drawn.

| Difference | Changed pixels | Share of the frame | Smallest pixel budget that hides it |
| --- | --- | --- | --- |
| A missing icon. The 24 x 24 icon of the middle card did not load. | 24 x 24 = 576 | 0.063% | 576 |
| A button moved 8 pixels. The 160 x 48 button sits 8 pixels further right, so a strip changes on each side of it. | 2 x 8 x 48 = 768 | 0.083% | 768 |
| A heading wrapped to two lines. The first line lost 240 x 40 pixels of text and a second line of 280 x 40 appeared under it. | 240 x 40 + 280 x 40 = 20,800 | 2.26% | 20,800 |
| All three together | 22,144 | 2.40% | 22,144 |

A difference fails a test on its own when its changed pixels exceed the budget:

| Pixels allowed to differ | Differences still caught, out of 3 | Note |
| --- | --- | --- |
| 0 | 3 | Nothing may differ. This is where Playwright starts when no budget is set. |
| 100 | 3 | The value in the example in the Playwright docs. Room for a little rendering noise. |
| 600 | 2 | Loose enough to hide a missing 24 x 24 icon. |
| 1,000 | 1 | The moved button now passes as well. |
| 9,216 | 1 | 1% of the frame: the same as a ratio of 0.01 on 1280 x 720. |
| 25,000 | 0 | Everything passes, wrapped heading included. |

## How does a visual regression test run?

1. **Render the page.** Open the page or component in a real browser, at a fixed window size, with fixed data.
2. **Take the screenshot.** Wait until the page has stopped changing, then capture it.
3. **Compare with the baseline.** Load the approved screenshot of the same page, browser and platform.
4. **Compute the diff.** Count the pixels that differ by more than the colour threshold and draw them on a diff image.
5. **Pass or review.** Within the allowed number of pixels the test passes. Over it, a person looks at the diff.
6. **Approve a new baseline.** An intended change replaces the baseline. An unintended one is a bug to fix.

## How does screenshot comparison work?

### Pixel-by-pixel comparison

Both screenshots are decoded to raw pixels and each pixel is compared with the pixel at the same position in the other image. The two images must have the same width and height. pixelmatch returns the number of mismatched pixels and can write a diff image. [3]

### Per-pixel colour threshold

Two pixels count as different only when their colours are further apart than a threshold between 0 (strict) and 1 (lax). The threshold is about one pixel, not about a share of the image. `threshold`: 0.2 by default in Playwright, 0.1 by default in pixelmatch itself. [2][3]

### Allowed count or ratio of different pixels

The test passes when the number of different pixels stays within a budget, given as a count or as a share of all pixels. A ratio grows with the image: 0.01 of a 1280 x 720 screenshot is 9,216 pixels. `maxDiffPixels` and `maxDiffPixelRatio` in Playwright, both unset by default. [1][2]

### Anti-aliasing detection

The half-shaded pixels that smooth the edge of a letter or a curve differ slightly from one render to the next. The comparison recognises them from their neighbours and leaves them out of the count. pixelmatch ignores anti-aliased pixels unless `includeAA` is true, and draws them yellow in the diff. [3]

### Perceptual and structural methods

The image is compared in small windows or as a layout instead of as single pixels. SSIM (structural similarity) moves a window over both images and scores how alike each window is. `comparisonMethod: "ssim"` in jest-image-snapshot. Applitools Eyes has match levels such as Strict and Layout. [4][10]

### The Playwright options that decide a comparison

| Option of `toHaveScreenshot` | Default | What it does |
| --- | --- | --- |
| `threshold` | 0.2 | Acceptable perceived colour difference between the same pixel in both images, from 0 (strict) to 1 (lax). [2] |
| `maxDiffPixels` | unset | How many pixels may differ. `threshold` decides which pixels count as different. [1][2] |
| `maxDiffPixelRatio` | unset | Share of all pixels that may differ, between 0 and 1. [2] |
| `animations` | "disabled" | Stops CSS animations, CSS transitions and Web Animations. Finite ones are fast-forwarded to their end, infinite ones are reset to their initial state. [2] |
| `caret` | "hide" | Hides the blinking text cursor. [2] |
| `scale` | "css" | One image pixel per CSS pixel, so a high-density screen does not change the image size. [2] |
| `mask` | none | Locators to cover with a box (pink `#FF00FF` unless `maskColor` is set) before the capture. [2] |
| `stylePath` | none | A stylesheet applied while the screenshot is taken, to hide or freeze volatile elements. It reaches into Shadow DOM and inner frames. [1][2] |
| `fullPage` | false | Captures the whole scrollable page instead of the visible window. [2] |

## Why are visual regression tests flaky?

A visual test is flaky when the two screenshots were rendered differently for a reason other than the change under test. Each cause leaves a recognisable pattern in the diff.

| Cause | What the diff looks like | Fix |
| --- | --- | --- |
| Font rendering differs between operating systems | A thin outline of changed pixels around every letter on the page. Sometimes a line breaks at a different word and everything under it moves. | Create baselines and run comparisons in the same environment, for example the same container image locally and in CI. Keep one baseline per browser and platform: Playwright puts both in the file name, as in `example-test-1-chromium-darwin.png`. [1] |
| Animations and transitions | One element caught at another moment: half faded, a few pixels off, a spinner at another angle, a text cursor on or off. | Stop them before the capture. `toHaveScreenshot` does it by default: `animations: "disabled"` and `caret: "hide"`. In other tools, add a stylesheet that turns animations and transitions off. [2] |
| Lazy-loaded and late content | Empty boxes or placeholders where images should be. If the late content has a different height, everything below it is marked as changed. | Wait for the element you need to be visible, and scroll to content that only loads on scroll. `toHaveScreenshot` retakes the screenshot until two in a row match, which catches late paints but not content that never started loading. [2] |
| Timestamps and random data | A small block of changed text in the same place on every run: a date, "3 minutes ago", an order number, a rotating avatar. | Use fixed test data and a fixed clock (`page.clock.setFixedTime()` in Playwright). Cover what cannot be fixed with `mask`. [2][5] |
| Anti-aliasing | Single scattered pixels along curved and diagonal edges, rounded corners and text. | Leave anti-aliasing detection on (pixelmatch ignores those pixels unless `includeAA` is true) and keep a small colour `threshold` instead of a large pixel budget. [2][3] |
| GPU, hardware and headless mode | Faint, spread-out differences in shadows, gradients, images and canvas drawings. Nothing has moved. | Run on the same kind of machine in the same mode every time. Playwright lists the host OS, hardware, power source and headless mode among the things that change rendering. [1] |
| Scrollbars | A changed strip down the right edge. When the scrollbar takes up width, the layout is narrower and text wraps differently. | Capture in one browser mode only, or hide scrollbars with a stylesheet applied at capture time (`stylePath` in Playwright). [1] |
| Cookie banners, ads and chat widgets | A large block that is there in some runs and not in others, or the same slot with different content each time. | Block the requests of third parties in the test, hide the elements with a capture stylesheet (the Playwright docs hide every `iframe` this way) or cover them with `mask`. BackstopJS has `hideSelectors` for the same job. [1][6] |
| Viewport size and device scale factor | No useful diff at all: the two images have different sizes, or every element has moved because the layout changed. | Set the window size and the scale explicitly. pixelmatch needs both images to have the same dimensions, BackstopJS fails a size change by default (`requireSameDimensions`), and Playwright keeps one pixel per CSS pixel with `scale: "css"`. [2][3][6] |
| Hover and focus states | One button or link in its hover colour, because the mouse pointer happened to rest on it. | Move the pointer away before the capture. The Playwright docs use `await page.mouse.move(-1, -1)`. [1] |

## Visual regression testing tools

Listed without ranking, from what each project says about itself.

| Tool | What it is | What it compares | Where it runs |
| --- | --- | --- | --- |
| Playwright `toHaveScreenshot` | Open source, part of the Playwright test runner | A screenshot of a page or an element against a golden file kept next to the test, using pixelmatch. | Wherever the tests run. One baseline per browser and platform. [1][2] |
| Applitools Eyes | Hosted commercial service | Checkpoints sent by its SDKs against a baseline, by match level: Exact is pixel to pixel, Strict looks for visible changes, Layout checks the position of elements. | SDKs inside your existing tests, comparison on the service. [10] |
| Argos | Open source (MIT) platform | Screenshots uploaded from CI by its SDKs for Playwright, Cypress, Storybook, Vitest, WebdriverIO and Puppeteer, or by its CLI. | Screenshots are taken in your CI. Review happens in a build, with a check on the pull request. [13][14] |
| BackstopJS | Open source (MIT) | Screenshots of the URLs, viewports and selectors listed in a config file, taken with Puppeteer or Playwright and compared with Resemble.js. `misMatchThreshold` is 0.1 percent by default. | Your machine or CI, with a `--docker` flag for consistent rendering. Its README says the project is looking for a new maintainer. [6] |
| Chromatic | Hosted service, from the team behind Storybook | Snapshots of Storybook stories and of Playwright and Cypress tests, pixel by pixel against a baseline. | Cloud browsers: Chrome by default, with Firefox, Safari and Edge available. [7][8] |
| Loki | Open source (MIT) | Storybook stories. The diffing engine is pixelmatch by default, with GraphicsMagick and looks-same as alternatives. | Chrome in Docker, a local Chrome, AWS Lambda, the iOS simulator or the Android emulator. [12] |
| Percy | Hosted service, a BrowserStack product | A DOM snapshot captured by its SDK, rendered again across browsers and widths, and compared with the approved baseline. | The snapshot is taken in your tests. Rendering and comparison happen on Percy infrastructure. [9] |
| reg-suit | Open source (MIT) command line tool | Any directory of image files against the previous set, with an HTML report. `thresholdRate` and `thresholdPixel` are 0 by default. | Your CI. Plugins keep snapshots in S3 or Google Cloud Storage and notify GitHub, GitLab or Slack. [11] |

## What to test and what not to

**Test:**

- Design-system components, each in every state: default, hover, disabled, error, loading.
- The pages where a visual bug costs the most: home, pricing, sign-up, checkout.
- Pages rendered with fixed, seeded data, so the content is the same on every run.
- States that are hard to assert in code: empty lists, very long names, right-to-left text.
- A small set of window sizes, for example one phone width and one desktop width.

**Leave out, or mask:**

- Third-party embeds: maps, video players, ads, chat widgets, social feeds.
- Live data: dashboards, charts, counters, clocks, anything that reads "today".
- Pages inside an A/B test, where two visitors rightly see different things.
- A whole long page when one component would do: one changed line marks everything under it.
- Every browser at every size from day one. Each baseline is a file somebody has to review.

## How to set up visual regression testing

1. **Pick a few targets.** Start with a handful of components and two or three key pages. Add more once these run without false alarms.
2. **Fix the environment.** One browser version on one operating system image, used both to create baselines and to run the comparison in CI. [1]
3. **Fix the window.** Set the viewport width, height and scale in the test config, so every screenshot has the same dimensions.
4. **Make the page deterministic.** Seed the data, set a fixed clock, block third-party requests and wait for fonts and images before the capture. [5]
5. **Write the assertion.** In Playwright it is one line: `await expect(page).toHaveScreenshot()`. Mask whatever still changes. [1][2]
6. **Create the baseline.** Run the test once. Playwright reports that a snapshot does not exist, writes the screenshot as the golden file and compares against it from the next run on. Commit the file. [1]
7. **Set the tolerance last, and keep it small.** Stay on the defaults until you see real noise. Then add a small `maxDiffPixels`, not a broad ratio. [1]
8. **Review, then approve.** Read every failure as a diff. When the change is intended, update the baseline with `npx playwright test --update-snapshots` and review the new image in the pull request. [1]

```ts
import { test, expect } from '@playwright/test';

test('pricing page looks the same', async ({ page }) => {
  await page.goto('https://example.com/pricing');

  await expect(page).toHaveScreenshot('pricing.png', {
    fullPage: true,
    mask: [page.locator('.last-updated')],
    maxDiffPixels: 100,
  });
});
```

Animations are already stopped and the text cursor hidden: both are defaults of `toHaveScreenshot`. [2]

## Where a screenshot API fits

A screenshot API such as CurlShot gives captures of deployed addresses one consistent rendering environment: staging next to production, or the same live page every morning. It captures and does not compare. CurlShot has no diffing feature, so the two files go to a comparison tool such as pixelmatch or reg-suit.

```bash
curl "https://curlshot.com/api/v1/screenshot?access_key=YOUR_ACCESS_KEY\
&url=https://example.com\
&viewport_width=1280\
&viewport_height=720\
&device_scale_factor=1\
&wait_until=networkidle\
&reduced_motion=true\
&block_ads=true\
&block_cookie_banners=true\
&block_chats=true\
&time_zone=UTC" \
  --output staging.png
```

| Option | Value | Why it helps |
| --- | --- | --- |
| `viewport_width, viewport_height` | 1280, 720 | The same window on every capture, so both images have the same size and layout. |
| `device_scale_factor` | 1 | One image pixel per CSS pixel. |
| `wait_until` | networkidle | Capture after the page has stopped fetching things. |
| `reduced_motion` | true | Asks the page to skip animations, and stops CSS animations and transitions at the moment of an image capture. |
| `block_ads, block_cookie_banners, block_chats` | true | Removes the blocks that come and go between runs. Blocking is best-effort. |
| `time_zone` | UTC | Dates on the page do not depend on where the request was rendered. |

The response cache is off unless you ask for it, so each request is a fresh render. More: [the screenshot API](https://curlshot.com/tools/screenshot-api), [waiting and timing](https://curlshot.com/docs/waiting), [blocking](https://curlshot.com/docs/blocking).

## Glossary

- **Baseline:** The approved screenshot that new screenshots are compared with. It shows how the page is supposed to look.
- **Golden image:** Another name for the baseline. Playwright calls the stored file a golden file or a snapshot.
- **Snapshot:** A stored capture used in a test. In visual testing it is a screenshot. Some services store the DOM and its assets instead and render the image later.
- **Diff:** The result of a comparison: the count of pixels that differ, and an image with those pixels painted in a strong colour.
- **Threshold:** How much difference is tolerated. It comes in two kinds: how far apart two colours may be before a pixel counts as different, and how many different pixels a test may have.
- **Flaky test:** A test that passes on one run and fails on the next without any change to the code under test.
- **Anti-aliasing:** The half-shaded pixels a browser draws along the edge of letters and curves to make them look smooth. Their exact shade varies between machines.

## Questions

### What is visual regression testing?

It is automated testing of how a page looks. A tool takes a screenshot, compares it with a stored screenshot that was approved earlier (the baseline) and reports any difference above a tolerance, so that a change in code cannot alter the layout unnoticed.

### How does screenshot comparison work?

Both images are decoded to pixels and compared position by position. A pixel counts as different when its colour is further from the baseline than a threshold, anti-aliased edge pixels are usually ignored, and the test fails when the number or share of different pixels exceeds the allowed budget. The images must have the same dimensions.

### Why do visual regression tests fail when nothing changed?

Because the screenshot was rendered differently: another operating system draws fonts differently, an animation was caught mid-way, an image loaded late, a date changed, or a cookie banner appeared. Run baselines and tests in the same environment, stop animations, fix the data and the clock, and mask what still moves.

### What are the default thresholds in Playwright?

For toHaveScreenshot the per-pixel threshold defaults to 0.2 on a scale from 0 (strict) to 1 (lax). maxDiffPixels and maxDiffPixelRatio are unset by default and can be given per assertion or in the expect section of the Playwright config. Animations are disabled and the text caret is hidden by default.

### What is a good maxDiffPixels value?

As small as your noise allows. Start with none, look at the real diffs, and add a small count such as the 100 used in the Playwright docs. A ratio is riskier on large images: 0.01 of a 1280 x 720 screenshot is 9,216 pixels, enough to hide a missing icon or a moved button.

### How do I update a baseline in Playwright?

Run npx playwright test --update-snapshots. Playwright rewrites the stored screenshots with the current rendering. Do it in the same environment that runs the tests, and review the changed image files in the pull request like any other change.

### Does CurlShot compare screenshots?

No. CurlShot captures screenshots of public addresses in a consistent browser environment, with a fixed window size, blocking and waiting options. Comparing two captures is done with a separate tool, such as pixelmatch or reg-suit.

## Sources

1. Playwright, "Visual comparisons": https://playwright.dev/docs/test-snapshots (checked 11 October 2026)
2. Playwright, "PageAssertions: toHaveScreenshot": https://playwright.dev/docs/api/class-pageassertions (checked 11 October 2026)
3. Mapbox on GitHub, "pixelmatch README": https://github.com/mapbox/pixelmatch (checked 11 October 2026)
4. American Express on GitHub, "jest-image-snapshot README": https://github.com/americanexpress/jest-image-snapshot (checked 11 October 2026)
5. Playwright, "Clock": https://playwright.dev/docs/clock (checked 11 October 2026)
6. BackstopJS on GitHub, "BackstopJS README": https://github.com/garris/BackstopJS (checked 11 October 2026)
7. Chromatic, "Visual tests": https://www.chromatic.com/docs/visual/ (checked 11 October 2026)
8. Chromatic, "Browser support": https://www.chromatic.com/docs/browsers/ (checked 11 October 2026)
9. BrowserStack, "Percy SDK and screenshot capture workflow": https://www.browserstack.com/docs/percy/integrate/percy-sdk-workflow (checked 11 October 2026)
10. Applitools, "Match levels": https://applitools.com/docs/eyes/concepts/best-practices/match-levels (checked 11 October 2026)
11. reg-viz on GitHub, "reg-suit README": https://github.com/reg-viz/reg-suit (checked 11 October 2026)
12. Loki, "Configuration": https://loki.js.org/configuration.html (checked 11 October 2026)
13. Argos on GitHub, "Argos repository": https://github.com/argos-ci/argos (checked 11 October 2026)
14. Argos, "Getting started": https://argos-ci.com/docs/getting-started (checked 11 October 2026)

## Go deeper

- [Screenshot API](https://curlshot.com/tools/screenshot-api)
- [Waiting and timing](https://curlshot.com/docs/waiting)
- [Blocking ads, cookie banners and chats](https://curlshot.com/docs/blocking)
- [Reduced motion and emulation](https://curlshot.com/docs/dark-mode-and-emulation)

## Read next

- [Why website screenshots look wrong, and the fixes](https://curlshot.com/blog/why-website-screenshots-look-wrong.md): A field guide to 11 ways a website screenshot goes wrong, from blank lazy images to cookie banners and cut-off pages, with the cause and the fix for each.
- [Screenshot API vs headless browser: the real cost](https://curlshot.com/blog/screenshot-api-vs-headless-browser.md): A cost calculator with every assumption editable, the hidden work behind headless Chrome, and an honest checklist for build or buy.
- [Viewport sizes cheat sheet: phones to desktops](https://curlshot.com/blog/viewport-sizes-cheat-sheet.md): Viewport sizes in CSS pixels, screenshot sizes and device pixel ratios of iPhone, Pixel, Galaxy, iPad and laptop screens, drawn to scale, and the sizes to test.