Visual regression testing, explained with pictures

Try it: how much tolerance hides a real bug?

A pixel budget that is loose enough to ignore rendering noise can also hide a real change. Slide between the two screenshots, then raise the budget and watch the three differences stop counting, smallest first.

The frame is 1280 x 720 = 921,600 pixels, and the counts are arithmetic on the solid shapes of the drawing. A ratio of 0.01 allows 921,600 x 0.01 = 9,216 pixels, which is 16 missing icons of 24 x 24.

How does screenshot comparison work?

Screenshot comparison is a pixel-by-pixel check with two tolerances on top: how different one pixel may be, and how many pixels may be different. Playwright does it with the pixelmatch library.^[1]

1. Baseline

   New

   8 differ

   Pixel-by-pixel comparison

   Both screenshots are decoded to raw pixels and each pixel is compared with the pixel at the same position in the other image. The two images must have the same width and height.

   pixelmatch returns the number of mismatched pixels and can write a diff image.^[3]

2. Counts as the sameCounts as different

   Per-pixel colour threshold

   Two pixels count as different only when their colours are further apart than a threshold between 0 (strict) and 1 (lax). The threshold is about one pixel, not about a share of the image.

   threshold: 0.2 by default in Playwright, 0.1 by default in pixelmatch itself.^[2][3]

3. 3 of 100: ratio 0.03

   Allowed count or ratio of different pixels

   The test passes when the number of different pixels stays within a budget, given as a count or as a share of all pixels. A ratio grows with the image: 0.01 of a 1280 x 720 screenshot is 9,216 pixels.

   maxDiffPixels and maxDiffPixelRatio in Playwright, both unset by default.^[1][2]

4. An edge

   Not counted

   Anti-aliasing detection

   The half-shaded pixels that smooth the edge of a letter or a curve differ slightly from one render to the next. The comparison recognises them from their neighbours and leaves them out of the count.

   pixelmatch ignores anti-aliased pixels unless includeAA is true, and draws them yellow in the diff.^[3]

5. Scored window by window

   Perceptual and structural methods

   The image is compared in small windows or as a layout instead of as single pixels. SSIM (structural similarity) moves a window over both images and scores how alike each window is.

   comparisonMethod: "ssim" in jest-image-snapshot. Applitools Eyes has match levels such as Strict and Layout.^[4][10]

The Playwright options that decide a comparison

These are options of toHaveScreenshot, with the defaults from the Playwright documentation. Each can be set per assertion, or once under expect.toHaveScreenshot in the config.^[1]

Visual regression testing tools

Visual regression testing tools differ in three things: what they capture, how they compare, and where the browser runs. The table lists eight of them without ranking, using what each project says about itself.

Prices and plan limits change often and are left out on purpose. Check the site of each project.

How to set up visual regression testing

Setting up visual regression testing is eight steps, and the order matters: make the screenshot repeatable before you write the first assertion, and choose a tolerance last.

import { test, expect } from '@playwright/test';

test('pricing page looks the same', async ({ page }) => {
  await page.goto('https://example.com/pricing');

  await expect(page).toHaveScreenshot('pricing.png', {
    fullPage: true,
    mask: [page.locator('.last-updated')],
    maxDiffPixels: 100,
  });
});