What Is Visual Regression Testing? How Screenshot Diffing Catches Layout Breaks

Sometimes a WordPress plugin or theme update completes without a single PHP error, the admin screen reports success, and yet the live site looks broken the moment you open it. A changed CSS load order, an overwritten font declaration, a clashing class name — these can silently wreck the layout without ever touching an error log. Because nothing throws, nothing gets logged, and nobody notices until a visitor complains. Comparing a screenshot taken before an update against one taken after is one way to catch this “no error, but the page looks wrong” class of problem mechanically. This approach is generally known as visual regression testing.

Note: “regression” in software means something that used to work correctly breaking as a side effect of some other change. Visual regression testing detects that kind of breakage specifically in how a page looks, rather than in its logic or output values.

Why Naive Pixel-Perfect Comparison Doesn’t Work

It’s tempting to assume that comparing two screenshots just means checking every pixel and flagging any difference at all. In practice, this approach fails almost immediately.

Even two screenshots of the exact same unchanged page will differ at the pixel level — sub-pixel font rendering jitter, slightly different anti-aliasing, a few pixels of timing-dependent rendering drift. None of that reflects an actual content change. Add pages with genuinely dynamic content — a rotating “latest posts” widget, an ad slot, a timestamp — and the visible content itself legitimately changes between captures. Treating any pixel difference as significant means the check fires constantly on noise, and once a monitoring signal cries wolf often enough, people stop paying attention to it — which defeats the purpose of having it at all.

A Two-Layer Tolerance Design

core/visual_compare.py‘s compare_screenshots() addresses this with two layers of tolerance rather than one.

Before comparing, both images are resized to the same fixed resolution (1280×800), converted to grayscale, and passed through a light blur filter (ImageFilter.SMOOTH). Dropping color information focuses the comparison purely on luminance/layout patterns rather than color-tone noise, and the blur absorbs the kind of few-pixel rendering jitter that font anti-aliasing produces even between two captures of an unchanged page.

On top of that preprocessing, the actual judgment happens in two stages:

1. Per-pixel tolerance (_PIXEL_TOLERANCE = 25): two corresponding pixels are only counted as “changed” if their brightness difference exceeds 25 out of a possible 255. Small rendering variance gets filtered out here.
2. Aggregate change-rate threshold (DEFAULT_LAYOUT_THRESHOLD = 8.0, i.e. 8%): only once the proportion of pixels exceeding the per-pixel tolerance crosses this percentage of the whole frame does the function report a “layout change detected”.

Setting that aggregate threshold as high as 8% is a deliberate choice. A page section with genuinely dynamic content — a rotating banner, a “latest posts” block — can vary between captures without pushing the overall change rate past that bar, as long as it only occupies a portion of the frame. A real layout break, on the other hand, tends to shift a much larger fraction of the page and is what’s meant to cross the threshold. The two-layer design is built to ignore small, localized differences while still catching large, structural ones.

The Bug Where Renaming a Site Broke Comparison Entirely

This module also carries a lesson learned from an actual production incident. The “before” screenshot was originally saved under a filename derived from the site’s display name (before_{safe_name}.jpg). The problem: a site’s display name is something the user can change at any time during normal operation. Rename the site, and the filename the code now looks for no longer matches the filename that was actually saved — the previously captured “before” screenshot becomes permanently unreachable, and the comparison can no longer run at all.

The fix was to stop keying the filename off the mutable display name and instead key it off a stable, internally assigned identifier — a UUID (_id) that stays the same regardless of how many times the display name changes. take_before_screenshot() checks whether a valid _id is present (via site_paths.is_valid_site_id()) and uses the ID-based path when it is, falling back to the old name-based path only for legacy records that predate the _id field. From that point on, renaming a site as many times as you like no longer breaks the link between the site and its stored screenshots.

The lesson generalizes beyond screenshots: whenever a persisted file, cache key, or reference is tied to a value the user is free to edit — a display name, a label — that link breaks the moment the value changes. Keying persistent references off a stable internal identifier instead of a mutable display attribute avoids that whole class of bug.

Separating What’s Hard to Test From What Isn’t

The accompanying test file, tests/test_visual_compare_paths.py, takes a deliberate approach to a common testing problem: the actual screenshot capture runs a real headless browser via Playwright, and driving that deterministically in a unit test is difficult — network conditions and rendering output aren’t reliably reproducible, and mocking the browser risks testing something that doesn’t faithfully represent its real behavior.

Rather than attempting to mock browser capture, the test suite isolates the deterministic part of the module — how screenshot files are located and named — and tests only that. It verifies that a valid _id produces the correct ID-based path, that an invalid or missing _id falls back to the legacy name-based path, and that when both an ID-based file and a legacy file exist for the same site, the ID-based one is preferred. These checks use fake files containing nothing but a JPEG magic-byte header and a marker string, with no real image or browser involved — enough to prove which file the lookup logic actually picked.

Splitting a module into a part that depends on external, non-deterministic factors (a browser, a network, live page content) and a part that’s pure internal logic makes it possible to hold the second part to a much higher testing bar, even when the first part resists conventional unit testing.

Failing Without Crashing

compare_screenshots() never raises an exception when a screenshot file is missing or when the image library (Pillow) isn’t installed — it returns a dictionary with changed, diff_pct, and error fields instead. Callers can inspect that dictionary and decide to treat “comparison failed” as “no warning” rather than aborting. Visual comparison is a secondary check layered on top of a maintenance run, not the main task — letting a failure in this check halt the entire maintenance process would be the wrong trade-off, so the function surfaces the failure as data and lets the caller keep going.

Summary

Turning something as hard to quantify as “how a page looks” into an automated check comes down to two design questions: what threshold counts as a meaningful change, and what identifier stays stable enough to key persisted data on. Both tend to become obvious only after something breaks in production — which is exactly why it’s worth designing generous thresholds and rename-proof identifiers in from the start.

The idea of “carving out the deterministic part to test” connects to a broader point about the shape of tests in general. Related post: Unit Tests vs. Regression Tests: Why the Same Feature Gets Tested Twice walks through how a regression guard like this post’s `test_visual_compare_paths.py` differs from an ordinary unit test that verifies a function’s spec.