Look through a maintenance tool’s test suite long enough and you’ll run into a small puzzle: a function already has a test, so why does another file add a second one for what looks like the same behavior? Two tests that appear to cover the same ground can actually exist for entirely different reasons.
This post walks through the distinction between “unit tests” and “regression tests,” using real test code from this project as the example.
Unit tests: verifying a function in isolation
Note: a unit test checks the smallest testable piece of a program — a function, class, or method — independently from the rest of the system.
A unit test’s job is simple: feed a function a range of plausible inputs and confirm the output matches expectations. It’s naturally written right alongside the implementation, or shortly after.
Take the helper functions that diagnose and fix SSH private key permission issues:
class TestIsPermissionError(unittest.TestCase):
def test_openssh_too_open(self):
msg = "Permissions 0644 for '/Users/x/.ssh/id_rsa' are too open."
self.assertTrue(key_perms.is_permission_error(msg))
def test_windows_openssh(self):
msg = "Permissions on the private key file 'C:\\Users\\x\\.ssh\\key' are too open."
self.assertTrue(key_perms.is_permission_error(msg))
def test_case_insensitive(self):
self.assertTrue(key_perms.is_permission_error("ARE TOO OPEN"))
Each test method feeds is_permission_error() a different plausible input — a typical OpenSSH message, the Windows-flavored wording, a case variation — and checks the return value. The POSIX (Mac/Linux) side runs real chmod calls; the Windows icacls path is mocked out since it can’t run in this CI environment.
The question this test is asking is: “does this function behave as designed?” It verifies a contract, and it can be written the moment the function exists, independent of anything that happens later in production.
Regression tests: sealing off an incident that already happened
Note: “regression” means slipping backward. A regression test checks that a bug fixed in the past hasn’t quietly come back with a later code change.
Regression tests are a different animal. They aren’t triggered by “let’s verify the spec” — they’re triggered by “this actually broke, in a real environment, once.”
Here’s an incident from development on v1.6.11. A pre-build version-consistency script (tools/bump_version.py) printed a success message containing an emoji (✅). On Mac, the console defaults to UTF-8, so nothing ever went wrong there. But on Windows with a Japanese locale, when this script ran as a subprocess with its output piped, Python defaulted to the locale encoding (cp932) — which can’t represent that emoji — and the script crashed with a UnicodeEncodeError. Worse, the caller (build_app.py) misreported that crash as a “version mismatch,” making the real cause harder to trace.
That incident produced tests/test_windows_cp932_safety.py, built as two layers:
Layer 1 (static check): scan the source of any script that runs as a subprocess during build/release, character by character, and confirm nothing in it fails to encode as cp932.
CP932_CRITICAL_SCRIPTS = [
"tools/bump_version.py",
]
def test_critical_scripts_are_fully_cp932_encodable(self):
for rel in CP932_CRITICAL_SCRIPTS:
with open(os.path.join(ROOT, rel), encoding="utf-8") as f:
text = f.read()
for ch in text:
try:
ch.encode("cp932")
except UnicodeEncodeError:
... # record the failure
Layer 2 (behavioral check): actually run the target script as a child process with PYTHONIOENCODING=cp932 set, and confirm it exits cleanly (return code 0) without raising UnicodeEncodeError. A static character scan alone can’t guarantee the script won’t crash at runtime, so the behavior itself gets checked separately.
The question this test asks isn’t “is this function correct?” — it’s “has that specific incident happened again?” The function itself was already fixed by the time this test was written, and the behavioral layer confirms the fix works. The reason this test stays in the suite indefinitely is to catch a future edit that quietly reintroduces an emoji into an output message.
Why a “correct” function still broke
The bump_version.py emoji output wasn’t a logic bug in isolation — printing a checkmark on success did exactly what it was supposed to do. The problem lived in an assumption about which environment, and by what path, that output would be read.
A unit test verifies a function’s contract in the abstract: given this input, does it return that? Looked at on its own, the emoji output passes such a test easily. The failure only surfaces when three conditions line up at once — a specific locale (cp932), output flowing through a pipe, and the call happening from inside a subprocess. No amount of exhaustively unit-testing the function’s normal logic would have caught that combination.
That’s exactly why an incident that actually happened, across environments, gets frozen into an automated test rather than left as a lesson learned once and then forgotten.
What the test filenames themselves signal
This repository’s test suite includes files like test_stability_v47.py and test_db_backup_v48.py, named after internal development round numbers. The header of test_db_backup_v48.py reads, in part:
Issue #299 (fixed 2026-04-24): regression guard for the
os.path.join(local_site_backup_dir, backup_file) bug — backup_file had
become a remote absolute path, so the second argument was treated as
absolute and the first argument was discarded entirely.
os.path.join() has a documented behavior: if a later argument is an absolute path, everything before it is discarded and only that absolute path is returned. That’s not a bug — it’s the documented contract. The actual incident was that the calling code assumed the string it passed in would always be a relative filename, when in practice a remote absolute path made it through, silently discarding the intended local backup directory.
A test carrying an issue number or round number in its filename or header comment is a signal that it exists to guard against a specific past incident, not to exercise the general behavior of a module. By contrast, a file named purely after the module it covers — like test_key_perms.py — is a unit test aimed at verifying that module’s spec.
The connection to visual regression testing
An earlier post on this blog, What Is Visual Regression Testing? How Screenshot Diffing Catches Layout Breaks, covers a variant of this same idea. That post froze “has the layout broken” into a pixel-comparison check; the cp932 example here freezes “has a specific piece of logic broken under a specific environment” into a code-execution check. The verification mechanism differs — image diffing versus running a script — but the underlying design is the same: take an incident that already happened once, and turn it into something that can be re-checked automatically, indefinitely.
Summary
Unit tests confirm that a function or module behaves according to spec, written around the same time as the implementation. Regression tests exist to make sure an incident that already happened once, in a real environment, doesn’t quietly happen again after a future code change — and they’re written after the fact. Both take the shape of “a test,” but they’re born at different moments and serve different purposes. When a test suite has multiple files touching what looks like the same feature, it’s often carrying two distinct kinds of intent at once: verifying a spec, and remembering an incident.