Atomic writes — how tempfile + os.replace prevent corrupted JSON

What happens if the power cuts out while a process is writing to a config file? Or if antivirus software on Windows briefly locks a file mid-write? If you naively overwrite a file with open(path, 'w'), whatever partial content existed at the moment of interruption is what remains on disk. For JSON, that usually means broken syntax — json.load() throws on the next startup, and the entire configuration is effectively lost. This article walks through a standard technique for preventing that: writing to a temporary file first, then swapping it in atomically.

Note: “Atomic” here means an operation either completes entirely or doesn’t happen at all — there’s no partial, observable in-between state. It’s the same sense of the word used for database transactions.

Why direct overwrites are dangerous

open(path, 'w') effectively truncates the file first and then writes the new content. If the process is interrupted during that window, the file is left empty or holding incomplete content.

# Dangerous: a crash mid-write leaves a corrupted file behind
with open('config.json', 'w') as f:
    json.dump(data, f)  # what if this gets interrupted?

The causes vary: a kill -9, a power outage, antivirus software briefly blocking file access on Windows, or a backup tool grabbing the file mid-write. This rarely reproduces during local development, but in a long-running production environment, it will eventually happen with near certainty.

The fix: write to a temp file, then swap it in

The core idea is simple. Never touch the target file directly. Write the complete new content to a temporary file first, confirm that write fully succeeded, and only then replace the target file with that temp file.

import json
import os
import tempfile

def atomic_write_json(filepath, data):
    dirpath = os.path.dirname(os.path.abspath(filepath)) or '.'
    fd, tmp_path = tempfile.mkstemp(dir=dirpath, suffix='.json.tmp')
    try:
        with os.fdopen(fd, 'w', encoding='utf-8') as f:
            json.dump(data, f, indent=4, ensure_ascii=False)
        os.replace(tmp_path, filepath)
    except Exception:
        os.unlink(tmp_path)
        raise

tempfile.mkstemp() generates a collision-free temporary filename and returns its file descriptor. The dir=dirpath argument matters here: placing the temp file in the same directory as the target file ensures the following os.replace() call stays within a single filesystem.

Why os.replace() is atomic

The core of this pattern is the final os.replace(tmp_path, filepath) call. Python’s official documentation states that os.replace() “will be an atomic operation on Unix” and behaves atomically on Windows as well, as long as both paths are on the same filesystem.

On POSIX systems (Linux/macOS), this maps to the rename(2) system call. At the kernel level, swapping a directory entry is a single indivisible operation. There is no intermediate state to observe — from the outside, the file is either in its pre-swap or post-swap state, never something in between.

On Windows, Python 3.3+ implements this so that overwriting an existing file is handled atomically (roughly equivalent to calling MoveFileEx with the MOVEFILE_REPLACE_EXISTING flag).

Thanks to this property, if the process dies in the middle of json.dump(), only the unnamed temp file is affected — the real config file remains untouched in whatever valid state it was in before.

Forcing persistence to disk with os.fsync()

Even with an atomic os.replace(), there’s a subtler risk: the content written by json.dump() might still be sitting in the OS page cache rather than physically on disk when power is lost. In that case the rename itself could complete, but the renamed file’s contents might not reflect what was actually written. os.fsync() addresses this.

with open(tmp_path, 'w', encoding='utf-8') as f:
    json.dump(data, f, ensure_ascii=False, indent=2)
    f.flush()
    try:
        os.fsync(f.fileno())
    except OSError:
        pass
os.replace(tmp_path, filepath)

f.flush() only pushes Python’s internal buffer out to the OS — it may still sit in OS-level cache. os.fsync() goes a step further and asks the OS to wait until the data is actually written to physical disk. Combining both steps ensures the temp file’s content is durably on disk by the time os.replace() runs.

Cleaning up after a failed write

If the write to the temp file itself fails partway through (disk full, permission error, etc.), os.replace() is never reached, so the target file is unaffected. But the half-written temp file is left behind on disk. Left unattended, these accumulate as clutter, so it’s worth deleting them explicitly on failure.

except Exception:
    try:
        os.unlink(tmp_path)
    except OSError:
        pass
    raise

The nested try accounts for the possibility that os.unlink() itself fails (already deleted, no permission, etc.). Re-raising the original exception (raise) ensures the caller still learns that the write failed.

Guarding against leftover .tmp files

Since os.replace() consumes the temp filename and swaps it into the target name on success, .tmp files normally don’t linger under regular operation. But in an extreme edge case — the process gets kill -9‘d in the narrow window right after the temp file is fully written but right before os.replace() is called — a stray .tmp file could theoretically be left behind.

This leftover doesn’t threaten data integrity (the real file stays intact), so it’s not urgent. But left unchecked over a long-running install, it quietly wastes disk space. A simple startup routine that scans the config directory for .tmp files older than some threshold (say, one hour) and deletes them closes this gap.

import time

def cleanup_stale_tmp_files(directory, stale_seconds=3600):
    now = time.time()
    for name in os.listdir(directory):
        if not name.endswith('.tmp'):
            continue
        path = os.path.join(directory, name)
        if now - os.path.getmtime(path) > stale_seconds:
            os.remove(path)

Summary

Replacing “write directly to the target file” with “write completely to a temp file, then atomically rename it” eliminates the observable in-between state entirely — and with it, the whole class of bugs where a config file ends up corrupted. The implementation overhead is modest, so it’s worth adopting from the start for any file you read and write frequently, like config or cache files.

This shares a family resemblance with another kind of pitfall that stays invisible during Mac development and only surfaces in real-world operation: cross-platform file locking in Python, which handles mutual exclusion between multiple processes. Atomic writes and file locking are actually complementary — in an environment where multiple processes might write concurrently, it’s worth combining both.