Skip to content
Engineering Notes

Config File Schema Versioning and Migration: Keeping Updates from Breaking Settings

You update an app and the settings file created by the previous version no longer loads. A key was renamed, a value changed shape. This is a common failure for any software that keeps a config file.

This article covers a standard design for it: give the config file a schema version, and convert old formats to the new one step by step (migration). The examples ran on Python 3.12.8, but the idea is independent of any library and works the same for JSON or YAML.

Note: “Schema” here means the rules for the file’s structure (which keys exist and what types the values have). The schema number is separate from the app’s own version number.

Why config files break

Code is replaced on every update. A config file, on the other hand, stays on the user’s machine in its old shape. New code assumes the new shape, so a mismatch causes:

  • A renamed key, so the value cannot be found (KeyError)
  • A changed value type, so it loads but cannot be used (TypeError, etc.)
  • No error at all, but the settings silently fall back to defaults and the user’s choices are lost

The third one is the hardest to notice. No exception is raised, so by the time someone reports “my settings disappeared,” the cause is hard to trace.

Experiment: changing the format without a version

Suppose the format changes like this:

  • Version 1: {"email": "...", "timeout": 20}
  • Version 2: the email moves under notify
  • Version 3: timeout splits into connect and read ({"connect": 5, "read": 20})

Reading a version 1 file directly with version 3 code gave:

old = {"email": "a@example.com", "timeout": 20}
old["notify"]            # KeyError: 'notify'
old["timeout"]["read"]   # TypeError: 'int' object is not subscriptable

Both at least stop with an exception. If the code used .get() with defaults instead, the settings would be ignored without any error.

The basic pattern: a version number plus a chain of converters

  1. Store schema_version in the file (treat a missing value as 1).
  2. Write one converter per adjacent pair of versions: 1→2, 2→3.
  3. On load, apply the converters in order from the file’s version up to the current one.
CURRENT = 3
def m1_to_2(c): c["notify"] = {"email": c.pop("email", "")}; return c
def m2_to_3(c): c["timeout"] = {"connect": 5, "read": c.pop("timeout", 30)}; return c
STEPS = {1: m1_to_2, 2: m2_to_3}

def load(raw):
    c = json.loads(raw); v = c.get("schema_version", 1)
    while v < CURRENT:
        c = STEPS[v](c); v += 1
    c["schema_version"] = CURRENT
    return c

Loading the version 1 file produced:

{'notify': {'email': 'a@example.com'}, 'timeout': {'connect': 5, 'read': 20}, 'schema_version': 3}

The user’s email and timeout=20 carried over, and the newly added connect received a default. A version 2 file only goes through 2→3.

Why chain adjacent steps

You could write a separate “to latest” converter for every old version, but the number of combinations grows with each release. With adjacent steps, adding a new version means adding exactly one function, and past converters are never edited.

Properties worth keeping

1. Loading twice gives the same result (idempotence)

Loading a file that is already current should change nothing. In the experiment, passing the converted result through load again produced an identical result (True). That makes it safe for migration to run on every startup.

2. Do not silently read configs that are too new

An older app sometimes meets a config written by a newer one (a rollback, or an old build on another machine). Guessing at an unknown format and saving it back can destroy data the newer version needs. In the experiment, a file with schema_version: 4 was stopped explicitly:

RuntimeError: newer config v4 > supported v3

Whether to refuse or to open read-only is a design choice, but do not overwrite silently.

3. Keep the pre-migration content

Once migration runs and the file is saved, its shape changes. Copying the original before saving lets you recover if a converter turns out to be wrong. Testing converters is of course also effective.

Practical notes

  • Bump the number only when the structure changes. Adding a key that can simply default may not need a new version.
  • Renames, removals, and type changes always need a converter. These are changes that make old files unreadable as they are.
  • Give each converter a test using a sample of the old format. “A version 1 file loads as version 3” is a cheap check that catches later breakage.
  • Do not rewrite old converters. Editing a past step makes results differ from users who already migrated. Correct mistakes in the next version instead.

Summary

  • Config files stay in their old shape on the user’s machine after the code updates.
  • Store schema_version and chain adjacent converters so the design scales as versions accumulate.
  • Make migration safe to run repeatedly, and never silently overwrite a config that is too new.
  • Keep the pre-migration content and write tests with old-format samples.

Take a look at whether your own tool’s config file has an item that records its version. If not, the next time you change the format is the best moment to add one. Related reading: UTF-8 vs cp932 for config files that fail to load because of encoding, and atomic writes for files corrupted by an interrupted save.