Unicode Normalization (NFC vs NFD): Why Identical-Looking Japanese Strings Don’t Match

A filename looks exactly the same on screen, yet a program says the two strings differ. A search finds nothing. Moving a file name created on a Mac to another environment can trigger exactly this kind of mismatch.

The cause is often Unicode normalization. This article explains why one character can be encoded in more than one way, how NFC and NFD differ, and what actually goes wrong, with results from Python 3.12.8.

Note: The tests ran on macOS 26.4.1 (APFS) with Python 3.12.8. Behavior varies by environment, so read the results as one example.

Two ways to write the same “が”

The voiced kana “が” can be represented two ways in Unicode:

• One code point: が (U+304C)
• Two code points: か (U+304B) plus a combining voiced mark (U+3099)

They look identical but the underlying codes differ. Normalizing to the first form is NFC (composed); to the second, NFD (decomposed).

I converted “ガイド資料.txt” to both forms and checked the length:

The two voiced kana add two code points and more bytes. And the two strings are not equal under ==.

import unicodedata as u

a = u.normalize("NFC", "がく")
b = u.normalize("NFD", "がく")
print(a == b)                      # False
print(len(a), len(b))              # 2 3
print(u.normalize("NFC", b) == a)  # True

Normalize both sides to one form before comparing and they match. That is all “normalization” means.

What happens with filenames

File systems treat names differently:

• The older macOS file system (HFS+) converted names to an NFD-style form when storing them.
• The current APFS keeps the name as given, but lookups do not distinguish between the forms.
• Common Linux file systems treat names as raw bytes and do not normalize.

On APFS, I created a file with an NFD name and then checked it with the NFC name:

import os, tempfile, unicodedata as u

d = tempfile.mkdtemp()
nfc = u.normalize("NFC", "がく")
nfd = u.normalize("NFD", "がく")
open(os.path.join(d, nfd), "w").write("x")

names = os.listdir(d)
print([(len(n), n == nfc, n == nfd) for n in names])
# [(3, False, True)]   <- stored as NFD
print(os.path.exists(os.path.join(d, nfc)))
# True                 <- found with the NFC name too

So the OS treats them as the same file, while a string comparison in your own code does not.

Where the mismatch hurts

• Comparing a name from a directory listing with one written in a config or log
• Transferring files to another OS and matching names
• Using a name as a dictionary or database key
• Duplicate detection showing two “identical” names

The shared cause: the strings were not normalized before comparison.

Fix: normalize once, at the boundary

Normalize incoming strings (filenames, input, config values) as they arrive:

import unicodedata

def norm(s: str) -> str:
    return unicodedata.normalize("NFC", s)

• NFC is the usual choice, since many web and data conventions assume it.
• Normalize only for comparison, matching and keys. When opening a file, use the original name the OS gave you.
• Keep the original name and the matching key as separate values so the original is never altered.

Watch out for NFKC

There are also compatibility forms, NFKC / NFKD, which fold look-alike characters together:

print(u.normalize("NFKC", "ﾊﾟ") == "パ")  # True (half-width kana become full-width)
print(u.normalize("NFC",  "ﾊﾟ") == "パ")  # False
print(u.normalize("NFKC", "①㈱"))          # 1(株)

This is handy for half-width kana, but “①” becomes “1” and “㈱” becomes “(株)”, so information is lost. Use NFC when the original must be preserved, such as filenames, and NFKC when absorbing input variation for search.

Summary

• One visible character can be a single code point (NFC) or a combination (NFD).
• The lengths and bytes differ, so == fails.
• APFS keeps the stored name but ignores the difference in lookups, so the OS finds the file while your own comparison does not.
• Normalize incoming strings to NFC before comparing; keep the original separately.
• NFKC changes the content, so choose it on purpose.

Check whether your own tools compare filenames as plain strings. Related reading: UTF-8 vs cp932 for encoding basics, and atomic writes for interrupted saves.