An investigation agent flagged “the license API PHP returns Japanese-hardcoded messages” and we sat down to fix it. But something felt off the moment we opened the file — the version running on the production server didn’t match the latest commit in the local repo. Stranger still, production had more recent features than our local checkout.
A bit of digging turned up the truth: months earlier, someone had hot-patched the production file in response to a different user issue, and that change had never been committed back to git. This post walks through how we detected that drift, and the two-stage strategy we used to merge production back into the local repo safely.
How this regression silently slips in
If we’d written the fix on top of our local repo and uploaded it to production, here’s what would have happened: all the production-only improvements get overwritten and quietly disappear.
In our case, the production file had a half-year-old language-handling addition for the “Early Bird Bonus” feature — when a USD customer buys, client_name is set to 'Early Bird Bonus'; for JPY customers it’s '早期利用特典'. None of that existed in our local git. A normal PR-merge-and-deploy cycle would have silently rolled back the Early Bird i18n logic, regressing English users’ display back to Japanese.
Catching this was half luck. Opening the file to start the fix, I noticed code I didn’t recognize, ran git blame, and the lines were nowhere in git history. That’s when alarm bells went off.
Two-stage rollforward — make production the source of truth first
The strategy we landed on was a two-stage merge.
Stage 1 (rollforward sync): Pull the production file straight into the local repo. Apply the diff in the “production → local” direction, not the other way. After this, the local repo’s HEAD matches what’s actually running on production.
# Pull the production file into the local repo
scp -i ~/.ssh/key layer2024@host:wpmm.jp/public_html/license/api/register_free.php \
/tmp/register_free_prod.php
# Record its MD5 first (we'll need this later for upload verification)
md5 /tmp/register_free_prod.php
# → 94b9c5a7...
# Replace local file with the production copy, then commit
cp /tmp/register_free_prod.php server/wpmm-license/api/register_free.php
git add server/wpmm-license/api/register_free.php
git commit -m "rollforward sync: pull v1.5.9 hotpatch from production"
The key move here is “commit the production state as a separate commit, before doing any feature work” — that single rollforward commit makes every subsequent diff a clean delta against production reality.
Stage 2 (additive): With the local repo now in sync, add only the new functionality as a follow-up commit. Don’t touch any existing code in the Early Bird section.
# Build the stage-2 commit
# - Keep the existing Early Bird block (lines 71-111) byte-for-byte
# - Add only the new lines (i18n fallback, lang query addition)
git diff HEAD~1 -- server/wpmm-license/api/register_free.php
# Eyeball the output: confirm zero deletion lines
“Zero deleted lines” is a mechanical safety check you can apply. The diff’s structure itself guarantees you’re not changing production behavior, just extending it.
Use MD5 to prove byte-for-byte preservation
If you’re going to claim “byte-for-byte unchanged,” prove it quantitatively. Before and after upload, we md5-hashed the preserved section to be sure.
# Hash just the preserved section (Early Bird, lines 71-111)
sed -n '71,111p' server/wpmm-license/api/register_free.php | md5
# Expected: matches the same range in the original production file
# Verify again after upload
ssh user@host 'sed -n "71,111p" wpmm.jp/.../register_free.php | md5'
INSERT INTO licenses statement, bind order, $client_name_label = $is_en ? 'Early Bird Bonus' : '早期利用特典' ternary, try/catch, period logic — all of it can be confirmed structurally unchanged. The hash check is mechanical proof.
Closing — “my git history is canonical” is a dangerous assumption
Three principles worth keeping from this round:
- Make “diff from the live server” a habit before starting work. Your local git is not necessarily the latest truth. Especially in projects with a hot-patch culture, production may carry changes that aren’t in your repo. Always
scpthe production file and diff against local before touching it - When you find drift, merge in two stages. Stage 1 (“rollforward sync”) makes production the source of truth in your repo. Stage 2 (“additive”) layers new features on top. Splitting the merge this way structurally prevents “regressing production improvements”
- Verify preservation with md5, not just words. Don’t just write “this section is unchanged” — extract the preserved range with
sed -n, hash it, and confirm before/after upload. Combined with the “zero deletion lines” diff check, you get two independent guarantees
The big lesson from this round was the principle of “always suspect that the local repo has drifted from production.” In projects with manual deployment or hot-patch traditions, this is a hard reality to escape. If you’ve gotten into the habit of treating git as the latest truth, a regression like this one is waiting to happen.
A one-minute scp and diff before starting can save hours of incident response later. Worth building into the muscle memory.