In May 2026, bash commands sent over SSH stopped working correctly on WordPress sites hosted on Sakura Internet. The root cause: Sakura’s default login shell is csh, which can’t interpret bash syntax. We patched the codebase, consolidating all bash-syntax c.run() calls behind a _safe_run helper, and the problem went away. (That original incident is documented in the csh login-shell portability article.)
Then in June 2026, a report came in: DB backups were failing every single time on all Sakura-hosted sites with an “SSH update error.”
Why the same problem came back
The cause: a new feature had walked into the same trap.
The v1.6.9 development cycle added progress monitoring to DB backups — sampling the backup file size every 30 seconds while the dump ran, then writing a one-line summary after completion. The script looked like this:
wp db export /path/backup.sql 2>&1 &
PID=$!
while kill -0 $PID 2>/dev/null; do
echo "size=$(wc -c < /path/backup.sql)"
sleep 30
done
wait $PID; exit $?
The script is correct bash. The problem is that paramiko’s exec channel runs commands through the user’s login shell on the server. On Sakura, that login shell is csh. csh can’t interpret these constructs:
| Syntax | What csh does |
|---|---|
2>/dev/null |
Ambiguous output redirect — or silently returns empty output (most dangerous) |
$! |
Can’t hold the background process ID |
while ...; do ... done |
for: Command not found style error |
VAR=val |
Variable assignment doesn’t work |
The earlier fix covered the existing code. The new feature called c.run() directly — outside the scope of that fix — and had no guard at all.
The first fix attempt: base64 encoding
The first approach: base64-encode the script body, send it as a string, and have the server decode and run it through /bin/sh.
import base64
def wrap_remote_sh_v1(script):
encoded = base64.b64encode(script.encode()).decode()
return f"echo {encoded} | openssl base64 -d -A | /bin/sh"
echo <base64> runs fine in any shell. The server decodes with openssl base64 -d and pipes the output to /bin/sh. The login shell only needs to handle echo, a pipe, and an external command call — nothing bash-specific.
Testing on a Sakura instance confirmed the progress script ran correctly.
It broke again on heteml
While validating the fix across environments, a new fact emerged:
heteml (ssh-layer.heteml.net) has neither openssl nor a base64 command.
The base64 approach depends on openssl base64 -d. On heteml, that’s command not found. The Sakura problem would be solved, but heteml would fail completely. Dependencies on external commands break silently on servers you haven’t tested against.
The final design: /bin/sh -c
To eliminate external-command dependencies entirely, the approach shifted to passing the script directly as an argument to /bin/sh -c.
import shlex
def wrap_remote_sh(script):
if '\n' in script or '\r' in script:
raise ValueError(
"wrap_remote_sh requires a single-line script "
"(csh single-quotes cannot span newlines); "
"join statements with ';' instead"
)
return "/bin/sh -c " + shlex.quote(script)
shlex.quote() is Python’s standard library function for POSIX single-quoting. Embedded ' characters become '\''. The result looks like this:
wrap_remote_sh("echo hi && wc -c < /tmp/file")
# → "/bin/sh -c 'echo hi && wc -c < /tmp/file'"
When csh receives this via the exec channel, it treats the single-quoted content as a single token and hands it to /bin/sh. The shell that interprets 2>/dev/null, $!, and while...do...done is /bin/sh, not csh. The login shell is irrelevant.
Zero external-command dependencies. /bin/sh exists on every server.
One constraint applies: the script must be a single line. csh single-quotes can’t span newlines — a multi-line script will produce Unmatched '. Multi-line logic needs to be joined with semicolons into a single line (note: do; is a shell syntax error in sh; use do <cmd> to connect statements inside a loop). Violations throw a ValueError so the issue is caught at development time, not at runtime.
Verified across three environments
After the fix, we verified on three environments:
| Environment | Login shell | openssl / base64 | Result |
|---|---|---|---|
| Sakura | csh | present | ✅ Progress script ran to completion |
| Xserver | bash | present | ✅ Same |
| heteml | bash | absent | ✅ No external commands needed |
The base64 approach had failed on heteml. The /bin/sh -c approach passed all three.
A static test to prevent it from coming back
Having wrap_remote_sh available doesn’t guarantee it gets used in the next new feature — that’s exactly the pattern that caused this regression in the first place.
The test suite now includes a static check: scan maintenance_agent.py for any c.run() call that contains csh-unsafe syntax without going through wrap_remote_sh:
def test_no_csh_unsafe_bare_crun_in_maintenance_agent(self):
src = (PROJECT_ROOT / "maintenance_agent.py").read_text(encoding="utf-8")
for i, line in enumerate(src.splitlines(), 1):
if "c.run(" not in line:
continue
if "wrap_remote_sh" in line:
continue
# csh-unsafe markers: redirects, command substitution, loops
if re.search(r'2>|`\$!`|\$\(|while |for.*in.*do', line):
self.fail(f"line {i}: csh-unsafe c.run without wrap: {line.strip()}")
If a future change adds a c.run() containing 2> without wrapping it, the test fails. The check is mechanical — no human review required.
Summary
| Aspect | Detail |
|---|---|
| Problem | A new feature called c.run() directly, outside the scope of the earlier csh fix |
| Root cause | paramiko runs commands through the server’s login shell; Sakura’s default login shell is csh, which can’t parse bash syntax |
| First attempt | base64-encode the script, decode with openssl base64 -d — broke on heteml (no openssl) |
| Final fix | Single-quote the script with shlex.quote(), pass it as the argument to /bin/sh -c |
| Regression prevention | A static source-code test detects unwrapped csh-unsafe c.run() calls automatically |
When a bug class that was already fixed comes back in new code, it usually means the original fix covered existing code, but a separate guard was needed for new additions. Either a development habit of checking “am I going through the safety wrapper?” or a mechanical test that enforces it — one of the two needs to be in place. Read alongside the original incident to see how the same bug class evolved over time.