Calling External Commands from Python: Quoting, Timeouts, and Exit Codes

Python tools often need to call external commands such as ssh, wp (WP-CLI), or rsync. subprocess.run() makes that a few lines of code, but “it ran” and “I will notice when it fails” are two different things.

This article covers three pitfalls that come up again and again — argument quoting, timeouts, and exit codes — with results from actually running them on Python 3.12. It describes general standard-library behavior, not any specific product, and it stays at the level of design principles rather than security-sensitive specifics.

Note: An exit status (return code) is the integer a command hands back to the OS when it finishes. By convention 0 means success and anything else means failure.

Pitfall 1: A string gets split on whitespace

There are two ways to call a command:

• List form: subprocess.run(["ls", "my file.txt"]) — each argument is passed separately.
• String with shell=True: subprocess.run("ls my file.txt", shell=True) — one string is interpreted by the shell (/bin/sh).

Here is what happens with a filename containing a space:

import subprocess
subprocess.run("ls my file.txt", shell=True)   # the shell splits on the space
subprocess.run(["ls", "my file.txt"])           # passed through as one argument

The first call printed two errors, ls: my: No such file or directory and ls: file.txt: No such file or directory; the second printed one: ls: my file.txt: No such file or directory. With the shell string, ls went looking for two different files.

The rule of thumb:

1. Prefer list form. Python fixes the argument boundaries and the shell never re-parses them.
2. Use shell=True only when you need shell features (pipes, redirection, globbing), and wrap any dynamic value in shlex.quote().

import shlex
shlex.quote("it's a file.txt")   # => 'it'"'"'s a file.txt'

One more thing: when you send a command to a remote host (ssh host "wp plugin list"), quoting happens in two stages — once locally, and once by the remote shell. Dynamic values inside the remote command string need shlex.quote() for the remote shell too. “It works locally but spaces break on the remote side” is usually this second stage being missed.

Pitfall 2: Without a timeout, you wait forever

External commands can hang on a dropped connection or a slow server. subprocess.run() waits indefinitely unless you pass timeout.

try:
    subprocess.run(["sleep", "5"], timeout=1)
except subprocess.TimeoutExpired as e:
    print("timeout:", e.cmd)

This raised TimeoutExpired after about 1.0 second, with e.cmd equal to ['sleep', '5']. On timeout, run() kills the child, waits for it, and then raises.

Three points to remember:

• Catch the exception. Otherwise the whole tool crashes. Record “this site timed out” and move on.
• With shell=True, the process that gets killed is the shell. Grandchild processes it started are not guaranteed to stop (the Python documentation carries a note about this). Another reason to prefer list form.
• Choose the value as a multiple of the normal duration. Too short cuts off healthy work; too long delays detecting real trouble. How to wait and retry is covered in Exponential Backoff vs Fixed-Interval Retries.

Pitfall 3: Ignore the exit code and failure looks like success

By default subprocess.run() does not raise when the command fails; it just returns a CompletedProcess. If you skip the check, an empty output from a failed command looks like “no results”.

r = subprocess.run("echo out; echo err >&2; exit 3",
                   shell=True, capture_output=True, text=True)
# r.returncode == 3 / r.stdout == 'out\n' / r.stderr == 'err\n'

You have two options:

• Pass check=True: a failure raises CalledProcessError, with the value in e.returncode.
• Inspect r.returncode yourself and branch.

The second is necessary because exit code 1 is not always an “error”. grep returns 1 when nothing matched (I confirmed this with grep x /dev/null). If you need to tell “not found” from “failed to run”, look up what each command’s codes mean.

Other quirks I observed:

The pipeline case is easy to miss: a failure in an earlier stage is hidden. When you consume piped output, consider checking each stage separately (for example, one run() per stage).

Treat stdout and stderr separately

With capture_output=True, standard output lands in r.stdout and standard error in r.stderr. In the example above they came back separately as 'out\n' and 'err\n'.

• Machine-readable results (such as JSON) usually go to stdout; warnings and logs usually go to stderr.
• Mixing them can let a single warning line break your JSON parsing.
• Output ends with a newline, so strip() before comparing.
• Without text=True you get bytes; on systems with non-UTF-8 defaults, set encoding= explicitly (see UTF-8 vs cp932).

Putting it together as a design

Wrap external calls in one function and give it a few promises:

1. Arguments arrive as a list (shell=True is the exception).
2. A timeout is always set — callers cannot omit it.
3. The result distinguishes success, command failure (with the exit code), timeout, and command-not-found.
4. On failure, the exit code and stderr are recorded.

That removes much of the “failed but looks fine” state at the design level. For maintenance tooling, a silently missed failure is the most expensive kind.

Calling an external command is easy to write but fails in many different ways. Getting quoting, timeouts, and exit codes right up front removes most of the mysterious “breaks sometimes” behavior.