What Is Idempotency? Designing Operations That Are Safe to Run Twice

“The job stopped halfway, so I ran it again.” Whether that is safe depends on one property: whether the operation is idempotent.

An operation is idempotent if running it many times leaves the system in the same state as running it once. Network failures, timeouts, double clicks, and overlapping schedules all mean the same operation will eventually run twice.

This article contrasts idempotent and non-idempotent operations using a tiny experiment, then walks through the common designs that make operations safe to repeat.

Note: The experiment used an in-memory SQLite database from Python’s standard library. It is a minimal example to illustrate the idea, not a measurement of any particular service.

What idempotent means

In math, an operation is idempotent when f(f(x)) = f(x). In everyday terms:

• The “up” button on an elevator is idempotent. Pressing it once or five times brings the elevator once
• “Subtract 100 from the balance” is not idempotent. Run twice, it subtracts 200
• “Set the balance to 900” is idempotent. Run any number of times, the balance is 900

The key is that the second and later runs do not change the result. It does not mean “no side effects”. It means the state converges to the same outcome.

Experiment: run the same SQL twice

We created an account with a balance of 1000 and an empty table, and ran each statement twice.

A useful test: does the statement describe a change relative to the current state, or the desired end state? “Subtract 100” and “add a row” are deltas, so every run moves the state forward. “Set to 900” and “make sure this keyed row exists” describe an end state, so repeated runs converge.

Why running twice is not an edge case

1. Retries: when a connection drops, a client cannot tell whether the request arrived. If it resends and the first one had actually succeeded, the work runs twice
2. Double clicks and double submits: if a screen is slow to respond, people press the button again
3. Overlapping schedules: the next run starts before the previous one finishes
4. Restarting after partial failure: if a job stops at item 5 of 10 and restarts from the beginning, items 1–4 are processed twice
5. Redelivered messages: many queue-style systems deliver “at least once”, so duplicates can arrive

The first case cannot be designed away: the sender genuinely cannot know whether it succeeded. Safe resending is a prerequisite for anything dependable.

Common ways to make an operation idempotent

1. Write state, not deltas

Rewrite bal = bal - 100 as bal = 900 whenever you can. Configuration changes, file replacement, and status updates can often be expressed as desired state.

File operations follow the same rule. “Create a directory” fails if it exists, but “ensure the directory exists” is idempotent: os.makedirs(path, exist_ok=True) in Python, mkdir -p in a shell.

2. Let a unique constraint reject duplicates

When appending is unavoidable, pick a key that identifies duplicates and enforce it with a database UNIQUE constraint. In the experiment, the UPSERT relied on a unique index on column k and said “if this k already exists, update it”.

Checking “does it exist?” in application code and then inserting leaves a gap: another process can slip in between. Put duplicate prevention where races cannot happen, which means the database constraint.

3. Use an idempotency key

For operations with no natural key, such as payments and orders, the client generates a unique identifier per intended operation and sends it with the request. The server records the key, and if the same key arrives again, it returns the earlier result instead of repeating the work.

• One key per intended operation. Reuse the same key when retrying, and use a different key for a different operation
• The server must store keys and results for some period
• A request that reuses a key with different content should be rejected

4. Leave a “done” marker

For batch and migration jobs, record how far you got and skip finished parts on rerun.

• Flag rows or timestamp them when processed
• For a data-format migration, check at the start whether the data is already in the new format and do nothing if so

“Check, then act” looks like the earlier race-prone pattern, but it is safe when only one process can run at a time (enforced by locking). If concurrent runs are possible, combine it with a unique constraint or an exclusive lock. We cover locking in Optimistic vs Pessimistic Locking.

Idempotent does not mean harmless

Idempotent is not “no side effects”. Deleting a file is idempotent (the second run finds nothing to delete and the state is unchanged), yet the first deletion is irreversible. Idempotency only guarantees that rerunning does not make things worse. It says nothing about whether the operation itself is desirable.

The return value may still differ. If the second delete reports “not found”, the state is the same but the caller sees an error. Code that retries should treat “already in the desired state” as success.

A checklist for your own operations

1. If this stops halfway and starts again from the beginning, what happens?
2. If two copies run at the same time, what happens?
3. Does it accumulate deltas (append, add, send an email)? Can it be rewritten as desired state?
4. If not, what is the key that identifies a duplicate, and is it protected by a database constraint?
5. Can a rerun treat “already done” as success rather than an error?

The same thinking applies to this tool’s own data-format migrations: they are written so that running them again on already-migrated data changes nothing, which keeps recovery after an interruption calm and predictable.

Summary

• Idempotent means running many times gives the same result as running once
• “Subtract 100” and “add a row” are not idempotent; “set to 900” and “ensure this row exists” are
• Retries, double clicks, overlapping schedules, and restarts after partial failure make running twice routine
• Tools for idempotency: write state, enforce unique constraints, use idempotency keys, leave done markers
• Idempotent does not mean harmless; it means rerunning does not worsen the state

Asking “is it safe to run this again?” while writing an operation makes recovery after a failure far less stressful.