Distributed Systems: The Failures You'll Actually Hit

The textbook version of distributed systems is CAP theorems and consensus proofs. Useful eventually, but not what breaks your service at 2am. What actually breaks it is more mundane and more common: a network call that half-succeeded, a retry that ran the same charge twice, two servers that disagree about what time it is.

These are the failures I've actually hit shipping distributed backends, and the mental models that made them tractable.

The core problem: partial failure

A function call in one process either returns or throws. A network call has a third outcome that doesn't exist locally: it might have worked, and you'll never know. You send a request, the other service processes it, and the response is lost on the way back. From your side it looks identical to "it never arrived."

That single fact — you can't tell success-with-lost-response from never-happened — is the root of most distributed-systems pain. Everything below is a response to it.

Retries create duplicates

Since you can't tell if a request succeeded, the natural response is to retry. But if the original did succeed, the retry runs it again. Retry a "charge the card" call and you've charged twice.

The answer isn't to stop retrying — it's to make operations idempotent: safe to run more than once with the same result. The standard technique is an idempotency key the caller generates, that the receiver records:

public Receipt charge(ChargeRequest req) {
    // Seen this key before? Return the original result, don't charge again.
    Receipt existing = receipts.findByKey(req.idempotencyKey());
    if (existing != null) return existing;
 
    Receipt receipt = gateway.charge(req.amount());
    receipts.save(req.idempotencyKey(), receipt);
    return receipt;
}

Now a retry with the same key is harmless. This is the single most valuable habit in distributed systems: assume every message may arrive more than once, and design the handler so that's fine.

Retries can also cause outages

There's a nastier side to retries. When a service gets slow, everyone calling it retries, which multiplies the load on the already-struggling service — a retry storm that turns a blip into a full outage. Two guards matter:

• Exponential backoff with jitter — don't retry immediately or in lockstep; wait longer each time, with randomness so callers don't synchronize into waves.
• Circuit breakers — after enough failures, stop calling the downstream entirely for a while and fail fast. Give it room to recover instead of hammering it.

Time is not shared

Every server has its own clock, and they drift. So System.currentTimeMillis() on server A and server B can disagree by seconds. This quietly breaks anything that uses wall-clock time to order events or resolve "who wrote last."

The lesson: don't use wall-clock timestamps to establish ordering across machines. Use logical ordering — sequence numbers, versions, or a dedicated ID scheme — when order actually matters. Clocks tell you roughly when; they don't reliably tell you what came first.

CAP, the practical version

The famous theorem, stripped to what it means on the job: when the network partitions — and it will — a system can stay consistent (reject reads/writes that might be stale) or stay available (serve possibly-stale data), not both. You're choosing which failure your users experience.

Most systems shouldn't pick globally; they pick per operation. A payment must be consistent — better to error than double-spend. A "view count" can be available and eventually correct — nobody cares if it's briefly off. Knowing which of your operations tolerate staleness is more useful than reciting the theorem.

The posture that holds up

None of this requires exotic tools. It requires assuming the worst at every boundary: every network call can fail halfway, every message can arrive twice, every clock lies, and every dependency will eventually be down. Design each interaction so those aren't catastrophes — idempotent handlers, timeouts and backoff, logical ordering, and a clear stance on consistency-vs-availability. Do that and the 2am pages get a lot rarer.