Defensive Microservices: Circuit Breakers, Bulkheads, and Fallbacks in TypeScript

1. Cascading Failures and Thread Exhaustion

A distributed system's most dangerous failure mode is not a dependency going down cleanly — that fails fast and is easy to handle — it is a dependency going slow. When a downstream service starts responding at 20 seconds instead of 200 milliseconds, every caller holding a connection open waiting on it accumulates: in a Node.js service, that means a growing number of in-flight promises and open sockets; in a thread-per-request runtime, it means literal thread pool exhaustion. Either way, the calling service's own capacity to handle any request — including ones that have nothing to do with the slow dependency — degrades in lockstep.

This is how a single slow database query in one microservice becomes a full outage three hops upstream: service C calls a struggling database, service B calls C and its request queue backs up waiting on C, service A calls B and now A's own health checks start failing because its event loop or thread pool is saturated with requests stuck behind B. None of A's other dependencies are unhealthy, but A itself becomes unavailable, and the outage has now cascaded across a boundary that had nothing to do with the original failure.

Timeouts alone do not fully solve this. A timeout bounds how long any single request waits, but if requests keep arriving faster than timeouts expire, you still accumulate an unbounded number of in-flight requests against the struggling dependency during that window. What actually stops the bleeding is refusing to send new requests to a dependency that is clearly unhealthy, immediately and cheaply, instead of discovering its unhealthiness fresh on every single call.

2. Implementing a Finite State Circuit Breaker

A circuit breaker wraps calls to a dependency in a state machine with three states. CLOSED is the normal state: requests pass through, and the breaker tracks a rolling window of successes and failures. OPEN is the tripped state: requests are rejected immediately, without even attempting the call, the moment the failure rate in the rolling window crosses a configured threshold. HALF_OPEN is the recovery-probe state: after a cooldown period, the breaker allows a small number of trial requests through to test whether the dependency has recovered, without committing to full traffic yet.

The transition logic matters more than the states themselves. CLOSED to OPEN triggers when the failure rate (not the raw count, to stay meaningful under variable traffic) exceeds a threshold — 50% failures in a rolling window of the last 20 requests is a common starting point — with a minimum request volume gate so a brief burst of 2 failures out of 2 total requests does not trip the breaker on statistically meaningless data. OPEN to HALF_OPEN is purely time-based: after a fixed cooldown (often 30-60 seconds), the breaker allows probing to resume. HALF_OPEN to CLOSED requires a run of consecutive successes on the probe requests; HALF_OPEN to OPEN happens on a single probe failure, sending the breaker back to full rejection rather than risking a flood against a dependency that has not actually recovered.