In Part 1, we looked at client-side load balancing, retries, backoff and jitter, and timeouts.
Those patterns help the client choose a healthy endpoint, tolerate transient failures, and bound how long a request can wait.
But a harder class of failure remains.
What if the downstream service is not temporarily unhealthy? What if it stays unhealthy? What if it becomes so slow that every caller starts exhausting resources? And what if one problematic dependency is able to consume enough capacity to take unrelated functionality down with it?
This is where three additional resilience patterns become important:
- Circuit Breakers
- Fallbacks
- Bulkheads / Connection Limits
The common theme is failure containment.
Circuit Breakers
Questions To Solve
How do we stop calling a dependency that is clearly failing or performing badly?
Suppose Service A repeatedly calls Service B:
Service A -> Service B
Service A -> Service B
Service A -> Service B
Service A -> Service B
If B is healthy, this is normal. If B is persistently unhealthy, continuing to send requests only consumes connections, latency budget, CPU, queue space, and retry capacity. It can also make B harder to recover because clients keep adding pressure.
We need a way to fail fast.
Common Design
The circuit-breaker pattern is usually modeled with three states:
failure threshold exceeded
+---------------------------------------+
| v
+---------+ +---------+
| CLOSED | | OPEN |
+---------+ +---------+
^ |
| |
| successful probe | recovery timer
| v
| +-----------+
+---------------------------------| HALF-OPEN |
+-----------+
|
|
failed probe
|
v
OPEN
Closed
Requests flow normally. Failures are observed. When the failure threshold is exceeded, the circuit opens.
Open
Requests fail quickly instead of being sent to the unhealthy destination. This protects the caller and gives the dependency time to recover.
Half-Open
After a recovery interval, a limited number of probe requests are allowed. Successful probes close the circuit. Failed probes reopen it.
The important idea is that recent failure history changes future routing behavior.
Outlier Detection at the Proxy Layer
A service mesh can implement a closely related mechanism at endpoint granularity.
Suppose a destination contains:
foo-1 healthy
foo-2 healthy
foo-3 returning errors
foo-4 healthy
Envoy can eject the known-bad endpoint from the load-balancing pool while keeping healthy endpoints available:
foo-service
+----------+----------+----------+
| | | |
v v x v
foo-1 foo-2 foo-3 foo-4
ejected
This is a form of circuit-breaking behavior, but the granularity matters. Endpoint outlier detection is not identical to one application-level CLOSED/OPEN/HALF-OPEN state for the entire remote service.
That distinction is useful when debugging why some traffic continues to flow while a bad host has been isolated.
Implementation in App Mesh
The design used App Mesh outlier detection on the listener of a virtual node. The client Envoy observes upstream failures and can eject endpoints that cross the configured threshold.
A simplified example:
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
name: sw-foo-service
namespace: sw-foo-service
spec:
listeners:
- portMapping:
port: 8080
protocol: http
outlierDetection:
maxServerErrors: 5
maxEjectionPercent: 50
interval:
unit: s
value: 10
baseEjectionDuration:
unit: s
value: 10
The important policy decisions are:
- how much failure is enough to declare an endpoint unhealthy;
- how long to eject it;
- how frequently to reevaluate;
- how much of the pool may be ejected at once.
Protection mechanisms need their own safety bounds. An overly sensitive detector can become an outage mechanism itself.
Fallbacks
Questions To Solve
If the preferred dependency cannot serve the request, can the application still provide something useful?
A circuit breaker answers:
Should I continue attempting this remote call?
A fallback answers a different question:
What should my application do when the call cannot succeed?
Common Design
Imagine a storefront normally returns personalized recommendations:
User Request
|
v
Recommendation Service
|
v
Personalized Results
If the recommendation service is unavailable, failing the entire storefront request may be unnecessary.
User Request
|
v
Recommendation Service ---- unavailable
|
v
Fallback
|
+--> cached recommendations
+--> popular products
+--> empty optional section
Other fallbacks might include:
- reading stale but acceptable cached data;
- switching to a secondary source;
- returning a degraded response;
- queueing work for asynchronous processing;
- disabling a nonessential feature.
Fallbacks let a system degrade gracefully instead of converting every dependency failure into a full user-visible outage.
Why Fallbacks Usually Belong in Application Code
This is an important boundary.
The mesh can determine that no healthy upstream is available and return an error such as 503 Service Unavailable.
Only the application knows whether that should mean:
return cached data
or:
queue the command for later
or:
fail immediately because correctness requires fresh data
A proxy understands network policy. It does not understand business meaning.
A useful ownership model is:
mesh
├── endpoint health
├── retries
├── timeouts
├── connection limits
└── routing
application
├── semantic recovery
├── alternate data source
├── degraded response
└── business correctness
Service mesh should remove cross-cutting communication concerns from application code. It should not remove application responsibility.
Bulkheads / Connection Limits
Questions To Solve
How do we stop one dependency or traffic class from consuming all available resources?
The term comes from ships. A hull is divided into watertight compartments so damage in one section does not flood the whole vessel.
+-------+-------+-------+-------+
| | FLOOD | | |
| | XXXXX | | |
+-------+-------+-------+-------+
The same principle applies to services.
Suppose one application calls payments, search, recommendations, and analytics. If every dependency shares one unbounded pool, a slow analytics service can consume enough threads or connections to prevent payment traffic from progressing.
Without isolation:
shared pool
+----------------+
payment ->| |
search ->| |
recs ->| SATURATED |
analytics>| |
+----------------+
With bulkheads:
payments [ pool A ]
search [ pool B ]
recommendations [ pool C ]
analytics [ pool D - saturated ]
Analytics may fail while payments continue to operate.
The goal is simple:
Make the failure domain smaller than the system.
Bulkheads in Service Infrastructure
Traditional application-level bulkheads may use dedicated thread pools, semaphores, executor pools, or queue limits.
At the proxy layer, the same principle can be approximated with controls such as:
- maximum connections;
- maximum pending requests;
- maximum concurrent requests;
- separate endpoint pools;
- separate workload pools.
These mechanisms are not identical, but they all bound how much of a finite resource one destination can consume.
Implementation in App Mesh
The design used connection-pool settings on a virtual-node listener as a proxy-level bulkhead.
A simplified example:
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
name: sw-foo-service
namespace: sw-foo-service
spec:
listeners:
- portMapping:
port: 8080
protocol: http
connectionPool:
http:
maxConnections: 50
maxPendingRequests: 20
Now the client proxy has a bound on how aggressively it can consume upstream resources.
Concurrency Limits Are Not Rate Limits
This distinction is easy to miss.
maxConnections = 50
does not mean:
50 requests / second
A concurrency limit bounds simultaneous resource use. A rate limit bounds work over time.
rate limit
|
v
how quickly requests enter
concurrency limit
|
v
how much work exists simultaneously
A robust platform may need both, but they solve different overload problems.
Combining the Patterns
These mechanisms are most useful as layers rather than isolated features.
Timeout
|
| bounds duration
v
Retry + backoff + jitter
|
| tolerates transient failure without synchronized storms
v
Outlier Detection
|
| removes consistently bad endpoints
v
Connection Pool / Bulkhead
|
| bounds resource consumption
v
Fallback
|
| preserves useful application behavior
v
A timeout, retry policy, circuit breaker, and concurrency limit form one feedback system. Changing one changes the behavior of the others.
For example, aggressive retries can multiply pressure before a detector reacts. Large connection pools can allow too much work to accumulate before timeouts release it. Tiny limits can protect a dependency but reject healthy traffic unnecessarily.
Resilience tuning therefore has to be treated as system design, not a collection of independent knobs.
Isolation Can Also Be Physical
Bulkheads do not have to stop at connection pools.
Some workload classes deserve separate compute resources:
interactive web traffic -> web worker pool
large exports -> export worker pool
backend ETL -> batch worker pool
Combined with the Traffic Channel pattern from the deployment series, dynamic routing can send each class to a separate set of service instances and even separate Kubernetes worker nodes.
sw-foo-service
|
dynamic routing
/ | \
/ | \
v v v
web api batch
| | |
node pool node pool node pool
Now isolation extends from connections to CPU and memory, reducing the blast radius of workload-specific pressure.
Resilience Is About Failure Containment
It is tempting to define reliability as “keep every request successful.” Distributed systems eventually make that impossible.
A more useful objective is:
When something fails, keep the failure bounded, observable, and recoverable.
The resilience patterns approach that objective from different directions:
- Client-side load balancing avoids unhealthy destinations and spreads work across healthy ones.
- Retries recover from carefully selected transient faults.
- Backoff and jitter keep retries from becoming a synchronized thundering herd.
- Timeouts bound how long resources can be held.
- Circuit breakers stop repeatedly spending resources on known-bad destinations.
- Fallbacks preserve useful application behavior when the preferred path is unavailable.
- Bulkheads keep one overloaded dependency from consuming resources needed by unrelated work.
The goal of microservice governance is not simply to run many small services. It is to establish system-wide rules for how those services behave when the environment is imperfect.