Distributed systems fail differently from monoliths.
When one function calls another in the same process, failure is usually immediate and obvious. When one service calls another over a network, the situation becomes ambiguous.
Did the server fail? Did the connection fail? Did the request reach the server but the response get lost? Is the server simply slow? Should we try another instance? Should we retry? How long should we wait?
These questions are why resilience cannot be treated as an afterthought in a microservice architecture.
I group the client resilience patterns into:
- Client-side Load Balancing
- Retries
- Timeouts
- Circuit Breakers
- Fallbacks
- Bulkheads / Connection Limits
This first part covers the first three.
Client-Side Load Balancing
Questions To Solve
When a service has multiple healthy instances, how does a caller choose one?
And if one endpoint becomes slow or unhealthy, how do we stop directing traffic to it?
foo-service
+--------+--------+
| | |
v v v
foo-1 foo-2 foo-3
10ms 12ms failed
A service mesh lets the caller-side proxy perform endpoint selection using the service-discovery data it receives from the control plane.
Client
|
Envoy
|
+----------+----------+
| | |
foo-1 foo-2 foo-3
healthy healthy unhealthy
Common Design
Client-side load balancing needs four things:
- a logical service name;
- discovery of the endpoints behind that name;
- health information for those endpoints;
- an algorithm for choosing a destination.
The important property is topology transparency. Autoscaling from three instances to twenty should not require a client deployment. Shrinking back to four should not require one either.
The service topology is platform state.
Implementation in App Mesh
With the App Mesh data plane, Envoy performs client-side endpoint selection after receiving the virtual service and endpoint configuration. The application calls the logical service; it does not carry its own endpoint cache or language-specific load-balancing library.
That separation matters as the platform becomes polyglot.
Retries
Questions To Solve
What happens when a call fails because of a temporary network problem or a transient server error?
Some failures disappear if the same operation is attempted again. That makes retries one of the simplest ways to improve perceived availability.
It also makes them one of the easiest ways to amplify an outage.
Common Design
The basic retry flow is straightforward:
request
|
v
attempt 1 ---- transient failure
|
v
attempt 2 ---- success
Without a retry, one transient fault becomes a user-visible error. With a bounded retry, the same fault may become only additional latency.
But suppose a downstream service is already overloaded. If every failed request is immediately attempted several more times, retries turn into load amplification.
A retry policy therefore needs explicit boundaries:
- which failures are retryable;
- maximum retry count;
- per-attempt timeout;
- total request deadline;
- whether the operation is semantically safe to retry.
Retry Only When It Is Semantically Safe
Retrying a read such as:
GET /products/123
is usually different from blindly retrying a command such as:
POST /payments
If the server completed the command but the response was lost, an unsafe retry can execute the business operation twice.
The mesh understands transport behavior. It does not understand arbitrary business semantics. Idempotency is still an application contract.
Backoff and Jitter: Preventing the Thundering Herd
Retry count alone is not enough.
Imagine a dependency becomes unavailable and thousands of callers observe the failure at approximately the same time. If every caller retries immediately, or every caller retries after exactly one second, the recovering dependency receives another synchronized burst.
failure
|
v
thousands of clients retry together
|
v
another traffic spike
|
v
service fails again
This is the thundering herd problem.
A safer policy introduces backoff. With exponential backoff, the delay grows after each unsuccessful attempt:
attempt 1
|
| 100 ms
v
attempt 2
|
| 200 ms
v
attempt 3
|
| 400 ms
v
attempt 4
A simplified model is:
delay = base_delay * 2^retry_number
Backoff reduces pressure, but synchronized clients can still line up on the same schedule:
Client A: 100ms -> 200ms -> 400ms
Client B: 100ms -> 200ms -> 400ms
Client C: 100ms -> 200ms -> 400ms
That is why retries also need jitter — randomness added to the delay.
Client A: 83ms -> 241ms -> 361ms
Client B: 126ms -> 174ms -> 492ms
Client C: 57ms -> 229ms -> 417ms
One common strategy is full jitter:
maximum_delay = min(cap, base_delay * 2^retry_number)
actual_delay = random(0, maximum_delay)
Conceptually:
Without jitter
clients
| | | | |
v v v v v
-----X-------------------X-------------------X-----
retry retry retry
With jitter
clients
| | | | |
v v v v v
------x--x-x------x----x---x----x-----x---x-------
retries spread across time
The practical objective is:
Give transient failures a chance to recover without turning retries themselves into another source of overload.
At large scale I also like the idea of a retry budget: cap the fraction of additional retry traffic relative to normal requests so that a failure cannot multiply traffic without bound.
Not every service-mesh API exposes explicit knobs for backoff and jitter. They are still part of the retry pattern and should be implemented at the layer that owns retry timing.
Implementation in App Mesh
In the original App Mesh design, retry behavior was declared on a route so the policy could be managed independently from application code.
A simplified example:
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
name: sw-foo-service-router
namespace: sw-foo-service
spec:
listeners:
- portMapping:
port: 8080
protocol: http
routes:
- name: default
httpRoute:
match:
prefix: /
action:
weightedTargets:
- virtualNodeRef:
name: sw-foo-service
weight: 1
retryPolicy:
maxRetries: 3
perRetryTimeout:
unit: ms
value: 500
httpRetryEvents:
- server-error
- gateway-error
tcpRetryEvents:
- connection-error
The specific numbers are workload-specific. The important thing is that the retry behavior is explicit and reviewable.
Timeouts
Questions To Solve
How long should a caller wait before declaring that a request has failed?
Without a timeout, a slow downstream dependency can consume resources indefinitely.
Service A
|
| waiting...
|
| waiting...
|
v
Service B
If enough requests do this, Service A can exhaust threads, connections, memory, queue capacity, or request slots. A downstream latency problem has now propagated upstream.
Timeouts turn an unbounded wait into a bounded failure.
Common Design
request starts
|
|--------- allowed execution window ---------|
|
v
timeout / fail
A timeout that is too long wastes resources while waiting for a dependency that may never respond. A timeout that is too short causes healthy but slower requests to fail unnecessarily.
The right value therefore depends on the service and the operation.
Timeout Budgets Across a Call Chain
Consider:
A -> B -> C -> D
If A has a five-second deadline, every downstream call cannot independently assume it owns five seconds. The latency budget belongs to the end-to-end request.
Total deadline: 5s
A
└── B
└── C
└── D
Retries spend from the same budget.
If the policy allows three attempts with a 500 ms per-attempt timeout, the outer request timeout must leave enough room for those attempts and their backoff delays. Retry and timeout settings therefore need to be designed together rather than tuned independently.
Different Operations Need Different Policies
A metadata lookup and a large CSV export should probably not share one timeout.
normal requests
|
v
foo-service
short timeout
/export/csv
|
v
foo-export-service
longer timeout
This is where the routing patterns become useful: a path-based route can move expensive work to a separate virtual node and apply a different resilience policy to it.
Why Put These Policies in the Mesh?
None of these patterns requires a service mesh. We could build discovery, load balancing, retries, and timeouts into every application library.
The problem appears when the platform contains multiple languages and frameworks:
Service A -> Python
Service B -> Go
Service C -> Rust
Service D -> Java
Now the organization owns several implementations of the same operational behavior, often with different defaults and release cadences.
The sidecar model separates network policy from business logic:
application
|
v
Envoy
|
v
network
The platform can manage cross-cutting communication behavior consistently while application teams remain responsible for business semantics.
What Is Still Missing?
Load balancing helps avoid unhealthy destinations. Retries help with transient failures. Backoff and jitter prevent synchronized retries from overwhelming a recovering dependency. Timeouts bound how long resources can be held.
But one question remains:
What should happen when a dependency is persistently unhealthy?
Continuing to call it can still waste resources, and even after we stop calling it the application may need another way to respond.
That takes us to Part 2: circuit breakers, fallbacks, and bulkheads.