Microservices make software easier to decompose, deploy, and evolve independently. But once an application becomes a graph of tens or hundreds of services, a problem that used to look trivial becomes part of the platform architecture:
How does one service reliably find and reach another service?
An instance may disappear. Another may be created seconds later. Autoscaling may change the endpoint set continuously. A deployment may temporarily run two versions. Some requests may need to reach one group of instances while other requests reach another.
Hard-coding IP addresses clearly does not work.
I group this problem into two routing patterns:
- Service Discovery — where is the service?
- Service Routing — where should this particular request go?
These patterns are foundational because the resilience and deployment patterns later in this series depend on them.
Service Discovery
Questions To Solve
How do we make microservices discoverable so clients can find them without hard-coding physical locations?
And when one instance becomes unhealthy, how do we make sure new traffic stops reaching it?
In a cloud-native environment, an IP address is temporary infrastructure state rather than service identity. Pods restart. Nodes disappear. Autoscaling changes replica counts. A caller should depend on a logical service name, not a physical endpoint.
Service Registry
+-------------------+
| foo -> 10.0.1.21 |
| foo -> 10.0.2.17 |
| foo -> 10.0.3.42 |
+---------+---------+
|
discovery / health
|
+----------------+----------------+
| | |
foo instance 1 foo instance 2 foo instance 3
healthy healthy unhealthy
Common Design
A typical service-discovery flow looks like this:
- A service instance starts and becomes discoverable.
- The registry associates it with a logical service name.
- Health information determines whether the instance remains eligible for traffic.
- A client resolves the logical service into a set of endpoints.
- The client-side load balancer selects one healthy endpoint.
- Failed or unhealthy endpoints are removed from the active pool.
- The endpoint set is refreshed as the topology changes.
Two separations are important.
Naming is not routing. The name sw-foo-service identifies a service. It should not tell a caller which machine currently runs it.
Discovery is not health checking. A service may be registered while still being unable to serve requests. A robust platform combines discovery with an independent health signal.
Why DNS Alone Can Be Too Coarse
Kubernetes gives every Service a stable DNS name such as:
sw-foo-service.sw-foo-service.svc.cluster.local
That solves the naming problem, but endpoint-aware traffic management sometimes needs more than a single DNS answer. If the data plane wants to eject one bad host while keeping the rest of the pool healthy, the proxy needs visibility into the individual endpoints behind the service.
That is why the design used a service-discovery source that exposed the endpoint set to Envoy instead of treating discovery as an opaque DNS lookup.
The goal is not to replace Kubernetes naming. It is to give the data plane enough information to make health-aware routing decisions.
Service Routing
Service discovery gives us a set of possible destinations.
Routing decides which destination should receive a particular request.
Questions To Solve
How do we give services a stable entry point while supporting policies such as:
- routing different paths to different workloads;
- routing requests with particular headers to isolated instances;
- splitting traffic between versions;
- testing a pre-release deployment without exposing it to normal users;
- applying traffic channels;
- applying consistent communication policies without putting routing logic into every application?
Common Design
I find it useful to separate routing into four responsibilities.
Static Routing
A stable logical name maps traffic to the service responsible for it.
For north-south traffic this is commonly handled by an API gateway or ingress layer:
/api/orders/* -> order-service
/api/users/* -> user-service
Dynamic Routing
The request itself participates in the decision. Useful inputs include:
- HTTP headers;
- URL paths;
- request methods;
- traffic weights;
- gRPC service or method names;
- deployment version or traffic-class metadata.
This capability is what later enables pre-release testing, traffic channels, canary releases, and A/B testing.
Admission Policies
Some concerns should be enforced consistently rather than reimplemented by every service. Authentication, authorization, connection limits, and other communication policies belong at common enforcement points.
North-south admission often belongs at an API gateway. East-west policy can be distributed through the service-mesh data plane.
Telemetry
The routing layer sees almost every service-to-service request, which makes it an ideal point to collect request counts, response codes, latency, source/destination metadata, retry events, and tracing context.
Routing therefore becomes closely related to observability.
From a Central Router to a Distributed Data Plane
A common early microservice architecture routes large amounts of east-west traffic through centralized proxies:
Service A
|
v
Central Router
|
v
Service B
It works, but eventually the router itself becomes a scaling boundary and a failure domain.
A service mesh moves the traffic policy closer to each workload:
Service A
|
Envoy
|
+--------------------+
|
Envoy
|
Service B
The control plane distributes configuration, while the data plane performs the actual request-level enforcement.
A useful summary is:
Centralize policy; decentralize enforcement.
This also solves an organizational problem. A Python service, a Rust service, a Go service, and a Java service should not each need a different implementation of service discovery, route selection, health filtering, retries, and tracing.
Implementation in App Mesh
The original implementation mapped the routing model onto three App Mesh abstractions:
- VirtualService — the stable logical destination used by callers;
- VirtualRouter — the routing policy behind that destination;
- VirtualNode — a concrete set of service instances.
Conceptually:
VirtualService
sw-foo-service
|
v
VirtualRouter
/ | \
/ | \
header path weight
| | |
v v v
VirtualNode VirtualNode VirtualNode
web export v2
Weight-Based Routing
A route can divide traffic between two versions:
routes:
- name: production
httpRoute:
match:
prefix: /
action:
weightedTargets:
- virtualNodeRef:
name: sw-foo-v1
weight: 90
- virtualNodeRef:
name: sw-foo-v2
weight: 10
This primitive later becomes the basis for canary releases.
Header-Based Routing
A request can explicitly select an isolated route:
routes:
- name: experimental
httpRoute:
match:
prefix: /
headers:
- name: X-Service-Variant
match:
exact: experimental
action:
weightedTargets:
- virtualNodeRef:
name: sw-foo-experimental
weight: 1
Header-based routing is especially useful when only controlled clients should see a deployment.
Path-Based Routing
An expensive endpoint can be sent to a dedicated workload pool:
routes:
- name: csv-export
httpRoute:
match:
prefix: /export/csv
action:
weightedTargets:
- virtualNodeRef:
name: sw-foo-csv-export
weight: 1
This is not only a routing technique. It lets us apply different resilience and capacity policies to different classes of work.
Route Priority Is a Correctness Property
Suppose we want:
X-Service-Variant: experimental -> experimental nodes
everything else -> default nodes
The specific rule must be evaluated before the catch-all rule. Otherwise the default route can shadow the intended dynamic route.
A good mental model is:
most specific
|
v
header / path / version rules
|
v
general service rule
|
v
default fallback
A routing table is effectively a small program executed for every request. Ordering and fallback behavior are therefore correctness concerns, not just configuration details.
What Routing Enables Next
Once we have endpoint-aware discovery and dynamic routing, several other governance patterns become much easier to implement.
Resilience: client-side load balancing, retries, timeouts, circuit breaking, and bulkheads.
Deployment: traffic channels, pre-release testing, canary releases, and A/B testing.
Security: workload identity, encrypted service-to-service communication, and authorization policy.
Observability: service metrics, access logs, correlation IDs, and distributed tracing.
Routing is not merely about moving packets from A to B. It is about turning a constantly changing collection of workloads into a platform with stable names, explicit policies, and predictable behavior.