In the Routing Patterns article, we established a way to discover services and route requests dynamically. In the Resilience Patterns series, we used that routing layer to make service-to-service communication more tolerant of failure.
Production adds another class of problems.
Different workloads can interfere with each other. A backend job can overwhelm the same instances serving interactive users. A new version may pass unit tests but still fail when connected to the real production dependency graph. A release may need to be exposed gradually or only to a selected population.
I group these problems into four deployment patterns:
- Traffic Channel / Multiple-Tenant Traffic
- Testing in Production / Pre-Release
- Canary Release
- A/B Testing
The common idea is simple:
Deploy the same logical service as multiple isolated sets of instances, then use dynamic routing to decide which set receives each request.
The original service-mesh design fully specified the first two implementations and defined the common design for the latter two. The detailed App Mesh implementation for canary and A/B testing was intentionally left for a later phase, so I keep that boundary here rather than inventing implementation detail that was not in the design.
Traffic Channel / Multiple-Tenant Traffic
Questions To Solve
How do we isolate different kinds of traffic so that the resource pressure created by one workload does not degrade another workload using the same logical service?
A common example is the collision between interactive requests and backend jobs.
interactive web requests ----+
|
v
foo-service
^
|
backend ETL / batch jobs -----+
If a large batch job suddenly drives CPU, memory, connection, or queue usage upward, interactive users see the same degradation because both traffic classes compete for the same service instances.
Common Design
Instead of treating foo-service as one undifferentiated pool, deploy separate sets of instances for different traffic classes.
foo-service
|
dynamic routing
/ | \
/ | \
v v v
web channel api channel default
| | |
instance set instance set instance set
A request carries a traffic-class marker, for example:
X-Traffic-Channel: web
The routing layer examines the header and sends the request to the matching set of instances.
If no dedicated channel matches, traffic should fall back to the default pool.
That fallback matters. A channel architecture that silently drops or misroutes traffic whenever a specialized pool is absent is fragile. The default route provides a predictable baseline.
Propagate the Channel Through the Call Graph
Routing only the first hop is not enough.
Suppose a web request enters Service A and then calls B and C:
client
|
| X-Traffic-Channel: web
v
Service A
|
v
Service B
|
v
Service C
If Service A forgets to propagate the channel metadata, B and C may fall back to their default pools, defeating end-to-end isolation.
Traffic classification is therefore request context. If downstream services participate in the same isolation model, the context has to travel with the request.
Compute Isolation Matters Too
Header-based routing isolates requests logically. It does not automatically isolate CPU and memory.
If the web, API, and batch deployments are all scheduled onto the same saturated worker node, the blast radius is still shared at the infrastructure layer.
The design therefore paired routing with Kubernetes scheduling:
web channel -> web worker nodes
api channel -> api worker nodes
default channel -> general worker nodes
Using labels and nodeSelector gives the pattern two layers of isolation:
- routing isolation — requests are sent to different pods;
- compute isolation — those pods can be scheduled to different worker pools.
This is much closer to a real bulkhead than merely adding another route.
Implementation in App Mesh
The implementation uses three resource concepts:
- one VirtualService as the stable logical entry point;
- one VirtualRouter containing the channel routing rules;
- one VirtualNode for each independently routed channel.
For a service with web, api, and default channels:
VirtualService
sw-foo-service
|
v
VirtualRouter
/ | \
/ | \
X-Traffic=web X-Traffic=api fallback
| | |
v v v
VirtualNode VirtualNode VirtualNode
web api default
Kubernetes Deployments
Each channel is deployed independently and labeled accordingly:
apiVersion: apps/v1
kind: Deployment
metadata:
name: sw-foo-service-web
namespace: sw-foo-service
spec:
replicas: 3
selector:
matchLabels:
app: sw-foo-service
traffic-channel: web
template:
metadata:
labels:
app: sw-foo-service
traffic-channel: web
spec:
containers:
- name: sw-foo-service
image: sw-foo-service:BUILD-29
ports:
- containerPort: 8080
nodeSelector:
traffic-channel: web
The API channel can use the same application image with a different traffic-channel label, replica count, and worker-node selector.
The channels are operational partitions, not different codebases.
Virtual Router
The routing layer matches the header and selects the corresponding virtual node:
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
name: sw-foo-service-router
namespace: sw-foo-service
spec:
listeners:
- portMapping:
port: 8080
protocol: http
routes:
- name: web-channel
httpRoute:
match:
prefix: /
headers:
- name: X-Traffic-Channel
match:
exact: web
action:
weightedTargets:
- virtualNodeRef:
name: sw-foo-service-web
weight: 1
- name: api-channel
httpRoute:
match:
prefix: /
headers:
- name: X-Traffic-Channel
match:
exact: api
action:
weightedTargets:
- virtualNodeRef:
name: sw-foo-service-api
weight: 1
- name: default
httpRoute:
match:
prefix: /
action:
weightedTargets:
- virtualNodeRef:
name: sw-foo-service
weight: 1
The catch-all default route must remain lower priority than the header-specific routes.
Virtual Nodes and Discovery Attributes
Each isolated instance set is represented by its own virtual node and can be associated with a discovery attribute such as traffic-channel=web.
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
name: sw-foo-service-web
namespace: sw-foo-service
spec:
podSelector:
matchLabels:
app: sw-foo-service
traffic-channel: web
listeners:
- portMapping:
port: 8080
protocol: http
serviceDiscovery:
awsCloudMap:
namespaceName: mesh.local
serviceName: sw-foo-service
attributes:
- key: traffic-channel
value: web
The same logical service can now expose multiple operational pools without requiring callers to know their physical endpoints.
Traffic Channels Are Not Full-Stack Clones
One important improvement over a traditional channel architecture is that isolation can be service-specific.
Suppose only Service B is overloaded by batch traffic:
Service A
|
+--------+--------+
| |
v v
Service B web Service B batch
| |
+--------+--------+
|
Service C
We do not need to duplicate the entire application stack just to isolate B.
That is one of the biggest advantages of combining service-level deployment with dynamic routing: isolate only the workloads that need isolation.
Testing in Production / Pre-Release
Questions To Solve
How do we perform end-to-end integration testing against a new version of one service using the real production dependency graph, without sending normal production traffic to that version?
Microservices are independently deployable, but they are not independently useful. A service is usually part of a larger call graph.
A new Service B can pass its own tests and still fail when connected to the real versions of A, C, D, or the actual production configuration.
Service A
|
+-------+-------+
| |
v v
Service B Service B'
production pre-release
| |
+-------+-------+
|
Service C
The goal is to run a controlled request through B' while every other hop remains on the normal production path.
Common Design
The pattern has four steps:
- Deploy the new version as a small isolated instance set using production-compatible configuration.
- Label and register that instance set separately from the normal production pool.
- Add a header-based route that only controlled test requests can match.
- Run end-to-end smoke tests through the real service graph, then remove or scale down the pre-release pool.
A request might carry:
X-Pre-Release: sw-foo-service
For the target service, the routing behavior becomes:
X-Pre-Release: sw-foo-service -> pre-release instances
anything else -> production instances
Normal users continue to reach the production version.
Why the Header Names the Service
A boolean such as:
X-Pre-Release: true
is not enough when a request crosses many services. It could accidentally route every service that has a pre-release deployment onto its experimental version.
Using the target service as the value keeps the test scoped:
X-Pre-Release: sw-foo-service
Service A stays production. Service B switches to its pre-release pool. Service C returns to production unless it is independently targeted by another test.
That makes the pattern useful for one-service-at-a-time end-to-end validation.
Implementation in App Mesh
The design built pre-release testing on the same header-based dynamic routing used by traffic channels.
Step 1: Deploy the Pre-Release Version
apiVersion: apps/v1
kind: Deployment
metadata:
name: sw-foo-service-pre-release
namespace: sw-foo-service
spec:
replicas: 2
selector:
matchLabels:
app: sw-foo-service
pre-release: "true"
template:
metadata:
labels:
app: sw-foo-service
pre-release: "true"
spec:
containers:
- name: sw-foo-service
image: sw-foo-service:BUILD-30
ports:
- containerPort: 8080
Step 2: Expose It as a Separate Virtual Node
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
name: sw-foo-service-pre-release
namespace: sw-foo-service
spec:
podSelector:
matchLabels:
app: sw-foo-service
pre-release: "true"
listeners:
- portMapping:
port: 8080
protocol: http
serviceDiscovery:
awsCloudMap:
namespaceName: mesh.local
serviceName: sw-foo-service
attributes:
- key: pre-release
value: "true"
Step 3: Add a Specific Route Before the Default Route
routes:
- name: pre-release
httpRoute:
match:
prefix: /
headers:
- name: X-Pre-Release
match:
exact: sw-foo-service
action:
weightedTargets:
- virtualNodeRef:
name: sw-foo-service-pre-release
weight: 1
- name: default
httpRoute:
match:
prefix: /
action:
weightedTargets:
- virtualNodeRef:
name: sw-foo-service
weight: 1
Step 4: Run End-to-End Smoke Tests
The test client sends:
X-Pre-Release: sw-foo-service
The request enters the normal production application but is diverted only when it reaches sw-foo-service.
After validation, the pre-release deployment can be scaled to zero or removed.
Testing in Production Is a Private Lane, Not Random User Exposure
The phrase “testing in production” can sound reckless if interpreted as exposing unverified behavior to arbitrary customers.
That is not the pattern here.
The production environment supplies the real dependency graph and configuration, while routing policy limits which requests can reach the new version.
The design is closer to a private lane through production than an uncontrolled experiment.
The safety requirements are therefore:
- a narrow and explicit route matcher;
- a reliable default route back to production;
- controlled test clients;
- strong observability for the pre-release pool;
- a fast way to remove the test deployment.
Canary Release
Questions To Solve
How do we roll out a new release to production gradually and safely?
Common Design
A canary release runs the new version in parallel with the old version and sends only a small percentage of normal production traffic to it.
Service B
|
weighted route
/ \
/ \
1% 99%
| |
v v
Service B' Service B
new version old version
The important difference from pre-release testing is the traffic source.
Pre-release traffic is explicitly selected test traffic. Canary traffic is a controlled percentage of real production traffic.
The release process becomes:
- deploy the new version next to the old version;
- start with a very small traffic weight;
- observe service and system metrics;
- increase the weight gradually if the new version behaves correctly;
- reach 100% and decommission the old version;
- if metrics regress, immediately route traffic back to the old version.
The routing primitive is a weighted traffic split.
1% -> 5% -> 20% -> 50% -> 100%
The exact progression is an operational decision, not part of the pattern itself.
What to Observe During a Canary
The design's broader observability model suggests watching the same signals used to understand service health generally:
- request latency;
- traffic volume;
- error rate;
- saturation;
- service access logs;
- distributed traces.
A canary is useful only if the platform can distinguish the new version's behavior from the old version's behavior.
Implementation Boundary in the Design
The design specifies weighted dynamic routing as the mechanism for canary release and defines the rollout behavior above. It does not include the detailed App Mesh manifests for this pattern; those were explicitly deferred to a later implementation phase.
So the architectural mapping is clear:
VirtualService
|
VirtualRouter
|
weighted route
/ \
old new
node node
but the concrete production manifest was not part of the completed design.
A/B Testing
Questions To Solve
How do we expose different application variants to different user populations so we can compare their behavior?
Common Design
A/B testing uses the same idea of multiple instance sets behind one logical service, but the routing goal is experimentation rather than release safety.
The design describes two routing strategies.
Weighted A/B Testing
If the experiment is population-based, such as 50% of requests seeing variation A and 50% seeing variation B, use a weighted traffic split.
Service B
|
weighted route
/ \
50% 50%
| |
v v
Variant A Variant B
Header-Based A/B Testing
If the experiment should target a chosen set of users or controlled clients, use request metadata.
For example:
X-Experiment-Variant: B
can route the request to Variation B, while requests without that marker continue to Variation A.
X-Experiment-Variant: B -> Variant B
anything else -> Variant A
This is the same header-routing primitive used by traffic channels and pre-release testing, but with a different semantic purpose.
Weighted Routing and User Stickiness Are Different Problems
A pure weighted request split distributes requests, not necessarily users.
If an experiment requires a particular user to remain in one cohort across requests, the system needs a stable experiment assignment mechanism in addition to service routing.
The service mesh can enforce the route once the cohort is represented in request metadata, but the business layer still owns how a user is assigned to a cohort.
That boundary mirrors the fallback pattern from the resilience series: infrastructure handles transport policy; application semantics stay with the application.
Implementation Boundary in the Design
As with Canary Release, the design defines weighted-based and header-based routing as the implementation mechanisms for A/B testing, but the detailed App Mesh resource manifests were deferred to a later phase.
The source-supported architecture is therefore:
VirtualService
|
VirtualRouter
|
experiment route
/ \
Variant A Variant B
with route selection based on either traffic weights or request headers.
One Primitive, Four Operational Patterns
These four patterns look different from the outside:
- Traffic Channel protects one workload class from another.
- Pre-Release Testing validates one new service version against the real production graph.
- Canary Release reduces rollout risk by gradually increasing exposure.
- A/B Testing compares product variants across controlled populations.
Underneath, they all share the same mechanism:
one logical service
|
v
multiple isolated instance sets
|
v
explicit routing policy selects the set
The difference is what the route means.
Traffic Channel -> workload class
Pre-Release -> controlled test request
Canary -> rollout percentage
A/B Testing -> experiment cohort
That is why dynamic service routing is such an important governance primitive. Once routing can select among isolated deployments without changing application code, the platform gains a reusable control surface for reliability, release safety, workload isolation, and experimentation.
Routing supplies the steering wheel. Deployment creates the roads.