Skip to content
Edouard Topin's Blog
Understanding CI/CD with unicorn-flavoured cotton candy / Series 03/03

A few customers first: canary and blue-green with TOP

The glitter passes its tests, but some orders fail. Explore canary releases, blue-green deployments and Argo Rollouts with TOP.

Edouard Topin
9 min read
TOP tries glitter at a small trial counter and discovers that group orders jam the dispenser.

TOP finally has his Magical Unicorn cotton candy. The horn holds. Orders arrive. The shop is running v2.

At the start, v2 serves Magical Unicorn; TOP imagines a glitter topping before offering it to customers.

He studies the result and whispers: “Good. But perhaps it needs intergalactic glitter.”

The new release passes its tests. The image builds. The new servers become ready. TOP places one glitter order: success. Then a group orders three portions. The dispenser jams.

Everything was green except those customers’ experience. After building the package and rolling it out, we will limit exposure to a change, observe its actual behaviour and decide whether to continue or stop.

Our defect is deliberate: in the lab, an order of three or more portions with glitter returns an error. That lets us explore the reasoning without waiting for a random failure.

The problem that opening checks do not reveal

  1. A ready releaseKnown checks pass, but not every possible situation was tested.
  2. Limited exposureChoose who encounters the new version.
  3. Useful observationMeasure orders by release and by use case.
  4. A decisionPromote, wait or abort, then bring Git back into agreement.

The initial test covers one portion with glitter. It does not cover groups. Readiness establishes that the server can receive traffic; it does not exercise every menu combination.

We should neither remove those checks nor expect them to predict every use. Instead, we add progressive delivery: a release process with controlled exposure and progression. The goal is to make uncertainty manageable and feed what we learn back into the next release.

TOP changes his question. “Does the machine start?” becomes: “What happens to the orders we give it?”

Canary: start with a small counter

For a canary, TOP keeps the stable counters running and opens a small trial capacity. Some traffic reaches the new version. We observe before increasing its exposure.

This is not a marketing taste test. Our purpose is to limit the consequences of a defect and gather technical evidence. An A/B experiment usually compares variants against a product objective. Routing mechanisms can resemble one another while the decisions they support differ.

On Kubernetes, Argo Rollouts adds a progressive delivery controller and a Rollout resource. Argo CD keeps its Git synchronisation role. The tools complement one another: synchronising a specification is different from controlling how quickly a release receives traffic. Argo Rollouts architecture.

Our basic example includes:

strategy:
  canary:
    steps:
      - setWeight: 10
      - pause: {}
      - setWeight: 50
      - pause: {}

An empty pause waits for a manual decision. It does not inspect metrics by itself. First install the stable release fully; these steps then govern an update, rather than manufacturing a comparison during an initial installation with no old version. Canary steps.

Ten percent of what?

TOP places one trial counter beside nine stable counters. “So exactly 10% of customers?”

Not necessarily. Without integrated traffic routing, a basic canary approximates the requested weight through replica counts. Requests do not form a perfectly regular queue: persistent connections and distribution affect what you observe. Controlling traffic weights requires a compatible Ingress, service mesh or another supported provider. Traffic management.

Our manifest uses ten replicas and no weighted traffic router. It teaches progression and pauses. It does not promise exactly ten requests out of a hundred. Keep that limitation visible in the article, the lab and any interpretation of its results.

The trial counter becomes candidate capacity. Inspect its errors before promotion; a Pod ratio does not guarantee an exact request percentage.

The trial counter becomes candidate capacity. Inspect its errors before promotion; a Pod ratio does not guarantee an exact request percentage.

Observe orders, not just status lights

Consider a fictional sample: one thousand orders, with one hundred reaching the candidate. Twenty of those hundred fail. The shop’s overall error rate is 2%, but the candidate’s is 20%.

The whole-shop average hides the trial counter’s problem. TOP needs the release, order type and observed volume. In our story, quantity is the decisive clue: one portion works; three jam the dispenser.

Before exposing the release, we write a small decision rule:

Observation Decision for this exercise
Candidate group orders fail Abort the trial and retain the stable release
Too few relevant orders Wait; do not conclude that everything works
Metrics are missing or collection fails Hold the decision and investigate measurement
Checked journeys, sufficient volume, no observed degradation Consider the next stage

These are demonstration rules. A real service needs its own error budget, time windows and business criteria. Latency also matters: an order that eventually succeeds after an intolerable wait may still represent a degraded service. The threshold is useful only when it reflects what customers need.

Argo Rollouts can perform analyses using AnalysisTemplate and AnalysisRun resources. A metrics provider supplies observations and conditions determine the outcome. Our lab keeps manual pauses; automatic analysis is not configured. Argo Rollouts analysis.

The application exposes counters at /metrics. That does not install Prometheus, configure scraping or define a good query. A counter without an observer is a notebook nobody reads. For more on these signals, see the observability foundations.

A new flavour also needs the right counter

v2 does not offer glitter yet. Sending it a glitter order can legitimately return HTTP 400. Do not confuse that result with v3 returning HTTP 500 for an order it claims to support.

To compare technical behaviour, observe journeys shared by both releases. To exercise the new feature, explicitly test the candidate and organise its exposure through suitable routing or separate activation after compatibility is established. A simple replica-count canary does not solve that menu contract.

TOP will therefore not advertise “glitter for everyone” before knowing where those orders will land. The small-counter illustration makes the boundary visible, but real software needs an explicit rule to preserve it.

Blue-green: prepare a second shop

Another approach is to prepare a complete new set of counters. The active shop serves customers while the candidate has a preview route. After verification, switch the active service to the candidate.

Argo Rollouts’ blue-green model uses an activeService and a previewService. Our example sets autoPromotionEnabled: false to wait for a decision. After promotion, a delay precedes scaling down the old set, allowing routing changes to propagate. Blue-green with Argo Rollouts.

Two ways to introduce a release: a small trial counter beside stable counters, or a complete second shop prepared before switching the entrance.

TOP likes being able to inspect the candidate shop before directing customers there. But both sets need capacity, and the switch exposes a lot of traffic at once. Keeping the old software available helps with returning to it; it does not undo operations already performed in a database or by a payment provider.

Approach What it controls Main consideration
Rolling update Gradual replacement of instances Compatibility during coexistence
Canary Gradual exposure of the candidate Routing and quality of observations
Blue-green Preparation and switching between two sets Capacity, connections and shared data

There is no universal winner. Useful questions for our small service are practical: can the team observe a canary? Is there room to duplicate the environment? Can it return to the old version without breaking data? A strategy is only as useful as the team’s ability to operate and verify it.

Two distinct routes: clients use activeService, tests use previewService. Promotion changes the active destination.

Two distinct routes: clients use activeService, tests use previewService. Promotion changes the active destination.

Abort the trial, then correct the plan

Group orders fail. TOP does not wait for the next stage. He aborts progression and lets the stable service take over.

kubectl argo rollouts abort candy-canary -n candy-lab

Abort does not edit the Git repository. Argo Rollouts manages delivery state; we must then revert the bad image change in Git or deliver its correction through the chosen procedure. The exact traffic consequences depend on the strategy and integration. Argo Rollouts and GitOps.

TOP returns to the laboratory with a precise example: three portions and glitter. He adds a regression test that fails on the defective release, fixes the defect and repeats the CI journey. The canary therefore improves future checks. It does not become an excuse to ship without tests.

This feedback is the useful end of the story. The production observation becomes a concrete input for the next change, with an expected result somebody can check again.

The return journey matters too: abort, correct Git, reproduce the defect and add the missing test.

The return journey matters too: abort, correct Git, reproduce the defect and add the missing test.

Your turn to jam the dispenser

The shared lab for all three episodes starts with a local reproduction. Stop the previous server, then run:

bun run scenario v3-bug
bun run test
bun run build
bun run start

The initial tests and build pass. In a second terminal:

bun run observe http://localhost:8080 --glitter

The script alternates one and three portions across twenty orders. Expect ten successes and ten HTTP 500 responses, while /readyz stays at 200. The fault is explicitly injected in order.mjs for the exercise. This is not simulated canary routing: all those requests target the same local instance.

Add this case to checks.mjs, after the single-order check and inside the block where glitter is enabled:

assert.equal(
  quote(catalog, { flavour: 'unicorn', quantity: 3, glitter: true }).status,
  200
);

The test should now fail on v3-bug. Stop the server, run bun run scenario v3-fixed, then repeat tests, build and startup. The same loop should succeed twenty times. We have turned an observation into protection against repeating the defect.

Follow the stages on VKS

The GitOps repository contains independent canary/ and bluegreen/ examples. Install a compatible Argo Rollouts controller, CRDs and plugin, then create an Argo CD Application for the chosen path. These are platform prerequisites, distinct from the shop’s code. Argo Rollouts installation.

First fully deploy a v2 image, then propose the v3 digest in Git. Observe the pause and Pods. For the basic canary, use an actual Service route; port-forward does not demonstrate distribution. The README explains how to interpret old-version responses and how to abort progression.

For blue-green, inspect the preview Service while the active Service continues to target the old release. Promote the candidate only after its checks. These examples install neither weighted routing nor Prometheus analysis. They make the decisions visible before automating them.

TOP closes his notebook. The recipe has a correct price. The horn fits the machine. Counters wait until they are ready. Group orders have their own test.

The glitter can wait a few minutes. A good shop would rather serve a customer than admire a green light.

Get the next one by email

New articles and series, sent when they are published. No other mail.

One click to unsubscribe, any time.

Back to blog
Share

Related articles

  1. 11 min read

    From factory to shop: GitOps and rolling updates with TOP

    Magical Unicorn is packaged but missing from the menu. Follow TOP through Git, Argo CD and VKS to replace a release while the shop serves.

  2. 8 min read

    From lab to factory: CI/CD with Magical Unicorn flavour

    TOP invents Magical Unicorn cotton candy. Red tests, a blocked PR and a versioned image: understand CI with GitHub Actions.

  3. 8 min read

    Migrating WebShop Without a Big Bang: Blue-Green Cutover, Data, and Rollback

    Prepare parity, data, DNS/LB cutover, and the reverse path to migrate WebShop without an irreversible move.

Follow along

New articles, thoughts, and updates.