Imagine a small business trying a new AI model for drafting replies to delivery enquiries. It gives the new version only a small share of live requests and keeps the old version for the rest. The dashboard still looks healthy. On the review desk, though, several drafts from the new version promise a delivery date the order record does not support. The rollout has limited the number of bad drafts. The dashboard has also made them harder to see.
A canary release lets a new version handle a limited share of real work before you decide whether to expand it. It gives you a chance to spot problems while most of the business still runs on the previous version. That opportunity depends on reading the small group separately and checking outcomes that matter to customers.
Keep the denominator visible
Google’s Site Reliability Workbook chapter on canarying releases describes comparing a small portion of production on the candidate version with a control and warns that service-wide metrics can obscure the candidate’s failures. Imagine 1,000 reviewed drafts: 950 from the old version and 50 from the candidate. Suppose reviewers flag nine old drafts and ten candidate drafts for unsupported promises. The combined rate is 1.9%. In the candidate group, it is 20%. The overall number describes the whole service; it tells the rollout owner very little about whether to expand the candidate.
These are invented numbers, and in a real trial the comparison depends on who gets which requests. If the candidate group handles mostly delayed parcels and the old version handles mostly routine tracking questions, the difference may partly reflect the cases. Where feasible, assign comparable requests to both groups during the same period. If you start with a deliberately narrow group, say which types of work the result covers and which still need to be tested.
Measure the output someone has to use
A drafting system can return a response quickly and still give staff something they should never send. For delivery enquiries, ask reviewers to check the order record and mark any draft that promises a date without support. Decide how to count a draft that states a reasonable estimate but words it as a guarantee. A simple rubric with two or three examples is more useful than a generic instruction to rate quality.
Keep an eye on what happens after the draft arrives. If staff must spend longer correcting it, a faster model response may not shorten the time to a safe reply. If the new version hands more requests to a person, check how many and whether the queue can absorb them. Label every draft by the version that actually produced it, including cases in which the workflow fell back to the old version. Otherwise a successful fallback can hide a problem you meant to measure.
Where practical, let reviewers judge drafts against the same rubric without showing the model version first. Keep serious failures separate from stylistic disagreements. A single unsupported promise may deserve an immediate pause if the workflow sends replies without review. In a trial where staff approve every draft, it may instead trigger a targeted investigation. Set that response before the first request arrives.
Decide what evidence is enough
The Google chapter also discusses the trade-off between keeping exposure low and letting the candidate see enough varied traffic to learn something useful. Ten routine enquiries on a quiet morning tell you little about an afternoon when tracking data is late. A low-volume business can prepare examples of uncommon but costly failures before using live requests. Then let the live trial answer questions that those prepared cases cannot, such as how often the new version increases the work on the review desk.
Write down who can pause the rollout, which finding triggers a pause and what happens to drafts already produced. Switching new requests back to the old version does not withdraw a draft sitting in an approval queue. Decide whether those drafts need a fresh check before anyone sends them. The companion guide on using an error budget to guide release decisions looks at a broader question: whether the service has room for another change at all.
Watch for a delay between the draft and the evidence you need. If staff review Friday’s drafts on Monday, Friday’s dashboard can show a full batch of successful responses before anyone has assessed the promises inside them. Mark unreviewed drafts as pending in the release report. A low error count means something different when half the candidate’s work has yet to be checked.
Make the release decision explicit
- Name the comparison group and record the version, number of requests and number actually reviewed for each group.
- Choose one or two customer-facing failures to track and define them with examples. Keep counts alongside rates so a handful of cases is visible.
- Check whether the trial includes the kinds of requests that matter, such as an order with conflicting delivery information or a request received at peak time.
- Agree in advance when to expand, hold or stop. If the evidence is thin, keep the exposure small or do more targeted testing before expanding.
A canary limits how much work is exposed to a change. It cannot make a mixed dashboard tell you what happened to that work. Before you expand an AI rollout, ask the release owner to show the candidate’s results on their own, explain the comparison and name the cases the trial did not cover. That is the decision the small rollout was there to support.
Source
Discover more from Marychuks.com AI, Psychology, Business & CreativeVerse
Subscribe to get the latest posts sent to your email.
