Shadow Testing: Let AI Make a Recommendation Before It Can Act

Imagine an online shop testing an AI assistant that routes support tickets. A customer writes, ‘The courier says it was delivered, but it isn’t here. Please don’t send a replacement yet.’ The assistant picks the replacement queue. On a dashboard the ticket looks neatly classified. To the customer, a replacement would be exactly the wrong next step. The time to discover that difference is while the assistant is still making recommendations that nobody has to follow.

A shadow test gives you that opportunity. Keep the existing workflow in charge. In parallel, have the proposed system record what it would do with a copy of the same case. Then compare the two with enough context to understand the disagreements. The goal is to learn where the new workflow deserves authority and where it still needs a human decision.

Make the boundary real

AWS announced shadow testing for SageMaker in November 2022. Its technical documentation describes copying inference requests to a second model variant while returning only the production variant’s response to the calling application. That is a concrete infrastructure example. The small business exercise here borrows the principle and applies it to the whole decision, including what would happen after the model produced its answer.

For the shop, the proposed system should put its route and reason in a review sheet or separate test queue. Give it no access to the actions you are withholding: sending messages, changing order records or creating shipments. Check where the test output goes, too. If an existing integration treats every entry in a queue as a command, a test recommendation in that queue can still trigger real work. Use a separate destination and run a harmless test case to confirm that nothing downstream fires.

You can start with past cases in a controlled replay, especially if you are working alone or cannot responsibly copy live customer messages. Remove unnecessary personal details and decide who can see the sample and when it should be deleted. Where you want to reproduce the original decision, reconstruct what was known at the time. Giving the system a later courier update that staff could not see makes the comparison misleading.

Review decisions, not just matching labels

Agreement with the existing process gives you a place to start. It does not tell you whether either answer was right. Perhaps the shop’s usual process sends every missing parcel to the courier team. In the example above that might be reasonable, but only if the person handling it also knows the customer requested a pause. Record both the proposed route and the condition that must travel with it. A system that preserves the queue but drops the instruction has still changed the decision.

Give the reviewer a few useful outcomes to choose from: both routes acceptable, proposed route better, proposed route worse, or insufficient information. Write a short reason when the choice would affect a customer or create work for another team. A reviewer might note that both answers sent a case to the same place, but the proposed answer would have raised its priority for no reason. That kind of distinction is more useful for improving the workflow than a single agreement percentage.

Look separately at ordinary cases and the ones where a mistake would matter. If almost every message is a delivery status enquiry, good performance on those enquiries tells you little about cancellations, conflicting instructions or cases with missing records. Keep deliberately chosen difficult examples separate from the ordinary sample. Use them to find weaknesses; do not use their frequency to estimate how often customers encounter those problems.

Decide what would earn a limited rollout

Before reviewing the results, write down which mistakes would stop the trial from advancing, and which disagreements you could live with. The shop might halt if the assistant overlooks an explicit instruction to hold an order, but accept a choice between two queues staffed by the same people. It might also require a useful explanation whenever the assistant asks for a human decision. Put an owner on the review and agree what to do if that person is away.

Record the cost of running the parallel process, including the time spent reading and resolving disagreements. A route that looks faster on a dashboard may add a second inbox for someone to monitor. Likewise, a shadow test cannot show how customers would react to an answer they never received or how staff would use suggestions in the middle of a busy shift. If the evidence supports a next step, try a limited rollout with a named reviewer and an easy way to return to the current process.

Keep the stages visible. Our article on the difference between accepting an AI job and delivering its result makes a related point: a system status is only useful if it tells people what has actually happened. In a shadow test, a logged recommendation means the test produced an answer. It does not mean a customer was contacted, an order was changed or the workflow earned permission to make either happen.

End the trial with a decision someone can act on. For example, the shop might allow the assistant to suggest routes for routine tracking enquiries while keeping disputed deliveries in the existing manual queue. Keep a few examples that explain the boundary. When a colleague asks why the feature handles one ticket but leaves another alone, those cases will give them a better answer than a general assurance that the tool passed testing.

Sources

• AWS (30 November 2022), Introducing Amazon SageMaker support for shadow testing.
• Amazon SageMaker AI documentation, Shadow tests.