AI Reply Trials: Assign the Customer Before the Message

A business receives sixty messages during a trial of AI-assisted replies. That sounds like sixty opportunities to learn. But if twelve customers each sent five messages, the business has twelve customers, each with a conversation history. Randomly changing the reply method for every message would give some people a mixture of both experiences.

Before asking whether the AI version performed better, ask who received which version. For a comparison between customer groups, the assignment should follow the customer through the part of the journey being tested. Otherwise, a clean-looking dashboard can conceal a tangled experiment.

The example below is fictional. It illustrates experiment design, not a claim that AI-assisted replies increase sales.

Define the workflow you want to compare

Imagine a small studio selling a recorded illustration course. It wants to compare two ways of answering first-time enquiries: its existing approved reply template and an AI-assisted draft checked by a staff member. Both must give accurate information about the same course, price and access terms.

The question is whether offering the second workflow changes the proportion of eligible customers who book within seven days of their first enquiry. It is not a contest to establish whether AI is universally more persuasive. Human review remains part of the version being tested.

Microsoft’s pre-experiment guidance, published on 31 July 2020, recommends a clear hypothesis and metrics capable of testing it. It also stresses planning whether enough randomly assigned units will be affected to detect the anticipated change.

For this studio, bookings would be the main outcome. Staff time and factual corrections would be additional checks: a booking advantage would not settle whether the workflow was practical to operate.

Assign the customer before sending the reply

The entity assigned to a version is called the randomisation unit. Statsig’s experiment documentation, checked on 24 October 2026, distinguishes users, devices and sessions. Its system keeps each value of the chosen identifier in the same variant. Those identifiers have different limits: a person can use several devices, while several people can share one device.

The practical implication for our fictional studio is to define an eligible customer, randomly allocate that customer to one workflow and retain the assignment for their trial conversation. A second email should retrieve the existing assignment rather than trigger another draw.

An existing customer reference may serve that purpose if it reliably represents the chosen unit. An unresolved identity should remain visible as a measurement problem. Quietly treating every new address, browser or message as a new person would change what the trial measures.

Keep repeated activity attached to its source

Return to the invented twelve-customer example. Sixty messages can describe workload, but they do not become sixty separately assigned customers. For the proposed yes-or-no booking outcome, there are twelve customer records to follow.

Keep the underlying events: the first eligible enquiry, assignment, replies and booking date. Summarise the booking outcome once per customer under the agreed definition. Message count can remain a separate outcome, such as follow-up messages per customer.

That distinction matters operationally. One reply method might invite extra questions. If every follow-up became another participant in the report, the workflow would help determine how many supposed participants it contributed. Retaining customer identities makes that circular counting visible.

Sometimes the customer is a whole organisation

The studio might also sell course access to companies. Three employees could discuss the same purchase, forward replies and make one purchasing decision. Assigning each employee separately would not necessarily create three isolated experiences.

In its 4 November 2020 explanation of organisation-level experiments, Microsoft describes assigning an entire tenant to one version when users need a consistent experience. It warns that activity within organisations is related and that changing to organisation-level assignment brings challenges for statistical analysis and the ability to detect effects.

Applied to this hypothetical corporate-booking trial, the company or booking account might therefore be the appropriate unit. That needs deciding before allocation. Ten companies with many contacts are still ten assigned companies; treating every contact as independent would not repair the limited number of organisations.

For that design, arrange analysis that accounts for the grouping. Do not borrow the precision of a large consumer experiment simply because the inbox contains many rows.

Plan for an inconclusive result

The GOV.UK guide to A/B testing, published in January 2020 and updated that September, advises planning sample size and duration around traffic and the expected behavioural change. Although written for digital health products, those design questions are relevant here. Its guidance explicitly allows for inconclusive results.

Give every included studio customer the same seven-day opportunity to book. If enrolment ends on Friday, a Friday enquiry cannot be judged as a non-booking that evening while Monday enquiries had longer to respond.

Agree the analysis and stopping approach in advance. A numerical lead on an interim dashboard is not, by itself, a reason to announce a winner. Where volume cannot support the intended comparison, narrow the claim. A pilot may reveal broken routing or difficult reviews without establishing an effect on bookings.

Rehearse the assignment before the trial

Use a few fictional customer records to check the proposed workflow before enrolling anyone. These are practical rehearsal questions, not a substitute for statistical validation:

• Does a repeat enquiry retrieve the customer’s original assignment?
• Can you trace each reported booking back to the assigned customer and observation window?
• If one company has several contacts, does the routing match the organisational unit you chose?

Record exceptions, including a customer receiving the wrong version. Decide how such cases will be handled in the analysis before inspecting which result looks more favourable.

Make a decision the evidence can support

Name the person who will decide what happens after the trial. MaryChuks.com’s guide to decision rights offers a useful way to separate that authority from the people contributing evidence.

A defensible report explains the tested workflows, assignment unit, observed outcomes and remaining uncertainty. It can recommend further testing, a limited operational change or stopping an impractical approach. Whatever the decision, it should remain possible to reconstruct which customer experienced which workflow. That record is what turns a pile of messages into an assessable comparison.

Sources

• Microsoft Experimentation Platform: Patterns of Trustworthy Experimentation: Pre-Experiment Stage (31 July 2020).
• Statsig: Experiments Overview (documentation checked 24 October 2026).
• Microsoft Experimentation Platform: Why Tenant-Randomized A/B Test is Challenging and Tenant-Pairing May Not Work (4 November 2020).
• Office for Health Improvement and Disparities: A/B testing: comparative studies (30 January 2020; updated 3 September 2020).