AI Trials: Count Who Made It Into the Results

A small studio tests whether AI-drafted replies help it answer project enquiries sooner. Half of eligible customers are assigned to the usual process and half to the new one. At the end, the team sees faster replies in the AI group. There is one odd detail: the report contains far fewer customers in that group. The report might be accurate about the people it includes. But if the slowest AI cases disappeared on the way to the dashboard, the studio may be about to reward the very problem it wanted to fix.

Before discussing the result, stop at the denominator. Who was supposed to be in each group, who actually appeared and what happened in between? You can ask these questions without a sophisticated experiment platform. They matter whether you run a trial in a spreadsheet or through a dedicated testing service.

Start with the split you planned

In a 2019 conference paper on online experiments, Aleksander Fabijan and colleagues describe sample ratio mismatch: the groups represented in the data differ from the allocation the experiment was designed to produce by more than you would reasonably expect from chance. The authors use examples from software experiments to show why it is a warning with several possible causes. A failed check does not, by itself, tell you which record was lost or where the problem began.

The expected split depends on your design. If you deliberately send only a quarter of customers through the AI workflow, unequal group sizes are normal. Even with random assignment intended to split customers equally, the numbers need not be exactly equal. An analyst can check whether an imbalance is surprising for the planned allocation and sample size. If your design fixes the number assigned to each group, compare the assignment log with the report directly: the log gives you the actual people to account for.

Count the unit you assigned. If each customer gets one assignment, count customers once for the group-size check. One customer who sends five messages is still one customer. You can measure individual reply times as a separate question, with an analysis suited to repeated messages. MaryChuks.com’s guide to assigning the customer before the message explains why that initial choice matters.

Trace the missing records through the workflow

Take a hypothetical trial where the assignment log has exactly 100 customers per group. The final report shows 98 under the usual process and 72 under the AI process. The team discovers that when the assistant cannot draft a reply, the enquiry goes to a manual queue. Those customers remain assigned to the AI workflow, but the report only takes AI cases with a saved draft. It has quietly changed the question from “What happens under this workflow?” to “What happens when this workflow produces a draft?”

Start with the assignment log and follow a few missing cases. For each one, look for the route it took, the records it created and the rule that decided whether it reached the report. Sometimes the experiment itself applied the wrong allocation. Sometimes the allocation was correct and a filter or missing event made it look wrong. In the studio example, the missing customers may be among those who waited longest. Simply removing an equal number from the usual group would make the counts look neater without making the comparison fairer.

Have a definition for an unfinished enquiry too. An enquiry still waiting at the reporting cutoff has no completed reply time. Filling the blank with zero would imply that it received an immediate reply; ignoring it can make the average look better. One useful check is to define a response window before the trial and compare the proportion of eligible customers who received a reply within it, once everyone has had that much time to be observed. Keep actual reply times and open cases visible alongside that measure.

Give the team a rule for pausing the decision

• Write down the expected allocation. Record the unit, planned proportions and eligibility rules before collecting results. Keep enough information to match an assigned record to its route through the trial.
• Reconcile the main counts. Compare assigned customers, customers routed to each process and customers with an observed outcome. Show manual fallbacks and unresolved cases separately so they do not vanish in a total.
• Investigate before calling a winner. If the groups are unexpectedly imbalanced, keep the result provisional. Repair the reporting rule if the necessary records exist; consider rerunning the trial if you cannot reconstruct trustworthy data.
• Keep narrower questions separate. It can be useful to know how fast drafts were reviewed when the tool produced one. Label that result clearly and report how often people needed a different route.

Equal-looking groups are no guarantee that the trial is sound. Both sides might lose records, or a timing rule might exclude the same kinds of enquiries from both. The group-size check is one way to tell when a dashboard needs a second look. For the studio, a persuasive result begins with a trail from each assignment to the outcome the customer actually experienced. If that trail breaks, fix the measurement before changing the workflow.

Run this check on a small batch before the full trial begins. Include one successful draft, one manual fallback and one enquiry that is still open. The point is to see whether your reporting process can represent all three outcomes. Discovering a missing route on day one gives you a chance to fix it before anyone depends on the results.

Source

• Aleksander Fabijan and colleagues, Diagnosing Sample Ratio Mismatch in Online Controlled Experiments, ACM SIGKDD (2019).