An AI summary of two trial runs says the second version of a booking form generated more complaints: twelve, up from eight. That sounds like a reason to restore the old form. Now add one line the summary left out: the first version handled 200 bookings, and the second handled 600. Complaints rose in number, but they fell from four to two per hundred bookings. Both descriptions are accurate. They answer different questions, and only one helps you compare the chance that a booking generates a complaint.
This is a hypothetical example, but the tension is familiar. The count tells you how much work the team had to handle. The rate tells you how common the problem was among the bookings observed. An AI assistant asked to write a short briefing has to choose what to include. If it reports the most striking count without the number behind it, a clean sentence can leave the reader with the wrong comparison.
Why the larger number can feel more persuasive
In a 1994 experiment, psychologists Veronika Denes-Raj and Seymour Epstein offered participants a chance to win by drawing a red jelly bean. Participants sometimes favoured a bowl with more red beans despite its worse odds. The paper gives the example of seven red beans among a hundred versus one red bean among ten. The larger bowl offers more winning beans to picture, but a single draw from the smaller bowl has the better chance.
A 1995 paper by Denes-Raj, Epstein and Jonathan Cole reported experiments where even equivalent low probabilities could feel different when expressed with larger or smaller numbers. Neither paper tested AI summaries or modern dashboards. The practical inference is narrower: the way a comparison is written can put the reader’s attention on part of the fraction, even when the arithmetic is straightforward.
That is one reason to keep the denominator, the total that the count is divided by, beside the headline number. Asking for it is a small act of editorial judgement. It is also a way to decide whether a summary should be making a comparison at all.
Decide what the number is meant to explain
Return to the booking form. If the business is planning support shifts, twelve complaints matter because they represent more cases to handle. If it wants to understand how often complaints occur, two per hundred compared with four per hundred matters. A manager could reasonably plan for more support work while keeping the newer form in use. A summary that collapses these decisions into a single verdict makes it harder to see that both responses can be sensible.
Check that you are dividing by the right thing. If one booking can generate several support messages, messages per booking measures workload, while the share of bookings that generate at least one complaint measures something different. Dividing complaints by site visits would answer another question again. A tool that has only a list of support messages cannot infer how many customers had an opportunity to complain. In that case the useful answer is a count and an explicit statement that the rate cannot be calculated from those records.
Even a correct denominator cannot repair an unfair comparison. Suppose the newer form ran during a quiet week and the older one was used during a promotion. The two groups may differ in what they booked, how they arrived or what kind of help they needed. The observed rates can still describe the two periods, but those rates alone cannot tell you the form caused the difference. That distinction belongs in the briefing, especially if the decision would be expensive to reverse.
Give the comparison a format you can check
Ask your assistant for one line per group with the outcome count, the eligible total, the date range and the resulting rate. Keep the counts alongside any common scale you use, such as per hundred bookings. Three returns out of ten and thirty out of a hundred both give 30%, but those fractions tell you how much activity each rate represents. If a rate looks surprising, calculate it from the original counts and look for rows that might have been counted twice or left out.
- Name the decision. Say whether you are comparing workload, the frequency of a problem or the effect of a change. Those are three different questions.
- Keep unknown totals visible. Ask the assistant to mark a missing denominator rather than silently substituting a nearby number.
- Check what qualified. Compare the period, the unit counted and the inclusion rules before treating two rates as peers.
- Use plain language for the finding. For the booking example: total complaints increased from eight to twelve, while complaints per hundred bookings fell from four to two.
Give the original table a place in the handover, even if the team reads only the executive summary during the meeting. A colleague should be able to reproduce the comparison without reopening your chat or guessing which rows were used. The MaryChuks article on checking a familiar name in AI research applies a related habit to another problem: go back to the evidence before turning a persuasive cue into a recommendation.
The next time a briefing tells you that something happened more often, pause before acting on the larger number. Ask whether there were more events, more chances for them to happen or both. The answer can change what you fix, what you staff for and what you still need to find out.
Sources
- Denes-Raj and Epstein: Conflict between intuitive and rational processing (May 1994).
- Denes-Raj, Epstein and Cole: The Generality of the Ratio-Bias Phenomenon (October 1995).
Discover more from Marychuks.com AI, Psychology, Business & CreativeVerse
Subscribe to get the latest posts sent to your email.