AI Near Misses: Record What Stopped the Mistake

An AI assistant drafts a message telling a customer their replacement order has been dispatched. In fact, the warehouse has only printed a label. A support worker spots the difference, edits the message and moves on. The customer gets an accurate update, so the weekly complaints report stays clean. But the business has just learnt something its report cannot show: the workflow nearly made a promise the warehouse had not fulfilled.

In this fictional example, correcting the message was the right immediate response. The leadership decision comes afterwards. Was the mistake caught by a check everyone performs, by a colleague who happens to know the warehouse system well, or by chance? Those answers point to different levels of protection. If the team records only what went wrong, it misses the chance to find out what kept the error from reaching the customer.

Define what is worth reporting

A near miss, for this purpose, is a specific event that could reasonably have caused a meaningful problem but stopped short of doing so. A rough first sentence is ordinary editing. A confident claim that an order has shipped, in a message ready to send, is different: a customer could use that information to decide whether to buy a replacement elsewhere. The boundary depends on the work. For one shop it might be an incorrect dispatch promise; for a creator it might be a scheduled email that directs readers to last year’s booking page.

There is a precedent for treating close calls as useful evidence. The US Occupational Safety and Health Administration’s December 2015 incident investigation guide recommends looking into close calls to identify underlying causes. NASA’s Aviation Safety Reporting System overview describes a voluntary, confidential and non-punitive way to receive aviation safety reports. These are examples from workplace safety and aviation. They do not establish that an identical approach will work in a small AI-assisted business. The idea worth borrowing is to make it easy to report a consequential event before harm becomes the only evidence you collect.

Record the catch while people can still explain it

Give staff a short way to report a close call from the place they already work. A small team could use four fields in its existing issue tracker. The description should contain enough detail to trace the event without copying an entire customer conversation into a shared board.

• What nearly went out or happened? Name the proposed message or action and the workflow it came from. A note that says ‘bad AI reply’ will be hard to use later.
• What made it wrong? Identify the evidence available at the time. In the dispatch example, the warehouse record showed that a label existed but did not confirm a handover to the carrier.
• What stopped it? Record whether the message was blocked by a rule, caught by a routine comparison or noticed because someone happened to check. If nobody knows, say so.
• What would have happened next? Describe a plausible consequence in plain language. Keep observed facts separate from what might have happened if the message had gone out.

Make room for the person who spots the problem to finish helping the customer first. If filing a report takes longer than correcting the issue, the reporting process competes with the work it is supposed to improve. A lead can follow up on a short report later. The person who caught the error does not also have to diagnose its cause.

Check whether the safeguard would work on another shift

Return to the dispatch example. If an experienced worker recognised the difference between a label and a carrier handover, ask whether a new starter could make the same distinction from the information on screen. Re-run a fictional version of the case using test data. If the distinction is hard to see, the team might show the last confirmed fulfilment event next to the draft or stop the draft from using ‘dispatched’ until a specified event is present.

A review can also reveal that a safeguard worked as intended. If a rule blocked the message and gave the worker a clear explanation, check that it covers the other dispatch routes before adding a second manual check. The companion article on investigating both the model and the human system is useful when an event needs deeper analysis. A near-miss note serves an earlier purpose: it preserves the catch so the team can decide whether a closer investigation is warranted.

Treat the count as a prompt to ask questions

If reports rise after you introduce a shorter form, the increase could mean more mistakes are occurring, more people are reporting them, or both. A quiet month could likewise reflect fewer close calls or less time to record them. Avoid using the raw count to rank people. Compare reports with the amount of work reviewed and, where practical, run the same spot check over comparable batches. The two sources answer different questions: what people noticed and chose to report, and what a consistent review found.

At the next review, pick one recurring or high-consequence case. Give the fix an owner and define what a fresh example should do: block the misleading statement, make the uncertainty clear, or send the case to someone with the right information. Then test that behaviour. The value of a near-miss report is that the next person on shift has a better chance of catching the same problem, even if they have never seen it before.

Sources

• US Occupational Safety and Health Administration, Incident Investigation: A Guide for Employers (December 2015).
• NASA, Aviation Safety Reporting System Overview.