Imagine a framing shop on the morning it opens bookings for a popular exhibition. Customers want to know whether their mounts will be ready for collection. At the same time, the site is using AI to generate room previews for people browsing frame styles. Both features call the same service. As demand rises, previews take longer, customers try again, and the collection page stops responding. The shop has a choice about what to keep serving, but only if it makes that choice before the queue fills.
Start with a question the customer would recognise: what can they still get done when the AI is busy? The shop could keep verified collection details available, defer some room previews and explain which requests it has accepted. That is a more useful plan than promising that every feature will simply slow down together.
Decide what to accept before work starts
In a December 2019 account of load shedding at Amazon, engineer David Yanacek explains the reasoning behind refusing some excess work early: it can help a service keep completing the requests it has capacity to handle. Google’s Site Reliability Engineering chapter on overload also discusses simpler responses and treating requests differently according to their importance. Those are engineering examples. For a small shop, the practical work is deciding what customers can reasonably postpone and what they need now.
Do not confuse importance with how loudly a feature announces itself. An expensive room preview may be prominent on the homepage, but a customer travelling across town needs accurate collection information more urgently. The shop can use the existing order record to show confirmed details without asking a model to rewrite them. If the record does not confirm readiness, say so and explain how to check. A simpler response is only useful if it preserves the facts that matter.
Some work can wait if the customer chooses to wait. Others should be declined immediately. A preview needed to make a purchase today may be useless tomorrow. The shop could offer to queue it, but the offer should say when the result is likely to arrive, and the customer should be able to decline. If there is no reliable estimate or no room in the queue, a clear message and a static gallery may be a better experience than an indefinite spinner.
Tell people whether their request was accepted
For each option, decide what the customer can safely assume. If the request was accepted, the site can show that it is queued and give the customer a way to check its progress. If it was declined, say that no preview was created and offer the next available route. An ambiguous message such as ‘Something went wrong’ leaves people wondering whether to start again. If the request did enter the queue, a second click can create more work unless the workflow recognises it as the same request.
Do not make an immediate retry the only choice. It can send customers back into the same overloaded path, especially if many of them try again together. In this example, a gallery of existing styles lets people keep browsing. A queue status lets someone who already submitted a preview check it without submitting another. The aim is to give customers an action that helps them, even when the original feature is temporarily unavailable.
Check what happens after the rush
When capacity returns, decide which deferred requests still deserve to run. A preview might have been cancelled, superseded by a later request or tied to an order the customer has already placed. An accepted request should end in a visible outcome, whether that is a result or a clear explanation that it expired. Draining the entire queue without checking whether the work is still useful can spend the capacity you just recovered.
Measure the effects in both routes. If the collection page works but previews are turned away all afternoon, the immediate priority may have been protected while another part of the customer experience suffered. Track what was completed, deferred, declined and eventually abandoned. Our guide to isolating AI workloads covers a complementary design choice: giving essential work a separate resource boundary. Decide what to admit even if those boundaries already exist.
Rehearse the decision with a small test
- Use an authorised test environment to simulate a busy period. Keep several collection checks running while preview requests build up, and verify that the details customers need remain accurate.
- Test the queued, declined and expired messages separately. For each, check whether a reader can tell if the request exists and what to do next.
- Repeat a preview submission and check that the customer does not get two jobs when they meant to check one. Test a cancelled request during recovery.
- Ask the person handling customer enquiries to review the fallback. Agree who can tighten or relax the limits, and what evidence they should look at first.
Then walk through the same exercise from a customer’s point of view. If you needed to leave for the shop in ten minutes, would the fallback tell you whether to go? If you were comparing frames, would you know whether your preview was still coming? Those answers reveal whether the policy protects a real task. The best overload plan gives customers a clear way forward while capacity is limited.
One thought on “AI Overload: Decide Which Requests Can Wait”