AI Service Limits: Plan for Bursts as Well as Volume

A small studio sends six caption requests to an AI service at once. Four are rejected. The owner looks at the advertised per-minute allowance and asks a reasonable question: how can six requests be too many when the minute’s allowance is larger than six? The missing detail may be when those requests arrived.

For a business commissioning an AI workflow, volume and arrival pattern are separate planning questions. A daily or per-minute figure does not necessarily describe the burst a service will accept at one moment. Before promising customers a turnaround time, ask how the workflow handles that difference.

Read the time window behind the allowance

Microsoft’s current Azure OpenAI quota guidance explains that requests-per-minute enforcement expects an even flow. It can evaluate arrivals over shorter periods, typically one or ten seconds. A burst can therefore receive a 429 response even when the total measured across a minute remains below the stated per-minute limit.

The same guidance distinguishes request-rate limits from token-rate calculations and notes that provisioned throughput is evaluated differently. Those details belong to the documented service and deployment type. They are reasons to inspect the applicable policy, rather than assume every AI provider uses one universal rule.

The business implication is practical: “our monthly usage is low” does not explain whether a Monday-morning batch fits the service’s short-term constraints. Ask the builder to describe both the overall workload and its busiest arrival pattern.

A bucket illustrates rate and burst capacity

Amazon API Gateway’s documentation describes token-bucket throttling, with a replenishment rate and a separate bucket capacity governing bursts. In that explanation, a token counts for a request; it is not an AI text token. AWS also cautions that its throttling settings are best-effort targets rather than guaranteed ceilings.

To make the distinction visible, consider an invented, strict admission policy for our fictional studio. This is a simplified model, not an AWS configuration or a claim about an AI supplier’s actual allowance:

• The bucket starts full with two admission credits and can never hold more than two.
• Credits replenish at one every five seconds, equivalent to a steady replenishment rate of twelve per minute.
• Each submitted request consumes one credit if a credit is available.
• A request arriving without a credit is rejected immediately; the model has no waiting queue, retries or other limits.

The replenishment rate is not a promise that twelve requests can arrive together. Nor is it a hard ceiling of twelve admissions in every minute: the initial credits and the bucket’s capacity are part of the policy too.

Submit all six fictional requests at second zero. The first two consume the available credits. The other four are rejected because no time has passed for replenishment. The fact that six is smaller than twelve does not change the empty bucket.

Pacing changes the admissions, not the definition of completion

Now submit the same six requests at seconds zero, five, ten, fifteen, twenty and twenty-five. Under the stated model, each request finds an available credit. All six are admitted, and the last submission occurs at second twenty-five.

A small JavaScript simulation checked both sequences for this article. It produced two admissions and four rejections for the simultaneous burst, and six admissions for the paced sequence. This checks the arithmetic of the invented policy. No live service was called, and no caption was generated.

The second sequence is one possible pacing arrangement, not a universal recommendation or the fastest schedule. Most importantly, second twenty-five is the last submission time. The model contains no processing-time information, so it cannot tell the studio when the final caption becomes available or passes review.

Keep those milestones separate in the operating brief: waiting locally, submitted, accepted, result received and approved for use. A timer that measures only submission can give an incomplete account of what the customer is waiting for.

Make rejection a visible operating state

Section 4 of RFC 6585, published in April 2012, defines HTTP 429 as too many requests within a given time. The response may include a Retry-After header indicating how long to wait. The standard does not prescribe one method for identifying the user or counting requests.

That code is useful evidence to retain, alongside the provider’s explanation and any timing headers. It does not, by itself, show that an application has stored the rejected job for later. Ask where pending work lives and which component is responsible for trying again.

For our fictional studio, a proposed design would keep the six approved jobs in a local work list and submit them under the documented pacing policy. Each job would retain its own status and result. If several workers share the same allowance, the builder would need to demonstrate how their combined traffic is controlled.

Retry behaviour deserves a separate review. MaryChuks.com’s guide to checking before repeating an AI workflow action explains why an uncertain outcome and a confirmed rejection require different handling. Repeated attempts should preserve the identity and purpose of the original job.

Ask for evidence from the busiest ordinary batch

A useful supplier demonstration should resemble the work the business expects to submit. For the studio, that means a batch rather than a single successful request. Agree an authorised test environment and ask the builder to show:

• The applicable allowance, enforcement window and scope: which account, deployment or other shared resource it covers.
• The planned arrival pattern, including jobs created together by a scheduled routine.
• The record of admitted, waiting and rejected jobs, with no missing work hidden by an overall success message.
• The handling of documented retry timing, a stopping point and a route for unresolved jobs.
• The time until usable results are returned and reviewed, separately from submission timing.

Keep the test bounded and record its conditions. A successful demonstration establishes what happened for that workload and setup. It does not establish that every future batch will finish by the same deadline.

When a workflow encounters limits, investigate the constraint before assuming the business needs a larger plan. The issue might be burst timing, shared traffic, a different token limit or another service condition. The owner needs an explanation supported by the actual records.

A useful capacity brief connects the volume of work, the way it arrives and the handling of each job. That gives the business a basis for deciding whether to pace, queue, reschedule or investigate additional capacity—and for making a customer promise that extends beyond a headline allowance.

Sources

Primary documentation checked on 30 October 2026. The studio, admission policy and simulation are illustrative; no production workload or supplier capacity test was performed.

• Microsoft Learn: Manage Azure OpenAI in Microsoft Foundry Models quota — current service-specific guidance.
• AWS: Throttle requests to your REST APIs for better throughput in API Gateway — token-bucket rate and burst guidance.
• IETF RFC 6585, section 4 — 429 Too Many Requests, April 2012.