Rate Limiting LLM APIs Without Breaking User Experience
Rate limiting for LLM APIs is fundamentally different from rate limiting stateless web APIs. The unit of cost is tokens, not requests. Responses are generated over seconds, not milliseconds. A poorly designed rate limiter creates user experience failures that are worse than a brief service disruption. This post covers how to do it right.
Token-Based Rate Limiting
Request-based rate limits make sense when all requests have similar cost. LLM requests vary by orders of magnitude — a one-sentence classification takes 50 tokens while a complex multi-document analysis takes 32,000. Rate limiting by request count allows expensive requests to exhaust budgets that should support far more inexpensive ones. Rate limit by tokens instead.
Multi-Level Rate Limits
Effective LLM rate limiting operates at three levels simultaneously: global limits protect your provider spend and SLA, per-tenant limits prevent any single customer from monopolizing capacity, and per-feature limits allow product teams to set budgets for individual AI-powered features. These levels should be configurable independently.
Queue-Based Rate Limiting
Hard rate limits that reject requests with 429 errors produce visible failures. Queue-based rate limiting defers requests instead: when the rate limit is hit, the request enters a queue and is executed when capacity becomes available. For user-facing features this requires clear UI feedback about queue position. For batch workloads, queueing is transparent and strongly preferred over rejection.
Communicating Limits to Users
When a user hits a rate limit, the quality of the error message determines whether they have a tolerable or frustrating experience. Specific, actionable messages — estimated wait time, link to upgrade options, alternative action suggestions — reduce support load dramatically compared to generic rate limit errors.