Fundamentals
What is Rate Limiting?
Rate limiting controls how much traffic a client or workload is allowed to send during a defined period or concurrent execution limit.
Client
1000 requests
Rate Limiter
Allowed: 100 req/min
Application
Protected
Rate limiting is a protection mechanism
Why?
Why Do We Need Rate Limiting?
Traffic can become dangerous even when every request is technically valid.
Prevent Abuse
Stop clients from sending unlimited API requests.
Protect Infrastructure
Prevent traffic from overwhelming databases, queues and downstream services.
Control Cost
Protect expensive resources such as AI inference, third-party APIs and payment providers.
Fairness
Prevent one tenant or user from consuming all available capacity.
DDoS Mitigation Layer
Rate limiting can be one part of a larger abuse and traffic-protection strategy.
Business Quotas
Different plans can receive different API limits.
Architecture
Where Should the Rate Limiter Live?
Rate limiting can be enforced at several layers, and large systems often combine them.
Internet Traffic
CDN / Edge / WAF
Coarse IP / bot limits
API Gateway
User / API key / route quotas
Application
Business-specific limits
DB / Downstream Services
Protected resources
Layer your limits
Identity
What Can We Rate Limit By?
The limiter key determines who shares the quota.
IP Address
Useful for unauthenticated traffic and basic abuse control.
User ID
Good for authenticated user-level quotas.
API Key
Common for developer APIs.
Tenant / Organization
Protect multi-tenant systems and enforce plan limits.
Endpoint
Expensive endpoints can have stricter limits.
Operation
OTP, password reset, AI generation or export can each have separate quotas.
rate-limit:
tenant:company_123:
user:user_42:
route:/api/paymentsAlgorithm 1
Fixed Window Counter
Split time into fixed intervals and maintain a request counter for each client and interval.
Limit:
100 requests / minute
Window 1:
10:00:00 - 10:00:59
Window 2:
10:01:00 - 10:01:59Boundary Problem
10:00:59
-> 100 requests accepted
10:01:00
-> another 100 requests accepted
Result:
200 requests may arrive
within approximately 1 second.Why use it?
Algorithm 2
Sliding Window Log
Store individual request timestamps and remove entries older than the rolling window.
Current time:
10:01:20
Window:
last 60 seconds
Stored timestamps:
10:00:25
10:00:40
10:00:55
10:01:10
10:01:18
Remove anything older than:
10:00:20
Count remaining timestamps.Benefit
Very accurate rolling-window enforcement.
Cost
Potentially store one timestamp for every request.
Algorithm 3
Sliding Window Counter
Approximate a rolling window using weighted counts from the current and previous fixed windows.
Previous window count = 80
Current window count = 20
Current window is 25% complete.
Approximation:
80 * 0.75 + 20
= 80 requestsGood compromise
Algorithm 4
Token Bucket
Tokens enter a bucket at a configured rate. A request is accepted only when enough tokens are available.
Refill
+10 tokens/sec
Token Bucket
Capacity = 100
API
Request allowed
Why Token Bucket is Useful
If the bucket has accumulated tokens, the client can send a short burst. But over time, traffic is still restricted by the token refill rate.
function allowRequest(bucket) {
now = currentTime()
elapsed =
now - bucket.lastRefill
newTokens =
elapsed * bucket.refillRate
bucket.tokens =
min(
bucket.capacity,
bucket.tokens + newTokens
)
bucket.lastRefill = now
if (bucket.tokens < 1) {
return false
}
bucket.tokens -= 1
return true
}Algorithm 5
Leaky Bucket
Requests enter a bucket or queue while work leaves at a controlled rate.
Burst Traffic
1000 req/sec
Bucket / Queue
Finite capacity
Backend
100 req/sec
Token bucket vs leaky bucket
Expensive Operations
Concurrency Limiting
Requests per second alone may not protect a resource when each request has a very different execution time.
Maximum concurrent AI requests:
20
Current in-flight:
20
New request:
REJECT / QUEUERPS Limit
Controls how many requests begin during a time interval.
Concurrency Limit
Controls how many operations can remain active at once.
Selection
Which Rate Limiting Algorithm Should You Use?
The correct choice depends on accuracy, burst tolerance, state size and complexity.
Fixed Window Counter
SimpleCount requests inside fixed time buckets such as 10:00:00–10:00:59.
Good For
Simple APIs where exact fairness around window boundaries is not critical.
Benefit
Very simple, cheap and easy to implement.
Trade-off
Allows burst traffic around window boundaries.
Sliding Window Log
AccurateStore timestamps of recent requests and count only timestamps inside the current rolling interval.
Good For
Low-to-medium traffic systems where precise rolling-window limits matter.
Benefit
Very accurate rolling limit.
Trade-off
Memory cost grows with request volume.
Sliding Window Counter
BalancedApproximate a sliding window by combining the current and previous fixed-window counts.
Good For
APIs needing smoother boundaries without storing every request timestamp.
Benefit
Good accuracy with much lower memory than a request log.
Trade-off
Still an approximation.
Token Bucket
BurstsTokens refill at a constant rate. Every request consumes a token.
Good For
APIs that want to allow controlled bursts while enforcing a long-term average rate.
Benefit
Allows bursts without giving unlimited traffic.
Trade-off
Needs atomic token refill and consumption logic.
Leaky Bucket
SmoothIncoming work enters a bucket and leaves at a controlled constant rate.
Good For
Protecting systems that need smooth downstream request flow.
Benefit
Turns bursts into predictable output traffic.
Trade-off
Queued requests may experience latency or be dropped when the bucket fills.
Concurrency Limiter
In FlightLimits how many expensive operations can execute at the same time.
Good For
Expensive DB queries, AI inference, uploads, report generation and slow downstream calls.
Benefit
Protects limited resources even when requests have variable durations.
Trade-off
Does not directly control total requests per second.
Distributed Systems
Local vs Distributed Rate Limiting
A limiter inside one application server is easy, but multiple application instances create a coordination problem.
Broken Local Design
User
Limit = 100/min
API 1
100 allowed
API 2
100 allowed
API 3
100 allowed
Effective limit may become 300/min.
Shared State
API 1
API 2
API 3
Redis
user:42 => 76 / 100
Implementation
Redis-Based Rate Limiting
Redis is commonly useful because application instances can share fast counters and expirations.
key =
"rate:user:42:202609291546"
count =
INCR key
if count == 1:
EXPIRE key 60
if count > 100:
rejectRequest()
INCR and EXPIRE must be handled safely
Concurrency
Why Atomicity Matters
Two requests can reach the limiter at exactly the same time.
Request A:
GET counter -> 99
Request B:
GET counter -> 99
A thinks:
99 < 100 -> allow
B thinks:
99 < 100 -> allow
Both update counter.
Limit is violated.Use atomic Redis operations, scripts or another shared state mechanism that performs check + update as one logical operation.
local current =
redis.call(
"GET",
KEYS[1]
)
if current and
tonumber(current) >=
tonumber(ARGV[1])
then
return 0
end
local value =
redis.call(
"INCR",
KEYS[1]
)
if value == 1 then
redis.call(
"EXPIRE",
KEYS[1],
ARGV[2]
)
end
return 1API Behavior
HTTP 429 & Rate Limit Responses
Clients should receive enough information to understand that the request was throttled and, where appropriate, when to retry.
HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: application/json
{
"success": false,
"message": "Rate limit exceeded",
"retryAfterSeconds": 30
}Do not make clients retry immediately
Traffic Spikes
Handling Burst Traffic
Rate limits need to distinguish between acceptable short bursts and sustained overload.
Normal Traffic
100 req/sec
10,000 req/sec
Sudden burst
Token Bucket
Allow bounded burst
Backend
Protected
Burst Capacity
Allows temporary traffic above the steady refill rate.
Sustained Rate
Controls long-term throughput.
Queueing
Some workloads can wait instead of being rejected.
Load Shedding
Optional requests may be rejected under extreme pressure.
Scale
Rate Limiter Hot-Key Problem
A single very popular identity can create enormous traffic against one Redis key.
R1
R2
R3
Millions
rate:api-key:popular-key
Hot Redis key
Local + Global Limits
Enforce small local limits first and use global storage for authoritative quota.
Sharded Counter Design
For some approximate limits, distribute counter work across multiple keys.
Edge Enforcement
Reject obvious traffic before it reaches central infrastructure.
Plan-Specific Architecture
Very high-volume tenants may need dedicated quota state.
Global Systems
Multi-Region Rate Limiting
A global limit becomes difficult when traffic is handled independently in several geographical regions.
Global User
Limit = 1000/min worldwide
India
400
US
400
Europe
400
Independent regional limits could allow 1200 instead of 1000.
Design Choices
Central Global Counter
More accurate globally, but adds latency and cross-region dependency.
Regional Quota Allocation
Split 1000 global tokens among regions. Fast locally but unused quota can become stranded.
Eventually Synchronized Counters
Lower latency but may temporarily exceed exact global limit.
Hybrid
Enforce strict local ceilings while asynchronously coordinating global quota.
Ask whether the limit must be exact
Case Study
BookMyShow-Style Flash Sale Rate Limiting
Imagine tickets for a blockbuster movie opening at exactly 10:00 AM.
Interview-style architecture
500K+ Users
Tickets open
CDN / Edge
Static assets + coarse abuse controls
Rate Limiter
IP + user + booking limits
Load Balancer
Booking 1
Booking 2
Booking N
Redis
Seat holds + quota state
Booking DB
Durable booking
Example Limits
Per IP
Reduce obvious bot floods.
Per User
Prevent one account from creating excessive booking attempts.
Per Show
Protect a highly popular show from overwhelming downstream booking state.
OTP
Prevent unlimited OTP generation.
Payment Attempts
Protect payment providers and reduce repeated abuse.
Search
Allow higher throughput because search is less correctness-sensitive than booking.
Layered Flash-Sale Protection
Edge filters obvious abusive traffic.
Gateway enforces IP, user and API-level quotas.
Booking service applies stricter business limits.
Token bucket allows a controlled burst.
Load balancer spreads accepted traffic.
Redis handles temporary distributed quota state.
Seat locking prevents concurrent booking conflicts.
Database protects final booking correctness.
Rate limiting does not solve seat concurrency
Production Engineering
Rate Limiter Failure Scenarios
A limiter is part of the critical request path, so its own failures need explicit handling.
Redis Is Down
Decide whether the endpoint fails open or fails closed. The correct behavior depends on the risk of allowing traffic versus rejecting valid users.
Race Condition
Use atomic state changes so concurrent requests cannot exceed the quota.
Clock Skew
Distributed algorithms relying on timestamps need a consistent time strategy.
Hot Key
A highly active user, API key or tenant may overload one quota key.
Limiter Adds Latency
Use low-latency storage, local optimization or hierarchical limits where appropriate.
Configuration Error
A limit of 10/min accidentally deployed instead of 10,000/min can cause an outage.
Regional Partition
Global quota enforcement may become inconsistent when regions cannot communicate.
Retry Storm
Rejected clients must use backoff instead of immediately retrying.
Fail Open vs Fail Closed
Fail Open
If limiter storage fails, allow the request.
Useful when availability is more important than strict quota enforcement.
Fail Closed
If limiter storage fails, reject the request.
Useful when the protected operation is very expensive or abuse-sensitive.
Observability
Rate Limiter Metrics
Monitoring tells you whether limits are protecting the system or simply blocking legitimate users.
Interview Preparation
Rate Limiting Interview Questions
These are the concepts you should be able to explain with trade-offs.
Why do we need rate limiting?
Where should a rate limiter be placed?
Rate limiting vs throttling — how do you think about them?
What can you rate limit by?
Explain Fixed Window Counter.
What is the fixed-window boundary problem?
Explain Sliding Window Log.
Why is Sliding Window Log memory intensive?
Explain Sliding Window Counter.
Explain Token Bucket.
Why does Token Bucket allow bursts?
Explain Leaky Bucket.
Token Bucket vs Leaky Bucket?
What is concurrency limiting?
RPS limiting vs concurrency limiting?
How would you rate limit across multiple application servers?
Why is Redis commonly used?
How do race conditions break a rate limiter?
How can Lua scripts help with Redis rate limiting?
What should happen after HTTP 429?
What is Retry-After?
How do you prevent client retry storms?
How do you support different limits for free and premium users?
How would you rate limit by API key?
How would you rate limit OTP generation?
How do you handle rate-limiter hot keys?
How would you implement a global multi-region rate limit?
Exact vs approximate rate limiting?
When would you fail open?
When would you fail closed?
How would you handle Redis failure?
How would you protect a flash sale?
How would you rate limit a BookMyShow-style booking API?
How would you protect an expensive AI inference endpoint?
What metrics would you monitor?
Quick Revision
Frequently Asked Questions
Short answers to common rate-limiting questions.
What is rate limiting in system design?
Rate limiting controls how much traffic a client, user, API key, tenant, route or other identity can send during a defined interval or concurrency limit.
Which rate limiting algorithm is most commonly used?
There is no universal best algorithm. Token bucket is useful when controlled bursts are allowed, sliding windows provide smoother rolling limits, and fixed windows are attractive when simplicity matters.
Why is Redis used for distributed rate limiting?
Redis provides fast shared state, TTL support and atomic operations, making it useful when multiple application instances need to enforce one shared limit.
What HTTP status code should a rate limiter return?
HTTP 429 Too Many Requests is commonly returned when the client exceeds an applicable request limit.
Should rate limits be based only on IP address?
Usually not. IP limits are useful for unauthenticated traffic, but authenticated systems often also limit by user, API key, tenant, route or operation because many users can share an IP address.
Learn how rate limiting fits into complete system designs.
Combine rate limiting with caching, databases, load balancing, queues, distributed locking, concurrency and failure handling in complete HLD interview designs.
Continue Learning