Learn load balancing from first principles: why it is needed, every major type of load balancer, L4 vs L7, routing algorithms, health checks, sticky sessions, reverse proxies, TLS, high availability, multi-region traffic, WebSockets, failure handling and a complete BookMyShow-style booking case study.
Fundamentals
A load balancer is a traffic-distribution layer placed between clients and a pool of backend servers. Its job is to decide which healthy backend should receive each new request or connection.
Clients
Web, mobile, API consumers
Load Balancer
Health check + routing decision
App Server A
Healthy
App Server B
Healthy
App Server C
Healthy
A single application server has finite CPU, memory, network bandwidth and connection capacity. Even if vertical scaling gives it more resources, it still remains one failure domain.
Horizontal scaling solves this by running multiple application instances. The load balancer makes horizontal scaling useful by spreading traffic across those instances.
Depending on the type, it may know backend health, connection count, response latency, request host/path, cookies, headers, server weights and other routing signals.
It should not send new traffic to a server that has been marked unhealthy, and it should support draining when a server is being removed or deployed.
Core interview idea
Load balancing is not only about performance. It also improves availability, enables horizontal scaling, supports rolling deployments, isolates failures and creates a stable frontend while backend instances are added or removed.
The user calls a stable domain or IP.
DNS/CDN/WAF may sit before the load balancer.
Only eligible healthy targets are considered.
For example Round Robin or Least Connections.
Response returns through the proxy or directly depending on architecture.
Classification
The phrase 'type of load balancer' can mean different things. In interviews, classify them by network layer, deployment model, traffic scope and who owns the routing decision.
How to answer this in an interview
Routes traffic mainly using IP addresses, TCP/UDP ports and connection information without needing to understand HTTP business semantics.
How it works
The balancer accepts a network connection and forwards packets or connections to one of the healthy backends. Because it does not need to inspect application-level fields such as URL paths, routing can be fast and protocol-agnostic.
Common use
TCP services, UDP services, databases, game servers, high-throughput network traffic and situations where application-aware routing is unnecessary.
Trade-off
It cannot normally make rich decisions such as /payments → payment-service or route by HTTP header, cookie or user plan.
Understands application protocols such as HTTP/HTTPS and can route using host, URL path, headers, cookies, methods or other request attributes.
How it works
The balancer typically terminates the client HTTP/TLS connection, reads the request and applies application-aware routing rules before opening or reusing a connection to the selected backend.
Common use
Microservices, REST APIs, web applications, path-based routing, canary releases, authentication-aware gateways and TLS termination.
Trade-off
More processing and more configuration than simple transport-level routing. It also becomes part of the application request path, so capacity and failure handling matter.
A dedicated physical appliance designed to distribute traffic using specialized hardware and vendor software.
How it works
Traffic is sent through a dedicated appliance installed in a data center. The device may provide L4/L7 balancing, TLS offload, health checks, security controls and advanced traffic policies.
Common use
Traditional enterprise data centers, regulated environments and organizations already invested in dedicated networking appliances.
Trade-off
Higher cost, procurement time, hardware capacity limits and operational overhead compared with software or managed cloud solutions.
Load-balancing software runs on normal virtual machines, containers or bare-metal servers.
How it works
Software such as NGINX, HAProxy or Envoy receives traffic and forwards it to configured or dynamically discovered upstream services.
Common use
Self-managed infrastructure, Kubernetes ingress, service proxies, private clouds and environments requiring fine-grained control.
Trade-off
Your team owns deployment, upgrades, scaling, security, observability and high availability unless another platform manages these concerns.
A cloud provider operates the load-balancing control plane and much of the data-plane lifecycle for you.
How it works
You configure listeners, target pools, routing rules, certificates and health checks while the provider handles infrastructure provisioning and scaling within product limits.
Common use
Cloud-native applications that need simpler operations, integrated autoscaling, TLS certificates, logging and regional or global routing.
Trade-off
Provider-specific behavior, quotas, pricing and reduced low-level control. Architecture still needs multi-zone design and healthy backends.
Receives traffic from the public internet and routes it to application entry points.
How it works
A public IP or globally advertised endpoint accepts requests, applies security and routing policies, then forwards traffic to private application instances.
Common use
Public websites, mobile APIs, SaaS applications and public REST/GraphQL APIs.
Trade-off
Must be protected with TLS, DDoS controls, WAF/rate limiting where appropriate and strict backend network rules.
Balances traffic only inside a private network, VPC or internal service environment.
How it works
Internal clients resolve or connect to a private endpoint that distributes requests across private backend instances.
Common use
Service-to-service traffic, internal admin systems, private APIs, database proxies and multi-tier architectures.
Trade-off
It is not directly reachable from the public internet, but internal authentication, authorization and network segmentation are still required.
The client or client library chooses a backend instance instead of sending every request through a central proxy.
How it works
The client receives a list of healthy service instances from service discovery and selects one using an algorithm such as round robin or random.
Common use
Some service-mesh and RPC architectures where clients can safely perform service discovery and routing.
Trade-off
Every client needs routing logic, discovery integration, retries and health awareness. Versioning that logic across many clients can be difficult.
Clients send requests to a load-balancing proxy, and the proxy selects the backend.
How it works
The client knows only the stable load-balancer endpoint. The balancer maintains or discovers the backend pool and performs routing centrally.
Common use
Most public web applications, APIs, ingress layers and architectures where clients should not know backend topology.
Trade-off
The proxy layer must be highly available and sufficiently provisioned because it sits directly in the request path.
Distributes users across regions or endpoints using DNS responses, Anycast, global proxies or traffic-management policies.
How it works
A global routing layer considers region health, geography, latency, policy or capacity and directs the user toward an appropriate regional entry point.
Common use
Multi-region applications, disaster recovery, latency reduction and global active-active deployments.
Trade-off
DNS caching and TTL can delay failover. Global traffic policy must also consider data locality, session state and database consistency.
Load Balancer
Classify it using multiple independent dimensions
Protocol Layer
L4 / L7
Deployment
Hardware / Software / Managed
Traffic Scope
External / Internal / Global
Routing Ownership
Client-side / Server-side
Core Interview Topic
L4 and L7 solve the same high-level problem—selecting a backend—but they operate with different information and therefore support different routing capabilities.
L4 routing works primarily with connection metadata such as source/destination IP, TCP/UDP port and protocol. It does not need to understand the meaning of an HTTP URL.
It is useful when the service is not HTTP, when very high transport throughput is required, or when application-aware routing is unnecessary.
Example: traffic arriving on TCP port 5432 can be distributed to a pool of PostgreSQL proxies without parsing HTTP because there is no HTTP request to parse.
L7 understands application protocols such as HTTP. It can read hostnames, paths, methods, headers and cookies and then route requests based on application meaning.
It is commonly used in microservice architectures because a single domain can route /users, /orders and /payments to different backend pools.
L7 can also perform TLS termination, redirects, header manipulation, authentication integration, canary routing and other application-layer features.
| Area | Layer 4 | Layer 7 |
|---|---|---|
| Primary information | IP, port, TCP/UDP connection | Host, path, header, cookie, HTTP method |
| HTTP awareness | No | Yes |
| Path routing | No | Yes |
| TLS termination | Not an application-level feature | Commonly supported |
| Protocol flexibility | Excellent for raw TCP/UDP | Best when application protocol is understood |
| Processing | Less application parsing | More request inspection |
| Typical example | TCP service or network load balancer | Web/API ingress or application load balancer |
Client
TCP connection
L4 Load Balancer
Connection-level decision
Backend
Selected healthy target
api.example.com
HTTPS requests
L7 Load Balancer
Host / path / header rules
/users
User Service
/orders
Order Service
/payments
Payment Service
Routing Strategy
After deciding which backends are healthy and eligible, the load balancer still needs an algorithm to choose the next target. Static algorithms use little live state; dynamic algorithms react to runtime load.
S1 → S2 → S3 → S1 → ...
Requests are assigned to healthy servers sequentially. Each server gets a turn, making the algorithm simple and predictable.
Best for
Backends with roughly equal capacity and requests with similar processing cost.
Trade-off
It does not know whether one server is already busy, slow or handling expensive requests.
S1 weight 5, S2 weight 3, S3 weight 2
Each backend receives traffic in proportion to a configured weight. A stronger machine can receive more requests than a smaller machine.
Best for
Mixed-capacity servers or gradual traffic shifting between old and new deployments.
Trade-off
Static weights may not reflect real-time CPU, memory, latency or queue depth.
Choose backend with minimum active connections
The next request goes to the healthy server currently handling the fewest active connections.
Best for
Long-lived or uneven requests where active connection count is a useful approximation of load.
Trade-off
One connection can be much more expensive than another, so connection count is not a perfect measure of actual resource usage.
Connections are normalized by server capacity
Combines live connection counts with backend capacity. A powerful server is allowed to hold more concurrent connections.
Best for
Heterogeneous fleets where request duration varies and server capacity is not equal.
Trade-off
Requires reasonable capacity weights and accurate connection accounting.
Prefer low latency + low active load
The balancer uses observed response time, often together with connection count, to prefer backends that are responding quickly.
Best for
Systems where backend latency varies meaningfully during runtime.
Trade-off
Needs continuous measurements. Poor sampling or temporary latency spikes can cause traffic oscillation.
Pick one random healthy backend
A healthy backend is selected randomly. Across a large number of requests and sufficiently uniform servers, traffic tends to spread out.
Best for
Simple large pools where a small amount of short-term imbalance is acceptable.
Trade-off
Individual decisions do not consider live server load.
Pick 2 random servers → choose the less loaded
The balancer samples two healthy backends and selects the less loaded one. This offers much better balance than pure random while avoiding a full scan of every backend.
Best for
Large backend pools where checking every server on every request would be expensive.
Trade-off
Needs a meaningful load metric and is more complex than basic random routing.
hash(clientIP) % N
A hash of the client IP determines the backend. The same IP tends to map to the same server while the backend set remains stable.
Best for
Simple client affinity where approximate stickiness is enough.
Trade-off
Many users can share one NAT IP, client IPs can change, and adding/removing servers can remap many clients.
hash(path or resourceKey) → backend
Routing uses a stable request attribute such as URL, tenant ID, user ID or resource key rather than only the client address.
Best for
Cache locality, tenant affinity or workloads where related requests should reach the same shard or worker.
Trade-off
Bad key distribution can create hot spots and the routing key must be available before backend selection.
hash(key) → position on hash ring
Both servers and routing keys are mapped to a logical hash ring. When a server is added or removed, only a portion of keys are remapped instead of nearly every key.
Best for
Distributed caches, partition-aware routing and systems where routing stability matters as the backend pool changes.
Trade-off
More complex than modulo hashing and usually needs virtual nodes or good weighting for balanced distribution.
CPU + memory + queue + latency + errors
Routing decisions use one or more runtime signals such as CPU, memory, queue depth, outstanding requests, error rate or custom load scores.
Best for
Specialized workloads where simple connection count is a poor proxy for backend capacity.
Trade-off
Telemetry is delayed and noisy. A badly designed feedback loop can cause traffic to bounce between servers.
Request 1
Request 2
Request 3
Round Robin
Cycle through healthy servers
Server A
Request 1
Server B
Request 2
Server C
Request 3
Algorithm choice does not replace capacity planning
Reliability
The balancer must know which targets can safely receive traffic. Health checking is therefore part of routing correctness, not only monitoring.
The balancer periodically probes a backend endpoint or opens a connection. After a configured number of failed checks, the backend is removed from rotation.
A recovery threshold is normally used before returning the instance to service so one intermittent success does not cause flapping.
GET /health/ready
200 OK
{
"status": "UP",
"ready": true
}The balancer can observe real production traffic: timeouts, connection resets, excessive 5xx responses or unusually high latency.
Passive checks are valuable because a process can answer a synthetic health endpoint while real user requests are failing.
Load Balancer
Maintains healthy target pool
Server A
Healthy ✓ receives traffic
Server B
Unhealthy ✕ removed
Server C
Healthy ✓ receives traffic
Liveness is not the same as readiness
During a deployment or scale-down, immediately killing an instance can terminate user requests. Instead, mark the target as draining: stop new routing, keep existing requests or connections for a configured grace period, then remove the server after in-flight work completes or a deadline is reached.
Session Affinity
Sticky sessions route repeated requests from the same user or session to the same backend. They can be useful, but they also couple users to individual instances.
User A
Session cookie or affinity key
Load Balancer
Maps User A → Server 2
Server 1
Server 2
User A keeps returning here
Server 3
It can preserve in-memory session state, improve cache locality or support a legacy service that was not designed to be fully stateless.
Traffic can become uneven, a failed server can lose local session state, and deployments become harder because users are coupled to specific instances.
Preferred scalable design
Related Concept
A reverse proxy represents one or more backend servers to clients. Load balancing is one capability a reverse proxy can provide, but reverse proxies can do much more.
The primary concern is distributing traffic across multiple eligible backend instances while considering health and a routing algorithm.
The same product may also perform TLS termination, routing and observability, but traffic distribution is the defining role.
A reverse proxy terminates the client-facing connection and forwards requests to backend services. It hides backend topology and can centralize cross-cutting behavior.
Common features include TLS, compression, caching, redirects, authentication integration, request/response header changes and load balancing.
Client
Knows only public endpoint
Reverse Proxy / L7 Load Balancer
TLS + routing + balancing + policy
Service A
Service B
Service C
Examples
Security
An application-aware load balancer can terminate HTTPS, inspect the HTTP request and optionally create a new encrypted connection to the backend.
Client
HTTPS
L7 Load Balancer
Certificate + TLS termination
Backend Service
Private application network
It centralizes certificate management and allows the L7 proxy to inspect HTTP routing information. It can also reduce repeated certificate configuration across every backend.
Some security and compliance models require encryption even inside the private network. In that case the load balancer creates another TLS connection to the selected backend.
Avoid Single Point of Failure
If every request depends on one load-balancer process, that process becomes a new single point of failure. The load-balancing tier itself must therefore be redundant.
Stable Frontend
DNS / Anycast / VIP / managed global entry
Load Balancer A
Zone A
Load Balancer B
Zone B
App 1
App 2
App N
One balancer handles traffic while another is ready to take over. Failure detection triggers failover to the standby.
It is conceptually simple but standby capacity is underused and failover time matters.
Multiple balancers serve traffic simultaneously. Traffic is distributed among them and the remaining instances continue if one fails.
This improves both availability and usable capacity, but shared state and configuration must be handled carefully.
Managed load balancers
Scale
At high scale, load balancing is only one layer. The system reduces unnecessary origin traffic before distributing the remaining requests across regions, services and backend instances.
Millions of Users
Web + mobile + API traffic
CDN / Edge Cache
Serve static and cacheable content near users
WAF / Rate Limiting
Protect origin and control abusive traffic
Global Traffic Manager
Choose healthy region
Regional LB A
Zone-aware backend pool
Regional LB B
Zone-aware backend pool
App 1
App 2
App N
Remove static and cacheable traffic before it reaches application servers.
Add stateless application capacity when sustained demand grows.
Avoid repeated database and service work for popular reads.
Protect the system from abusive or accidental overload.
Move non-immediate work out of the synchronous request path.
Scale reads, partition data and protect the write path independently.
Do not say 'add a load balancer' and stop
Multi-Region
Global load balancing chooses a region or edge entry point before regional load balancing chooses an individual backend.
Global Users
India, Europe, US and other regions
Global Traffic Manager
Latency + geography + health + policy
India Region
Regional LB + app fleet
Europe Region
Regional LB + app fleet
US Region
Regional LB + app fleet
Geographic proximity, measured latency, region health, regulatory requirements, capacity, cost and explicit business policy can all affect region selection.
Routing users to another region is much easier than keeping data correct across regions. Replication lag, write ownership, data residency and conflict handling often dominate the multi-region design.
DNS failover is not instantaneous
Long-Lived Connections
WebSocket connections remain open for a long time, so the balancing problem is different from short independent HTTP requests.
Client
HTTP upgrade request
WebSocket-Aware Load Balancer
Select backend when connection opens
Realtime Server
Connection remains attached
A server with many long-lived WebSockets can remain busy even when it is receiving few new HTTP requests. Least Connections or capacity-aware routing is often more meaningful than plain Round Robin.
If User A is connected to Server 1 and User B to Server 4, realtime events may need a shared pub/sub or messaging layer so one application server can reach users connected elsewhere.
Deployment behavior
Complete HLD Example
A ticket-booking platform has two very different traffic patterns: extremely heavy read traffic for discovery and seat maps, and correctness-sensitive write traffic when users lock and purchase seats.
Important note
Imagine tickets for a highly anticipated movie open at 10:00 AM. Thousands or millions of users may refresh the same city, movie and showtime pages within seconds. Most of those requests are reads: movie metadata, theatre list, show timings and seat maps. A smaller percentage reaches the critical booking path, where the same seat can be requested by many users at nearly the same time.
Discovery endpoints receive huge fan-out. CDN, application caching and read replicas can reduce origin/database load.
Multiple users can select the same seat. Correctness requires atomic locking or reservation logic with expiration.
Payment can succeed, fail, time out or return late. The booking workflow must be idempotent and recoverable.
Users
Search, browse, seat map, booking
CDN / Edge
Static assets + cacheable public content
WAF + Rate Limiter
Protect high-demand endpoints
L7 Load Balancer / API Entry
Health-aware routing by path/service
Search Service
Movies + theatres
Show Service
Timings + seat map
Booking Service
Locks + reservation state
Payment Service
Payment orchestration
Cache / Redis
Hot reads + short-lived locks
Booking Database
Authoritative seat + booking state
Event / Queue Layer
Notifications + async processing
What the load balancer solves here
The request reaches the L7 load balancer and is routed to a healthy Show Service instance. Frequently requested show metadata can be served from cache.
The service reads current seat state from an appropriate cache/database model. Available, locked and sold seats must be distinguishable.
The booking request is routed to any healthy Booking Service instance. The application should not rely on that one server keeping the lock only in local memory.
The system attempts to transition A10 from AVAILABLE to LOCKED for this user/order, normally with a short TTL. Only one competing request should succeed.
Persist a booking/order identifier and the locked seats before redirecting the user into payment. The request should carry an idempotency key so retries do not create duplicate bookings.
The payment service creates or tracks the payment attempt. The seat remains locked only for the booking window rather than forever.
On verified success, the authoritative seat state is changed to BOOKED/SOLD and the booking becomes CONFIRMED. Database uniqueness or conditional updates provide another correctness boundary.
If payment fails or the reservation TTL expires, the seat returns to AVAILABLE. Late payment callbacks need reconciliation so a user is not charged for an unavailable seat without recovery handling.
Confirmation email/SMS, analytics, invoice generation and other non-critical tasks can be processed through an event or queue layer instead of blocking the booking response.
User A
Requests Seat A10
User B
Requests Seat A10 at same time
Load Balancer
Requests can go to different Booking Service instances
Booking Instance 1
Handles User A
Booking Instance 2
Handles User B
Seat Lock / Inventory Store
A10 can be locked by only one booking
User A
Lock acquired → continue to payment
User B
Lock rejected → choose another seat
Why sticky sessions do not solve double booking
Posters, static JavaScript, CSS and other cacheable resources should not consume booking-service capacity.
Search/show services can run many stateless replicas behind load balancers and use cache/read replicas where appropriate.
If demand exceeds safe booking capacity, rate limiting, waiting rooms or queues are safer than allowing every request to hammer the database.
Seat locks, idempotency and database constraints remain authoritative even when thousands of application instances are serving traffic.
Duplicate callbacks or client retries should not confirm the same booking twice or charge twice.
Email, SMS and analytics should not make the synchronous seat-confirmation transaction slower.
AVAILABLE
|
| atomic lock(userId, bookingId, expiresAt)
v
LOCKED
| | lock expires / payment fails
| -----------------------------> AVAILABLE
|
| verified payment success
v
BOOKEDProduction Reliability
A production load-balancing design is incomplete unless you explain what happens when backends become unhealthy, retries increase load, deployments drain connections or the balancing layer itself fails.
Health checks should mark the instance unhealthy and remove it from new routing. Existing connections may fail and need safe retry behavior at the appropriate layer.
A shallow /health endpoint may still return 200. Track latency, timeouts, queue depth and passive error signals so a slow instance can be drained or receive less traffic.
Run multiple balancer instances across failure domains. Use active-active or active-passive failover and a stable frontend such as managed anycast, VIP or DNS/global routing.
Use CDN and caching to absorb read traffic, autoscale stateless services, enforce admission control/rate limits and queue work that does not need synchronous execution.
Retries multiply traffic during an outage. Use bounded retries, exponential backoff, jitter, deadlines and retry budgets; do not blindly retry non-idempotent operations.
Mark an instance as draining, stop sending new requests, allow in-flight requests or long-lived connections to finish, then terminate the instance after a safe deadline.
Client
Request with deadline
Load Balancer
Connect + request timeout
Slow Backend
May time out without crashing
Bounded Retry + Backoff + Jitter
Avoid synchronized retry storms
Be careful retrying writes
Interview Preparation
After studying this chapter, you should be able to answer these questions with theory, trade-offs and a diagram.
What is a load balancer and why do we need one?
What problems are solved by horizontal scaling plus load balancing?
What are the different types of load balancers?
Hardware vs software vs managed cloud load balancer?
External vs internal load balancer?
Client-side vs server-side load balancing?
What is the difference between L4 and L7 load balancing?
When would you choose L4 over L7?
Round Robin vs Weighted Round Robin?
Round Robin vs Least Connections?
What is Weighted Least Connections?
What is Least Response Time?
What is Power of Two Choices?
IP Hash vs consistent hashing?
Why is consistent hashing useful when nodes change?
What are active and passive health checks?
Liveness vs readiness?
What is connection draining?
What are sticky sessions and why can they be harmful?
Load balancer vs reverse proxy?
What is TLS termination and backend re-encryption?
How do you prevent the load balancer from becoming an SPOF?
Active-active vs active-passive load balancing?
How would you handle millions of requests per second?
Where should CDN, WAF, rate limiter and load balancer sit?
How does global load balancing work?
Why can DNS TTL delay regional failover?
How do you load balance WebSockets?
How do retries create retry storms?
How would a load balancer fit into BookMyShow system design?
Why does load balancing not solve double booking?
How would you lock a seat while a payment is in progress?
What happens if payment succeeds after a seat-lock timeout?
Quick Revision
Short answers for the most important load-balancing concepts.
A load balancer is an entry layer that distributes incoming traffic across multiple healthy backend instances. It improves scalability, availability and failure isolation by preventing every request from depending on one server.
Common classifications include Layer 4 and Layer 7, hardware and software, managed cloud, external and internal, client-side and server-side, plus DNS or global traffic load balancing.
L4 makes routing decisions mainly from transport-level information such as IP addresses and TCP or UDP ports. L7 understands application protocols such as HTTP and can route by host, path, header, cookie or method.
There is no universal best algorithm. Round Robin is simple for uniform backends, Least Connections helps with uneven request duration, weighted algorithms handle different capacities, and hashing is useful when routing affinity or cache locality matters.
Load balancing distributes browsing, show search, seat-map and booking requests across healthy service instances. It does not itself prevent double booking; seat locking, idempotency, database constraints and transactional booking logic are needed for concurrency correctness.
Yes. Production systems use multiple load-balancer instances or managed highly available load-balancing services, often across zones and sometimes across regions.
Move from requirements to APIs, services, databases, caching, load balancing, queues, concurrency, scalability and failure handling in complete interview-style systems.
Continue Learning