Networking

Reverse Proxy / Web Server

Accept, route, and serve requests: a few event-loop workers multiplexing tens of thousands of connections, pooled upstream connections, graceful drain on reload, and backpressure so the loop never outruns its memory.

~35 min · intermediate

Problem & Requirements

Sit in front of a set of backend servers, accept client connections, parse each HTTP request, pick an upstream, forward the request, and stream the response back. The proxy holds two sockets per in-flight request — one to the client, one to the upstream — and its job is to shuttle bytes between them while doing routing, header rewriting, and TLS in the middle. Clients are numerous and often slow; upstreams are few and should be kept busy. A single box is expected to carry tens of thousands of simultaneous connections.

That connection count is what rules out the obvious design. A thread per connection costs a stack and a scheduler slot each, and at ten thousand idle-but-open connections the machine drowns in threads doing nothing — the C10k problem. So concurrency is the event loop's job, not the thread scheduler's: a handful of workers, one per core, each looping over nonblocking sockets and reacting to readiness, carry all those connections on a few threads. Connection pooling keeps that cheap on the upstream side by reusing established TCP+TLS connections instead of paying setup per request. Graceful shutdown lets the thing be redeployed without dropping the requests already in flight. Every decision below serves the rule that a worker must never block.

Functional

serve() accepts client connections, parses requests, routes by host/path to an upstream, and proxies the response back.
Reuse upstream connections across requests rather than dialing a fresh one each time.
Reload configuration and shut down without dropping in-flight requests.

Non-functional (back-of-envelope, single node)

Quantity	Target	What it forces
Concurrent connections	~50k–100k/node	Event-loop multiplexing, never a thread per connection
Worker count	= CPU cores	Shared listen socket; near-shareless workers, each its own loop
Added latency (p50)	sub-millisecond	Nonblocking I/O and a warm upstream pool, no per-request connect
Upstream connection reuse	high (keepalive)	Pooling, or TCP+TLS handshake cost dominates every request
Memory per connection	a few KB of buffers	Bounded buffers and backpressure, or slow peers balloon the heap
Reload / deploy	zero dropped requests	Drain in-flight work before exit; two worker generations briefly coexist
Throughput	tens–hundreds k rps/node	The hot path stays on the loop; blocking work moves off it

The treacherous row is memory per connection. Request rate is the easy axis — the loop dispatches ready sockets and the pool keeps upstreams warm. The killer is the slow peer: an event loop will happily accept far more work than it can drain, and a slowloris client trickling one byte at a time, or a stalled upstream, pins file descriptors and grows buffers while requests-per-second look fine. Bounding that is build step six, and it is the line between a proxy and a memory exhaustion incident.

Design

Six components, each tied to the principle it applies:

Listener and accept loop — a nonblocking listen socket registered with the event loop; new connections are accepted without blocking and handed to the loop. This is the entry point to the reactor.
Event loop (reactor) — one thread per worker runs epoll/kqueue, waits for socket readiness, and dispatches each ready fd to its handler. This is the event loop / reactor, the heart of the whole design.
Request handler and router — a small state machine per connection that reads the request, picks an upstream by host/path, and pumps bytes between client and upstream as each side becomes ready.
Upstream connection pool — per-upstream sets of established connections checked out per request and returned on completion, with stale ones discarded. This is connection pooling.
Worker supervisor — a master that spawns one worker per core over a shared listen socket and forwards signals for reload and drain, so workers stay independent and shareless. Concurrency across cores reuses the reactor per worker.
Drain and backpressure controller — stops accepting on shutdown and lets in-flight work finish (graceful shutdown), and caps connections, buffers, and timeouts so a slow peer can't grow the loop without bound (backpressure).

The two named systems are both open source and worth reading as the reference, and they split on process model. NGINX runs a master process plus one single-threaded worker process per core; each worker owns an event loop, the workers share the listen socket and accept independently, and a config reload spawns a new generation of workers while the old ones drain and exit — the graceful-shutdown machinery of step four made operational. Envoy runs a single process with a main thread plus one worker thread per core, each thread its own event loop over libevent; it leans hard on a thread-local, mostly-shareless worker design with the main thread coordinating, pushes configuration dynamically over xDS rather than reloading, and swaps binaries with a hot restart that passes listen sockets between processes. Keep the two apart: NGINX's distinctive work is the worker-process reload model, Envoy's is dynamic config and the thread-local architecture, and both implement the same per-worker reactor and per-worker connection pools the build constructs.

Build it

Start with the literal design: accept a connection, spawn a thread, read the request, dial the upstream, forward, copy the response back, close. It is correct and easy to read, and it fixes the contract — one request, one upstream, bytes copied through. Its failure is structural: every connection costs a thread and a stack, a blocking recv parks that thread doing nothing, and at ten thousand mostly-idle connections the box is buried in threads. The next step removes the thread-per-connection model with an event loop.

Replace one-thread-per-connection with one thread that watches all of them. Set sockets nonblocking, register them with the OS readiness primitive (epoll on Linux, kqueue on BSD), and loop: ask which fds are ready, dispatch each to its handler, never block on any single one. This is the reactor / event loop, and it is what lets a single thread carry tens of thousands of connections. The failure it exposes lives on the upstream side — each request still dials a brand-new upstream connection, paying a TCP handshake (and a TLS handshake, in reality) every time, which the next step amortizes.

A handshake per request is wasteful when the proxy talks to the same few upstreams over and over, and TLS makes it expensive enough to dominate latency. Keep a pool of established connections per upstream: check one out for a request, return it on completion, and discard any that died while idle. This is connection pooling — the keepalive behavior NGINX and Envoy both implement toward upstreams — and it turns the common case into zero-handshake reuse. The remaining failure is operational: a deploy kills the process outright, dropping every in-flight request and severing pooled connections mid-response.

Killing the process on reload or deploy severs requests that are halfway through, which clients see as resets. Drain instead: on the shutdown signal, stop accepting new connections and close the listener so the port is free for the replacement, let the event loop run until in-flight requests finish or a deadline passes, then close idle pooled connections and exit. This is graceful shutdown, and it is exactly NGINX's SIGQUIT worker drain and the spirit of Envoy's hot restart. The failure that remains is that one event loop is one core — the rest of the machine sits idle.

A single-threaded loop leaves every other core idle. Spawn one worker per core, all sharing one listen socket via SO_REUSEPORT so the kernel spreads incoming connections across them; each worker runs its own reactor and its own pools, with a master process supervising and relaying the reload and drain signals from step four. Workers stay independent and shareless, the model behind NGINX's worker processes and Envoy's worker threads, which avoids cross-thread locking on the hot path. The cost surfaces under load: nothing yet bounds how much work a worker accepts, so a slow client or stalled upstream lets connections, buffers, and file descriptors grow without limit.

An event loop accepts work eagerly and has no instinct for self-preservation, so a flood of connections or a single slow peer grows its memory and fd table until the worker dies. Put limits at the door: cap concurrent connections and shed past the cap, give each connection a header-read deadline to kill slowloris, bound read and write buffers, reap idle connections on a timeout, and fail fast with 503 when the upstream pool is exhausted rather than queueing without limit. This is backpressure, the defense the requirements table flagged as the real difficulty, and it keeps memory per connection bounded under adversarial load.

import socket, threading

class Proxy:
    def __init__(self, upstream):
        self.upstream = upstream                  # (host, port)

    def serve(self, port):
        lsock = socket.socket()
        lsock.bind(("", port)); lsock.listen()
        while True:
            conn, _ = lsock.accept()
            threading.Thread(target=self._handle, args=(conn,)).start()

    def _handle(self, conn):
        req = conn.recv(65536)                     # blocks this whole thread
        up = socket.create_connection(self.upstream)   # fresh conn per request
        up.sendall(req)
        conn.sendall(up.recv(65536))               # blocks again
        up.close(); conn.close()

Tradeoffs

Decision	What it buys	What it costs
Event loop / reactor	Tens of thousands of connections on a few threads	Any blocking call — DNS, disk, CPU-heavy work — stalls every connection on that loop
Connection pooling	Amortized TCP+TLS, lower latency and upstream load	Stale and half-open connections to detect; serial HTTP/1.1 reuse can head-of-line block
Graceful shutdown	Zero-drop reloads and deploys	Drain time and the complexity of two worker generations coexisting
Worker per core (shared listener)	Scales across cores with no hot-path locking	Per-worker pools and caches duplicate state; accept load can land unevenly
Backpressure and limits	Bounded memory and fds; resistance to slow-peer attacks	Rejects or delays legitimate load near the caps; the caps are a tuning judgment

Scaling it up

The toy moves bytes; a real proxy terminates TLS and speaks modern HTTP, which reshapes the connection model. Terminating TLS adds handshake CPU and session-resumption caches, and ALPN negotiation means a single accepted connection may be HTTP/1.1, HTTP/2, or HTTP/3. HTTP/2 multiplexes many concurrent streams over one connection, so the unit of work is a stream rather than a socket and flow control moves up into per-stream windows. HTTP/3 runs over QUIC on UDP and abandons the TCP socket model entirely, which both NGINX and Envoy implement as a separate datapath rather than a tweak to the one above.

Routing in production is to a cluster, not a single upstream. That brings health checks to drop sick backends, outlier detection to eject ones that start failing, retries with budgets so a retry storm doesn't amplify an incident, circuit breaking to stop sending to an overwhelmed cluster, and a balancing policy (round-robin, least-request, consistent hashing) over the live members. Envoy models this explicitly as clusters configured over xDS; NGINX expresses it through upstream blocks and its load-balancing and health-check modules. The connection pool from step three becomes per-cluster-member and interacts with all of the above.

Configuration cannot require a full restart at scale, so reload generalizes the graceful-shutdown step. NGINX reloads by spawning a new worker generation against the new config while old workers finish their connections and exit, and binary upgrades pass the listen socket to a freshly-execed master so no connection is refused during the swap. Envoy's hot restart does the same socket hand-off between processes over a domain socket, and its xDS control plane streams listener, cluster, route, and endpoint updates into running workers with no restart at all — the drain logic in step four is the local half of that machinery.

Observability and buffering are where slow peers are actually tamed. Per-request access logs, metrics, and distributed tracing all add work to the hot path that must stay cheap and nonblocking. The choice between buffering a full request/response and streaming it changes the memory profile and the slow-client exposure directly, and HTTP/2 flow-control windows are the protocol-level form of the backpressure step — a way to tell a fast sender to wait without dropping the connection. Getting buffering and timeouts wrong is how a proxy that passes load tests falls over to a slowloris in production.

The cardinal sin underneath all of it is blocking a worker, so any unavoidable blocking work is pushed off the loop. NGINX offloads blocking disk reads to a thread pool precisely so a slow filesystem can't stall an event loop; DNS resolution, large TLS handshakes, and CPU-bound transforms get similar treatment. From here the concrete follow-ons are an L7 load-balancer prototype that builds health checks, outlier detection, and retry budgets over a cluster; a TLS-termination prototype covering handshakes, session resumption, and ALPN; and an HTTP/2 multiplexing prototype that turns the per-socket state machine here into a per-stream one with flow control. Each extends this reactor-plus-pool core without rebuilding it.

References

Kegel, The C10K problem — why thread-per-connection fails and event-driven I/O is the answer: https://www.kegel.com/c10k.html
Schmidt, Reactor: An Object Behavioral Pattern for Demultiplexing and Dispatching Handles for Synchronous Events: https://www.dre.vanderbilt.edu/~schmidt/PDF/reactor-siemens.pdf
epoll(7) — the Linux readiness primitive the event loop is built on: https://man7.org/linux/man-pages/man7/epoll.7.html
Inside NGINX: How We Designed for Performance & Scale — master/worker processes, the event loop, shared listener: https://blog.nginx.org/blog/inside-nginx-how-we-designed-for-performance-scale
nginx chapter, The Architecture of Open Source Applications — worker model and reload/upgrade: https://aosabook.org/en/v2/nginx.html
Envoy Threading model and Life of a Request — main thread plus worker threads, thread-local design: https://blog.envoyproxy.io/envoy-threading-model-a8d44b922310 · https://www.envoyproxy.io/docs/envoy/latest/intro/life_of_a_request
Envoy hot restart and xDS dynamic configuration: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/hot_restart
Foundations referenced inline: event loop / reactor · connection pooling · graceful shutdown · backpressure