Distributed Services and System Design

What must a response promise?

A system design is a set of promises under load and failure. A payment record, a video listing and a cached image need different promises. Start with what a completed write and a successful read mean. Then decide which component owns each promise.

Scalability is not a list of distributed components. Replication copies state. Sharding divides ownership. Caching reuses a result. Load balancing chooses a worker. Queues separate admission from completion. Each mechanism solves one problem while introducing another.

The following examples are conceptual, not a description of a provider’s production architecture. The video-platform case embeds the original design exercise as a worked system, rather than a separate interview page.

Describe load before choosing components

A read-heavy news site with slow-changing articles benefits from caching. A write-heavy social feed benefits from durable ingestion and asynchronous materialization. Neither label selects a database by itself. Estimate bytes and operations, then inspect the actual query and write paths.

Add resources where work is limited

Vertical scaling adds CPU, memory, storage or bandwidth to one node. It can be simpler but has hardware and operational limits. Horizontal scaling adds nodes and requires distributing work or state. It does not create infinite capacity or automatic redundancy.

A practical evolution separates the database from the web tier, adds application replicas and routing, then addresses data replication, caches, sessions and geography. Queues and sharding enter when their specific constraints appear. This is a sequence of possible changes, not a mandatory nine-step blueprint.

Stateless application workers keep durable session state outside the worker, allowing another worker to resume a request. The external session store becomes a dependency that needs its own capacity, consistency and failover plan. Stateful workers can also scale, but need explicit placement, migration and recovery.

Horizontal cost is not universally linear, and vertical cost is not universally exponential. Coordination, replication, network and idle headroom can dominate. Memory-mapped files still cause disk I/O on page faults. Microservices do not isolate failures without timeouts, backpressure and dependency boundaries.

Route work without hiding state

Layer 4 routing uses transport/network information such as addresses and ports. Layer 7 routing can use HTTP paths, headers and cookies. TLS termination decrypts at the proxy; decide whether the backend hop also needs encryption. Public backend-address hiding is not a complete security boundary.

A reverse proxy can integrate a web application firewall, but health, security and workload routing are separate functions. Hardware appliances, Nginx, HAProxy and managed services are implementation choices. Redundant load balancers need their own state, routing and failure-domain design.

Sticky sessions can improve cache locality but reduce routing freedom. Serving Models shows the same tension for large model histories.

A timeout is suspicion, not proof

A missed heartbeat indicates that a node might be dead, slow or unreachable. Promoting another writer on that evidence alone can create two active leaders. Failure detection must connect to an ownership protocol.

Use leases, epochs or fencing tokens where appropriate. A storage service can reject commands from an obsolete leader even if that leader resumes. Drain in-flight work during maintenance, and define which retries are safe. Health checks need to test the relevant service path rather than only a process being alive.

Geographic redundancy helps only when the recovery path, data and dependencies survive the affected region. Test controlled failure scenarios with safeguards; killing production nodes indiscriminately is not an availability strategy.

Moving a service IP address to a standby is one failover option. Address takeover alone does not transfer application state or revoke the old writer. Routing, state recovery and fencing must agree on the new owner.

Copies, leadership and failover

An acknowledged asynchronous write can be lost if its only durable holder fails. Synchronous replication changes the acknowledgment boundary, but a quorum of copies alone does not specify leader safety or reads. Define durability separately from freshness.

Active–active traffic does not necessarily mean every database node can write. Multi-primary does not inherently guarantee faster writes or eventual-only semantics. The protocol and invariant decide. Hot standby is ready sooner than cold standby, but has ongoing resource cost.

Name the guarantee observed by a client

The word strong is ambiguous. Name the required contract rather than assuming any read at any node is current. Leader-based reads, read barriers, versions and recovery are part of the mechanism; synchronous copies and locking are not universal recipes.

VoIP and games may prefer timely state over waiting for an old update. That application choice does not make every associated database eventually consistent. Different operations within one service can require different guarantees.

A partition separates knowledge

Suppose one network partition isolates C from A/B. A/B acknowledge a new value. C cannot determine that value from its local history. A read at C must either wait/refuse under the freshness contract or return a possibly stale answer.

CAP concerns this inability to guarantee both linearizable register behavior and completion of every operation at nonfailed nodes when communication can be lost indefinitely. An error response is not a successful operation that satisfies the register specification. Partition tolerance is a failure assumption, not a feature that can be disabled to make the network reliable.

Do not read “choose two” as three symmetric product choices. A majority can remain available while a minority blocks; CAP availability still fails because some eligible requests cannot complete. The consistency of a register is also different from ACID transaction atomicity.

Latency also matters when the network works

PACELC adds a useful design question: under a partition, availability versus consistency; else, latency versus consistency. Synchronous cross-region coordination can cost latency even when every link works.

This classification is a way to discuss choices, not a complete protocol or fixed database label. Cache reads, account updates and content publication can choose different paths. Record which operation sacrifices which guarantee and under what failure condition.

Intersection is necessary geometry

If the sets were disjoint, their union would need $R+W$ distinct members, exceeding N. This proves intersection, not the behavior of a database. Concurrent writes, incomplete writes, version ordering, reading the latest committed state and membership changes still need rules.

Two write quorums intersect when $2W>N$. That prevents disjoint acknowledgment sets in the fixed universe, but does not itself serialize conflicting updates. Sloppy quorum can acknowledge temporary substitute holders, so ordinary fixed-set overlap reasoning may no longer apply.

Convergence requires conflict and repair rules

Asynchronous copies require repair. Hinted handoff records a temporary holder and later returns data to its intended replica. Anti-entropy compares replicas; Merkle-tree hashes can narrow which ranges differ.

Version vectors can identify causal ancestry and concurrent versions; they do not choose the desired application outcome automatically. Last-write-wins needs a declared ordering, often timestamps, and can discard concurrent updates. Clock skew can make a timestamp order differ from real-time completion.

An application can merge suitable operations or reject conflicts. State the invariant first: losing an increment and replacing a profile image are different outcomes. Eventual convergence assumes updates eventually propagate and the merge/repair rules agree.

Calculate availability with declared assumptions

Two 99% components in series give 98.01%. Two independent interchangeable 99% replicas give 99.99% under ideal failover. A common 99% gateway lowers the latter to 98.9901%. Shared power, software or configuration failures invalidate naive independence.

An availability SLO is a target; a service-level agreement (SLA) is a contractual commitment. Neither guarantees uninterrupted operation. A request-success measure and a time-uptime measure can disagree. Failover detection, recovery and capacity must be included in the measurement.

Reuse reads without inventing consistency

A cache checks for a reusable value before performing expensive work. Read-through delegates a miss to a loader. Cache-aside lets the application read the origin and populate the cache. Write-through routes writes through a layer that updates durable storage and its cache according to a declared acknowledgment rule.

Time to live (TTL) bounds reuse time; it does not keep two copies synchronously consistent. Two simultaneous writes to cache and database are not automatically atomic. A crash between them or concurrent update can leave stale data. Invalidation, versioned keys and ownership rules must handle that gap.

Cache low-churn articles or product data when the tolerated freshness permits it. Protect against miss storms and cold starts. Popular-key read caching can help a shard hotspot, but a write hotspot needs another solution. State whether stale content can be served when the origin fails.

Queues acknowledge a stage of work

A durable queue smooths bursts and decouples acceptance from background work. It does not remove the work. If arrival exceeds sustainable service, backlog grows until the system applies backpressure, rejects work or exhausts storage.

Tell the client what was acknowledged: received, durably queued, committed or published. A user can see an optimistic local tweet before global publication, but that view is not proof of a durable database write.

Workers can retry after a lost acknowledgment. Use idempotency keys, deduplication or transactional outbox/inbox boundaries where appropriate. Do not infer exactly-once effects from a queue delivering once in an ideal example. Track oldest message age and retry rate, not only queue depth.

Choose storage APIs and query paths

Block storage exposes addressable blocks. A filesystem or database can build structured storage above it. Object storage exposes objects by key and metadata, making large immutable media and backups convenient. Object storage is not automatically a database for relational query patterns.

Low latency, input/output operations per second (IOPS) and cost depend on the actual service tier, request size and access pattern. “Block for small files, object for large files” is a starting heuristic, not a law. Store media objects separately from queryable metadata when that separation fits the workload.

A log-structured merge-tree (LSM-tree) buffers writes and flushes sorted runs, then compacts runs. Writes incur compaction/write amplification; reads use indexes, filters and caches to avoid scanning every file. More runs do not imply reads inevitably become slower forever. Compaction also affects tail latency.

Speed reads by maintaining extra structure

An index maintains another access path so a query can avoid scanning every row. It consumes space and adds write maintenance. Composite-column order must match predicates and ordering; a multicolumn index does not accelerate every multicolumn query.

Inspect the query plan, selectivity, rows visited and storage/cache behavior. Fixing a missing index or expensive join can be preferable to sharding. Denormalization saves reads but introduces update consistency work. Measure the complete request rather than only one indexed lookup.

Partition data by its access pattern

Sharding divides a dataset across owners. Replication copies each partition for redundancy. Multiple shards do not create backup copies: losing one unreplicated shard loses its subset.

Choose keys with sufficient cardinality and useful locality. Hash distribution helps spread keys; range distribution preserves range scans but can concentrate recent writes. A tenant key keeps tenant queries local but can create a large-tenant hotspot.

Cross-shard joins, transactions and foreign-key enforcement depend on the database’s protocol. They are not universally impossible, but coordination can be costly. Denormalization can reduce cross-shard work at the cost of maintaining multiple representations.

Resharding must copy data while handling new writes, update routing and decide when the old owner can stop. Capacity balances need bytes, request demand and migration traffic, not only key counts. Vitess is an example of MySQL sharding infrastructure, not a substitute for key design.

Limit remapping, not hot-key demand

Consistent hashing maps a key to a successor on a ring. Adding a position changes a nearby interval rather than every key’s modulo destination. Under ideal uniform balance, adding a node to N nodes moves about $\frac{1}{N+1}$ of keys. That is a fraction under assumptions, not a running-time bound. Copying moved records still depends on their number and size.

Virtual nodes place several positions for one physical owner and improve balance statistically. They do not split one indivisible popular key. Read caches, a different data model or explicit hot-key partitioning can address that demand. High cardinality alone does not guarantee equal request load.

Discover endpoints through cached delegation

stub/cache → recursive resolver
            → root referral → TLD referral
            → authoritative answer
answer/TTL returns through resolver to client

DNS is a distributed naming database, not a content-transfer path. The resolver can reuse cached answers or delegations, so a lookup does not always visit every level. Authoritative servers answer for their zones; recursive resolvers perform the lookup for clients.

TTL is measured in seconds and describes cached record lifetime. Updating an authoritative record does not instantly purge existing recursive caches. Lowering a new TTL does not retroactively shorten previously cached lifetime. DNSSEC authenticates signed DNS data; it does not encrypt queries or prevent every endpoint attack.

GeoDNS and anycast are different

GeoDNS changes answers using location or policy. DNS can also choose weighted, latency-oriented or failover answers. Resolver caching means answer weights do not translate into exact per-request traffic shares, and failover is not instantaneous.

Anycast advertises one service address from multiple sites. Network routing selects a site according to topology and policy, not guaranteed geographic distance or minimum RTT. DNS itself can use anycast; that does not make GeoDNS and anycast synonyms.

DDoS resilience needs capacity, isolation and recovery rather than one routing label. Long-lived sessions can be sensitive to anycast route changes. Health-based DNS and load-balancer routing operate at different timescales.

Deliver from an edge or fetch the origin

client → selected edge → cache hit → response
                      └→ miss → origin → cache/response

A Content Delivery Network (CDN) places cached content near users. A pull cache fetches on demand; push/prewarming prepares selected assets before demand. Popular immutable objects benefit from long-lived versioned keys. Low-reuse objects can consume space and transfer without many hits.

TTL, invalidation and versions determine freshness. An origin outage can still be served from a valid or explicitly permitted stale cache; a cold miss cannot magically recover unavailable content. Falling back directly to origin during an edge outage can overload the origin.

Edge computation can customize some responses but does not remove state or policy requirements. Cache keys must include relevant variant and authorization boundaries. Record request hit ratio and byte hit ratio separately: a tiny metadata hit and a large video hit save different traffic.

Compare origin egress, cache-fill traffic and CDN delivery charges under the same workload. Reduced origin traffic does not guarantee a lower total bill. Rarely reused objects can incur transfer costs without enough saved origin work.

A video platform as a worked workload

Scope the case to upload and watch. Ads, recommendation, abuse prevention and analytics are separate services. The requirements are durable uploads, accessible playback, low startup latency and scalable popular content.

The original assumes only about 5% of videos become widely popular. Use that as an assigned skew scenario, not a verified popularity distribution. Listing freshness can be eventual while upload durability and authorization require stricter guarantees. Do not choose one consistency label for the whole site.

Separate upload, processing and publication

upload → durable raw object + metadata status
       → durable processing job → encode renditions
       → encoded objects + validated manifest
       → publish metadata reference / invalidate listing

Store raw and encoded objects separately from title, description, owner and processing status. A document database is one option for metadata, not a requirement. Managed object replication must be configured and verified; provider branding alone does not prove recovery or durability.

Resumable/multipart upload avoids restarting huge files after every network interruption. Acknowledge durable acceptance separately from ready-to-watch publication. Use checksums and idempotent job identity. Atomically publishing a manifest reference prevents readers from discovering incomplete renditions.

Encoding can begin after durable completion or operate incrementally when the codec/container and pipeline support it. The original “cannot encode during upload” is not universal. Bound retry, malformed-file and resource costs before declaring media ready.

Carry units through the worker estimate

Little’s-law style service accounting uses jobs/s × seconds/job. Do not multiply a rate into another rate accidentally. At70% assigned utilization, this example needs49,604 workers before additional failure/burst constraints.

A worker may process several rendition jobs per upload, have heterogeneous performance or share resources. Measure actual service demand and the arrival distribution. A100%-utilization average leaves no spare capacity for variable arrivals; an average-count estimate is not a queue-latency guarantee.

Start playback before transferring the whole object

Fetch authorized metadata/manifest through the application path. Serve encoded media chunks from the CDN, loading enough initial content to start playback before the whole object arrives. Metadata can use a least-recently-used (LRU) cache; media byte reuse belongs to the delivery cache.

Multiple bitrate renditions let a client adapt to available bandwidth and buffer state. Segment duration trades startup/seek overhead against buffering flexibility. “No buffering with correct internet” is not a guarantee: network variation, origin misses and encoding availability still matter.

HTTP media delivery is not synonymous with TCP. HTTP/3 uses QUIC over UDP and can serve prerecorded media. Protocol, retransmission, congestion control and live versus on-demand buffering are separate choices.

Upload rate limits protect ingestion and encoding. A recommendation service can use viewer metadata but need not sit on the critical path for every chunk.

Observe latency, lag and failure boundaries

Collect metrics, traces and logs with explicit boundaries. Prometheus is primarily a metrics system; Grafana visualizes data; OpenTelemetry instruments/export traces, metrics and logs. Do not describe a metrics collector as the entire logging system.

Observe tail latency, queue age, database plans, replication lag, cache-hit bytes, per-shard skew, error classes and failover outcomes. Sample routine events where safe, retain critical audit evidence and alert on actionable service symptoms. Sampling can hide rare failures if its policy is careless.

The end-to-end trace should identify what the user waits for. Profiling applies the same dependency reasoning to GPU work; Compute, Memory, and Performance separates latency, throughput and resource bounds.

Change one bottleneck and remeasure

For heavy reads, examine repeated work and cache freshness. For heavy writes, examine durable admission, indexes, compaction and backlog. For one slow query, inspect the plan before splitting the database. For a popular key, measure demand before adding uniformly distributed shards.

For failures, state which component can decide ownership and which copy survives. For geographic latency, separate discovery, content caching and coordination. For large media, separate payload storage, query metadata and publication.

Change one limiting path, then remeasure the complete contract. Replicas, CDNs and queues can move the bottleneck rather than remove it. A correct design explains what happens during overload, partitions, retries and recovery, not only the successful request path.

Sources and connected topics

• Original eight-concept design reference ↗; Original video-platform design reference ↗.
• Gilbert and Lynch: Perspectives on CAP ↗; Dynamo ↗.
• RFC1034 ↗; RFC4786 ↗. Targeted mechanism checks, not a full audit of every implementation.

The animation shows an original single-write partition scenario. Quorum controls prove only replica-set geometry. Worker budgets use assigned service times, not an encoding benchmark.