Chrysolites Tech Academy

Designing and Shipping Web Apps · Course book

Chapter 1: A Method for Designing Systems

Pairs with Lesson 1: System Design: Thinking Before Building

System design is the craft of deciding how the parts of a large piece of software fit together before any of it is built. There is rarely one correct answer. What marks out an experienced engineer is a method: pin down what is needed, estimate the load, start with the simplest design that works, find where it will break, and explain each trade-off. This chapter walks through that method with numbers you can check.

Start with requirements

A request such as "design a photo sharing app" is deliberately vague, and the first job is to make it precise. Jumping straight to databases and servers is the most common mistake, because you cannot judge a design without knowing what it is for. Requirements come in two kinds.

Functional requirements say what the system does. For the photo app these might be: users can upload a photo, follow other users and see a feed of recent photos from people they follow. Listing them also settles what is out of scope. If comments and private messages are not on the list, you do not design them.

Non-functional requirements say how well the system must do it. How many people use it? How quickly must the feed appear? Is it acceptable for a new photo to take a few seconds to show up for followers? How much downtime can be tolerated? These answers shape the design far more than the feature list does. A feed for a thousand users and a feed for a hundred million are different problems that happen to share a description.

Ask before you draw. Good clarifying questions include: who are the users, how many are there, what is read most often, what is written most often, and what must never be lost?

Estimate the load

Rough arithmetic, often called a back-of-the-envelope estimate, turns "a lot of users" into numbers you can design for. Precision is not the aim; knowing whether you face hundreds or millions of requests is. Assume the photo app has 1,000,000 people using it each day.

Traffic. Suppose each active user causes 20 requests a day by opening the app and scrolling. That is 1,000,000 × 20 = 20,000,000 requests a day. A day has 24 × 60 × 60 = 86,400 seconds, so the average is 20,000,000 ÷ 86,400, which is about 230 requests a second. Traffic is never even, so allow for a busy period at around three times the average: roughly 700 requests a second.

Storage. Suppose one user in ten uploads one photo a day and a photo averages 2 megabytes. That is 100,000 photos, or 100,000 × 2 = 200,000 megabytes, which is 200 gigabytes a day. Over a year this comes to 200 × 365 = 73,000 gigabytes, or 73 terabytes.

Two conclusions follow at once. First, reads vastly outnumber writes: 20 million requests against 100 thousand uploads is a ratio of 200 to 1, so the design should make reading cheap. Second, photos will not fit comfortably in an ordinary database; they belong in storage built for large files, with only their addresses kept in the database.

The building blocks

Nearly every web system is assembled from the same few components, shown in Figure 1.1.

UsersApp serverApp serverApp serverCacheDatabaseReplicaQueueWorkerCDN (images) Loadbalancer fast copies of popular datalasting store, copied to a replicaslow jobs done in the background
Figure 1.1 — A typical web architecture. Requests enter through a load balancer, are handled by identical application servers, and rely on a cache, a database and a queue behind them.
  • A load balancer receives every request and passes it to one of several identical application servers, so no single machine is overwhelmed and a failed one can be skipped.
  • Application servers run the code. If they keep no user data of their own between requests, any server can handle any request, and you can add more at will.
  • The database is the lasting record. A replica is a continuously updated copy that can answer read requests and take over if the primary fails.
  • A cache holds recently used results in memory so that repeated questions are answered without asking the database again.
  • A queue holds jobs that need not finish before the user gets a reply, such as creating smaller versions of an uploaded photo. A worker processes them in the background.
  • A content delivery network, or CDN, stores copies of files such as images in many locations so that each user downloads from somewhere nearby.

Begin with the smallest arrangement that meets the requirements: one server and one database. Then add a component only when you can name the problem it solves.

Finding and fixing bottlenecks

A bottleneck is the part that limits the whole system, the way the narrowest section of a pipe limits the flow. Fixing anything else achieves nothing, so the skill lies in finding the right part.

There are two ways to add capacity. Vertical scaling means a bigger machine: simple, but it has a ceiling and leaves a single point of failure. Horizontal scaling means more machines: harder to arrange, but with no fixed limit and with spares when one fails.

For a system dominated by reads, the cache usually gives the largest gain. Suppose a database query takes 100 milliseconds and a cached answer takes 5. If a fraction of requests, the hit rate, is served from the cache, the average time is the weighted mix of the two. At a hit rate of 80 percent it is 0.8 × 5 + 0.2 × 100 = 4 + 20 = 24 milliseconds. At 90 percent it is 0.9 × 5 + 0.1 × 100 = 4.5 + 10 = 14.5 milliseconds.

100 ms52.5 ms24 ms14.55 0%50%80%100%cache hit rate 050100 average response time (ms)
Figure 1.2 — Average response time when a cached answer takes 5 ms and a database query takes 100 ms. Every extra point of hit rate removes almost one millisecond.

A cache is not free. The copy it holds can fall out of date when the original changes, so you must decide how long an entry may live and what happens when the underlying data is edited. For a feed of photos, showing a result that is thirty seconds old is harmless. For an account balance it is not, and that kind of data is usually read straight from the database.

Look also for any single point of failure: a component that, if it stops, takes the whole system down with it. One database with no replica is the classic example, and so is a single load balancer. The cure is always the same, a second copy ready to take over, and the cost is always the same too: more machines to pay for and keep in step.

When the database itself is the limit, replicas spread the reads across several copies. If writes are the limit, the data can be split across several databases by some key such as the user's ID, a technique called sharding. It is powerful but adds real complexity, so it comes last.

Weighing trade-offs

Every design choice buys something and costs something. Naming both sides out loud is the heart of system design. Table 1.1 sets out some of the standard choices.

ChoiceOption AOption BDeciding question
Getting updates to the userOrdinary HTTP requests: simple, and each one stands aloneWebSockets: a connection kept open so the server can push data at onceMust updates arrive within a second, as in chat?
Type of databaseRelational: fixed tables, strong guarantees, flexible queriesNon-relational: looser structure, often easier to spread over many machinesIs the data highly connected, and how strict must correctness be?
Doing slow workDuring the request: simple, but the user waitsOn a queue: quick reply, but the result arrives laterDoes the user need the result immediately?
Copies of dataAlways identical: every reader sees the latest writeAllowed to lag briefly: faster and more availableIs a few seconds of stale data acceptable?

Availability is a trade-off with a price tag. It is quoted as the percentage of time a system is working, and each extra nine is far harder to reach than the last. A year has 365 × 24 = 8,760 hours, which gives the allowances in Table 1.2.

AvailabilityAllowed downtime in a yearHow it is worked out
99%87.6 hours, about 3.65 days8,760 × 0.01
99.9%8.76 hours8,760 × 0.001
99.99%About 53 minutes8,760 × 0.0001 = 0.876 hours

A chat service whose messages must appear instantly justifies WebSockets and extra machinery. A booking site must never sell the same room twice, so it accepts slower writes in return for strict correctness. A music service can let a play count lag by a minute without anyone noticing. The right design depends on what the product can and cannot tolerate, which is why the requirements came first.

Check your understanding

  • What is the difference between a functional and a non-functional requirement? Give one of each for a chat app.
  • An app has 500,000 daily users who each make 40 requests a day. What is the average number of requests a second? (Answer: 20,000,000 ÷ 86,400, about 230.)
  • A cached answer takes 10 ms and a database query 200 ms. What is the average response time at a hit rate of 75 percent? (Answer: 0.75 × 10 + 0.25 × 200 = 7.5 + 50 = 57.5 ms.)
  • Why is it useful for application servers to keep no user data between requests?
  • How much downtime a year does 99.5 percent availability allow? (Answer: 8,760 × 0.005 = 43.8 hours.)
  • Name one situation where a queue is a better choice than doing the work during the request.

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.