Chrysolites Tech Academy

DevOps Foundations: Linux, Pipelines and Delivery · Course book

Chapter 2: From Code Change to Running System

Pairs with Lesson 2: DevOps Engineering: Review, Deploy, Monitor

"DevOps" is used to mean a job title, a set of tools and a philosophy, which makes it confusing. At heart it is a simple idea: the people who write software and the people who run it should work as one team, and the path from a code change to a running system should be automated, fast and safe. This chapter follows that path in three stages. First, how code is checked automatically before it is accepted. Second, how accepted code is deployed without disrupting users. Third, how a running system is watched so that problems are spotted before users report them.

What DevOps changes

In a traditional organisation, developers write code and hand it to a separate operations team to install and run. The two groups have different goals: developers are rewarded for shipping features, operations for keeping systems stable, and every release is a tense, manual event that happens rarely. Problems found in production go back across the wall, and weeks pass between writing a line of code and seeing its effect.

DevOps removes the wall. Teams own their code from writing through to running it, and repetitive steps are automated so that releasing becomes routine. Small changes released often are safer than large changes released rarely, because each small change is easy to understand and, if it goes wrong, easy to undo.

StageWithout automationWith DevOps practice
Checking codeSomeone remembers to run the tests, sometimesEvery change is tested and checked automatically
Reviewing changesReviewers imagine how the change behavesReviewers try the change in a temporary environment
ReleasingRare, manual, done out of hoursFrequent, scripted, done at any time
RunningUsers report problemsMonitoring and alerts spot problems first

Automating code review

Test-driven development, TDD, reverses the usual order of work: write a test describing what the code should do, watch it fail, then write just enough code to make it pass, and tidy up. The tests become a permanent safety net, so later changes that break existing behaviour are caught at once.

Continuous integration, CI, makes that safety net automatic. Every time a developer proposes a change, a CI service builds the code on a clean machine and runs the full set of tests. The result appears on the proposed change, typically as a green tick or a red cross, and changes that fail cannot be merged. Integrating small changes continuously, many times a day, avoids the painful "integration week" in which months of separately developed work are combined and found not to fit.

Two further checks are common. Code coverage measures what share of the code the tests actually run. If the tests execute 42 of a program's 50 lines, coverage is 42 ÷ 50 = 84 percent. Coverage shows which parts are untested, but it does not prove the tests are good: a test can run a line without checking that it does the right thing. Linting runs a tool that checks code for likely mistakes and for breaches of a team's style rules, such as unused variables or inconsistent formatting, so reviewers can focus on what the code does instead of on spacing.

A CI pipeline is usually described in a YAML file stored with the code, listing steps such as installing dependencies, running the linter, running the tests and building the application. Because the file lives in the repository, changes to the pipeline are reviewed like any other change, and anyone can see exactly what checks a contribution must pass. A good pipeline is fast, ideally a few minutes, because developers wait for it; slow pipelines tempt people to skip them.

The most recent addition is the ephemeral environment: for each proposed change, the CI system starts a temporary copy of the whole application, with its own address, running that change. Reviewers, designers and testers can click through the real thing before approving it. When the change is merged or abandoned, the environment is deleted.

Deployment strategies

Before looking at how code is released, it helps to compare two ways of packaging it. A virtual machine includes a complete operating system and takes minutes to start. A container packages an application with only the libraries it needs and shares the host machine's operating system kernel, so it starts in seconds and uses far less memory. Containers make it practical to run many copies of an application and to replace them quickly, which the strategies below depend on.

Suppose an application runs on 10 servers and a new version is ready. Simply stopping all 10, installing the new version and starting them again causes an outage. Two strategies avoid this.

A rolling deployment replaces servers in small batches. With batches of 2, two servers are taken out of service, upgraded and returned, then the next two, and so on: five rounds in all. At least 8 of the 10 servers are serving users at every moment. The drawback is that for a while some users get the old version and some the new, so the two must be compatible with each other and with the same database.

A blue/green deployment keeps two complete environments. "Blue" runs the current version and receives all traffic. The new version is installed on "green" and tested there while blue carries on. When green is ready, the load balancer switches all traffic to it in one step. If anything goes wrong, switching back to blue is equally instant. The price is running two full environments, at least briefly.

Users Loadbalancer Blue: version 1kept ready for rollback Green: version 2tested, now live before the switch after the switch One change at the load balancer moves every user; one change moves them back.
Figure 2.1 — A blue/green deployment. The new version is fully installed and tested before any user reaches it.

When deployments are triggered automatically every time tests pass on the main branch, the practice is called continuous deployment. Rolling and blue/green strategies are what make it safe enough to do many times a day.

Running many copies of an application raises two more needs. Autoscaling adds servers when demand rises and removes them when it falls, so the system is neither overloaded at the peak nor wastefully idle overnight. Service discovery lets the parts of a system find each other as servers come and go: instead of a fixed address written into configuration, each service registers itself in a directory that others consult.

capacity (servers × 100)demand 05001,000 1510hour requests per second (illustrative)
Figure 2.2 — Autoscaling in an illustrative day. Each server handles 100 requests a second, so the system runs 2 servers at quiet times and 10 at the peak.

Watching the running system

Once software is live, you need to know how it is behaving. Application performance management, APM, rests on two kinds of information.

Logs are time-stamped records of events written by every part of the system. With dozens of servers and containers, each writing its own logs, finding the lines relevant to one failed request is hopeless by hand. Log aggregation collects every log into one searchable store, so you can ask, for example, for every error in the last hour from the payment service.

Metrics are numbers measured continuously. A few vital ones cover most needs: how many requests arrive each second, how many fail, how long they take, and how busy each machine's processor, memory and disk are. Response time needs care, because averages hide trouble. Suppose ten requests take 80, 95, 100, 110, 120, 130, 150, 180, 400 and 900 milliseconds. The average is 226.5, but half the requests took 120 or less, the median, while the slowest tenth took 900. Teams therefore track percentiles, such as the 90th or 99th, which show what the slowest users experience.

Logs and metrics answer different questions. A metric tells you quickly that something is wrong, for example that failures jumped at 14:05. Logs tell you why, by showing the actual error messages from those failures. In practice a team watches dashboards of metrics and dives into the aggregated logs when a number looks odd.

Metrics are useful only if someone notices when they go wrong. Alerts watch them automatically and notify the people on duty when, say, the error rate stays above one percent for five minutes. Good alerts are few and meaningful: an alert that fires constantly for harmless reasons trains people to ignore it.

The feedback loop. CI tells developers within minutes whether a change works; monitoring tells them within minutes whether it behaves well in production. DevOps is, above all, about shortening those loops.

Check your understanding

  • What problem does continuous integration solve?
  • Tests run 150 of a program's 200 lines. What is the coverage, and does high coverage guarantee good tests? (Answer: 75 percent; no, tests can run lines without checking results.)
  • What is an ephemeral environment, and who benefits from it?
  • With 12 servers upgraded in batches of 3, how many rounds does a rolling deployment take, and how many servers are always serving? (Answer: 4 rounds; at least 9.)
  • Give one advantage and one cost of a blue/green deployment.
  • If each server handles 100 requests a second and demand is 640, how many servers does autoscaling need? (Answer: 7.)
  • Why are percentiles more useful than averages for response times?

Install Chrysolites Tech Academy

Android / Chrome / Edge: open the browser menu and choose Install app or Add to Home screen.

iPhone / iPad (Safari): tap the Share button, then Add to Home Screen.