DevOps Foundations: Linux, Pipelines and Delivery · Course book
Chapter 4: Pipelines That Deploy to the Cloud
Jenkins is a separate server that you install and look after. Many teams now prefer CI/CD that is built into the place where their code already lives. GitLab, a platform for hosting code, includes a complete pipeline system, GitLab CI/CD, configured by a single file in the repository. This chapter uses it to build the full journey from a code change to a website running in the AWS cloud: how pipelines are structured, how merge requests and code review fit in, how files are deployed to cloud storage safely, and how a containerised application is shipped to a managed hosting service.
Pipelines, stages and jobs
A GitLab pipeline is defined in a file called .gitlab-ci.yml at the root of the repository. Whenever code is pushed, GitLab reads the file and runs the pipeline it describes. The basic unit is a job: a named list of shell commands, run in a fresh environment, usually a Docker container started from an image you choose. Jobs are grouped into stages, which run in order; jobs within one stage run in parallel.
stages:
- build
- test
build website:
stage: build
image: node:16
script:
- npm install
- npm run build
artifacts:
paths:
- build
test website:
stage: test
image: node:16
script:
- test -f build/index.html
- npm test
Each job runs in a brand-new container, so nothing carries over from one job to the next by default. If the build job produces a folder of files that the test job needs, the build job must declare it as an artifact; GitLab saves it and hands it to later jobs. The jobs themselves are carried out by runners, machines that pick up waiting jobs, run them and report back. GitLab provides shared runners, or you can register your own.
If any command in a job finishes with a non-zero exit code, the job fails, and by default the pipeline stops before later stages. Pipelines fail for a handful of common reasons: a typo in the YAML, often wrong indentation; a command that is not installed in the chosen image; a file that a job expected but an earlier job did not save as an artifact; or a test that genuinely failed. GitLab shows each job's full log, and the first error in it is usually the one to fix.
| Concept | What it is | Example |
|---|---|---|
| Pipeline | Everything that runs for one push | Pipeline #128 for the latest commit |
| Stage | A step in the sequence; its jobs run together | build, test, deploy |
| Job | A named list of commands run in one container | "test website" |
| Artifact | Files saved by a job for later jobs or for download | The build folder |
| Variable | A named value available to jobs | The name of the storage bucket |
| Runner | A machine that executes jobs | A shared runner provided by GitLab |
| Environment | A named place the application is deployed to | staging, production |
Variables make a pipeline reusable. A value such as the name of the bucket to deploy to can be defined once at the top of the file, or in the project's settings, and referred to in any job as $BUCKET_NAME. GitLab also provides many predefined variables automatically, such as the current branch name and a short identifier for the commit being built, which are handy for naming builds and images. Because every job starts from a clean container, it helps to think of each job as a small, self-contained script that receives exactly what it needs through variables and artifacts, and nothing more.
Continuous integration with merge requests
Continuous integration depends on a disciplined way of combining work. Instead of everyone pushing directly to the main branch, each change is developed on its own branch, and when it is ready the developer opens a merge request asking for it to be merged into main. The pipeline runs on the branch, and the merge request shows its result. Teams usually configure GitLab so that a merge request cannot be merged unless the pipeline has passed.
The merge request is also where code review happens. Colleagues read the changes, comment on particular lines and approve or ask for changes. Automated checks handle the mechanical questions, such as whether the code builds and the tests pass, so reviewers can concentrate on whether the change is correct, clear and sensible.
Beyond unit tests, pipelines often include integration tests, which check that parts work together, for instance by starting the built website in the job and requesting a page to confirm it contains the expected text. A sensible structure orders stages from cheapest and fastest to slowest, so that most problems are caught within a minute or two and expensive tests run only on code that has already passed the quick ones.
Deploying to the cloud safely
Amazon Web Services, AWS, offers hundreds of services. Two are enough for a static website. S3, the Simple Storage Service, stores files in containers called buckets and can serve a bucket's contents as a public website. The AWS command-line interface, run inside a job, uploads files to it: a single sync command copies the build folder to the bucket, transferring only files that changed.
To do that, the job needs AWS credentials, and how they are handled matters a great deal. Credentials must never be written into .gitlab-ci.yml, which anyone with access to the repository can read. Instead they are stored as CI/CD variables in GitLab's project settings, marked as masked, so their values are hidden in job logs, and protected, so they are available only to pipelines on protected branches such as main. The credentials themselves should belong to a dedicated AWS user created with IAM, AWS's identity and access management service, and given only the permissions it needs, here permission to write to one bucket and nothing else. If the key ever leaked, the damage would be limited to that bucket.
Not every job should run on every branch. A rules section can restrict the deploy job to the main branch, so that feature branches are built and tested but never deployed. After deploying, a post-deployment test requests the live site and checks for expected content, confirming that the deployment actually worked instead of assuming it did.
GitLab's environments add structure to this. A typical pipeline deploys first to a staging environment, a copy used for final checks, and then to production. Repeated configuration between the two deploy jobs can be shared through templates in the YAML file, so that a change to the deployment steps is made once. Deploying to production automatically is continuous deployment; requiring a person to press a button for the final step is continuous delivery.
Shipping a containerised application
Static files are the simple case. An application that runs code on the server needs a running environment. Packaging it as a Docker image makes it portable: the Dockerfile in the repository describes the image, and a pipeline job builds it. To run Docker commands inside a job that itself runs in a container, the job uses a special Docker service alongside it.
The built image is pushed to GitLab's own container registry, and tagged with the pipeline's identifier so every version can be traced back to its commit. A test job then starts the container and checks that it responds before anything is deployed.
For hosting, the course uses AWS Elastic Beanstalk, a service that runs an application from a description of what to deploy and handles the servers, load balancing and health checks itself. The deploy job tells Elastic Beanstalk which image to run, by uploading a small file naming the image and its tag. Because the image lives in a private registry, Elastic Beanstalk also needs credentials to pull it, again supplied through protected, masked variables. A final post-deployment test confirms that the new version is live and reports the version it is serving.
Managed services like Elastic Beanstalk trade control for convenience. You do not choose every setting of the underlying servers, but you also do not have to patch them, replace failed machines or configure the load balancer by hand. For a small team, that trade is usually worthwhile; larger organisations often move on to orchestration platforms that offer more control, at the cost of more to manage.
The result is the complete loop: a developer opens a merge request; the pipeline builds, tests and packages the change; reviewers approve; merging to main deploys it to staging and then to production; and a final check confirms it works, all without anyone logging into a server.
Check your understanding
- What is the difference between a stage and a job in GitLab CI?
- Why must a build job declare its output as an artifact if a later job needs it?
- Name three common reasons a pipeline fails.
- Why should a merge request be blocked until its pipeline passes?
- What do the masked and protected settings on a CI/CD variable each do?
- Why give the pipeline's AWS user permission to write to only one bucket?
- What distinguishes continuous delivery from continuous deployment?
This course book is available to read online only at tech-academy.chrysolites.org.