What CI/CD solves

Where you are

Every track so far has ended with something running: a container, a cluster, an API. This track is about the road in between: how a change gets from your laptop to production, the same way every time, without anyone having to remember the steps.

That road is called CI/CD, short for continuous integration and continuous delivery. You'll see it in almost every cloud and DevOps job description, usually next to a tool name: GitHub Actions, Jenkins, GitLab. This track builds one real pipeline in two of them side by side. GitHub Actions is the one my own project, CivicGrid, runs on. Jenkins is the one many companies, especially large and government ones, still run on their own servers. Learning both shows you which ideas belong to CI/CD itself and which belong to a tool.

This first lesson is all ideas, so there's nothing to install yet.

A group report

Picture four people writing a report together. Each takes a section, goes off, and writes for three weeks. The night before it's due, they paste everything into one document. Two sections say opposite things. One person used last year's figures. The headings are in three different styles, and someone's section quietly deleted the introduction while they were editing their copy. Fixing it takes all night, and nobody is sure the result is right.

Now picture the same group doing it differently. Everyone adds their work to the shared document every day, in small pieces. Each time someone adds something, a careful editor reads the whole document straight away and checks it against a list: figures are current, headings match, nothing contradicts anything else. When something's wrong, it's caught the same day, while the person still remembers what they changed, and it's a five-minute fix.

That second way is continuous integration. Take it one step further: because the document is checked every day, it's always in a state you could hand in. Handing it in becomes a decision, not a crisis. That's continuous delivery. And if the group agreed that every version that passes the editor gets sent to the teacher automatically, that would be continuous deployment.

In software, the editor is a machine running your checks, the list is your tests, and the shared document is the main branch of your Git repository.

The problems it fixes

Before CI/CD, shipping software often looked like the first version of that report. These are the problems it was invented for, and you may recognise some of them:

• "It works on my machine." Code passes on the developer's laptop because of something only that laptop has: a file that was never committed, a library installed months ago, a setting nobody wrote down. A pipeline runs on a clean machine every time, so anything missing shows up straight away.
• Merging hurts. When people work apart for weeks, their changes collide, and untangling them is slow and risky. Merging small changes often keeps each collision small.
• Steps get forgotten. A release that's a list of manual steps (run the tests, build the image, tag it, push it, update the server) works until the day someone skips one, usually late on a Friday. A pipeline does the same steps in the same order every time.
• Big releases are scary releases. If you ship once a month, each release carries a month of changes, and when something breaks, any of them could be the cause. Shipping small changes often means each one is easy to check and easy to undo.

None of these are about a particular tool. They're about doing the boring, important steps automatically, so people can't skip them.

Integration, delivery and deployment

The three terms build on each other, and the difference is where a person still has to step in.

• Continuous integration (CI) means everyone merges small changes into the main branch often, and every change is automatically built and tested on a clean machine. If a check fails, the team hears about it within minutes.
• Continuous delivery (CD) adds packaging: every change that passes is turned into something ready to release, like a container image, without any manual steps. Releasing it is a single, deliberate action that a person takes.
• Continuous deployment goes the last step: every change that passes goes straight to production, with nobody in between.

The "CD" in CI/CD usually means continuous delivery, but people use it for both, so it's worth asking which one a team means.

Which one is right depends on the team. Continuous deployment needs a lot of trust in your tests, because nothing stands between a merge and your users. Plenty of good teams stop at continuous delivery on purpose, so a person chooses the moment a change goes live.

A pipeline, and why tests are gates

The machinery that does all this is called a pipeline: a series of stages that every change passes through, in order. A typical one checks the code's style (called linting), runs the tests, builds a package, and hands it on to be released.

The key idea is that each stage is a gate. A change only moves forward if the stage before it passed. If the tests fail, nothing gets built, and nothing broken reaches your users. A failing pipeline isn't the pipeline going wrong: it's the pipeline doing its job, catching the problem early when it's cheap to fix.

Here's a real one. This is the journey every change to CivicGrid's API takes. Press Play to watch, then try Break the tests to see a gate stop a change, and Show Jenkins to see the same journey in Jenkins terms.

A few things worth noticing:

• Three checks run at the same time. Lint, tests and the Helm check don't depend on each other, so running them side by side saves time. The build waits for all three.
• The image is tagged with the commit. sha-664d196 points at exactly one commit, so you always know what's running. Lesson 4 is about why that matters.
• The pipeline stops at the GitOps commit. The pipeline builds and publishes the image, but a separate commit says which image the cluster should run. That split comes back in Lesson 8, and it's why my setup is continuous delivery, not continuous deployment.

needs: [lint, test, helm-lint]

Hosted or self-run: two kinds of tool

Every CI/CD tool needs two things: a description of the pipeline, and machines to run it on. Where those machines come from is the biggest difference between the tools in this track.

GitHub Actions is hosted. Your pipeline is a YAML file in your repository, called a workflow, and when something happens, like a push or a pull request, GitHub starts a fresh virtual machine called a runner, runs your jobs on it, and throws it away afterwards. You don't run any servers. It's free for public repositories and comes with a monthly allowance of minutes for private ones.

Jenkins is self-hosted: open-source software you install and run yourself, on your own servers or in your own cloud account. Your pipeline is a file in your repository called a Jenkinsfile. A central server, the controller, reads it and hands the work to one or more agents, the machines that do the building. Jenkins can do almost anything through its plugins, and it runs inside networks that can't reach the internet, which is a big part of why large companies and government agencies still use it. The cost is that it's yours to look after: upgrades, plugins, backups and security are your team's job. Lesson 9 is about exactly that.

GitLab CI/CD, the third name you'll see in job descriptions, sits in between: GitLab can host your runners like GitHub, or you can run the whole of GitLab yourself like Jenkins. Its pipeline file is YAML, much like a GitHub workflow. Lesson 2 maps all three side by side, so you can read any of them.

The point to take away: the ideas are the same in every tool. Stages, gates, clean machines, building once and releasing the same thing everywhere. Learn them once, and switching tools is mostly a matter of new names.

What a pipeline can't do

One honest warning, much like the one in the Compliance track. A green pipeline only means your checks passed. It can't tell you anything your checks don't test.

If a feature has no tests, the pipeline will happily pass a broken version of it. If the tests only check that a function runs, not that it gives the right answer, a wrong answer sails through. A pipeline makes your checks automatic and impossible to skip, which is a huge improvement on remembering to run them. But the checks are only as good as what you write. That's why Lesson 3 is about making tests that actually guard something, and Lesson 7 adds security scanners that catch problems most tests never look for.

Spot the problem

Three situations from real teams. Work out what's wrong before you read the explanation.

Works on my machine. A developer runs all the tests on their laptop, they pass, and they merge. The pipeline fails straight away on the same tests: a module can't be found. The developer is sure the pipeline is broken. What's more likely? They created a new file and forgot to commit it. Their laptop has the file, so the tests pass there; the pipeline starts from a clean copy of the repository, which doesn't. The pipeline isn't broken: it caught exactly the problem it's there for.

Continuous, once a month. A team says they do continuous integration because they have a pipeline. But each developer works on their own branch for three or four weeks, and merges come in a rush at the end of the month, each with dozens of conflicts. What are they missing? The "continuous" part. A pipeline on its own isn't CI; merging small changes often is. With weeks between merges, the pipeline checks each change too late to keep integration easy.

The deploy that didn't wait. A pipeline has three stages: tests, build and deploy. One day a change with a failing test reaches production anyway. Looking at the setup, the build and deploy stages were set to start as soon as the pipeline began, alongside the tests, instead of after them. What went wrong? The gate was missing. The stages ran at the same time instead of in order, so nothing was waiting for the tests to pass. In GitHub Actions, that's what needs: is for; in Jenkins, stages run in order unless you put them in a parallel block. Lesson 3 covers both.

Where you are now

You know what CI/CD is for: catching problems early, on a clean machine, through the same steps every time. Continuous integration builds and tests every change; continuous delivery makes every passing change ready to release; continuous deployment releases it automatically. A pipeline is a series of gates, and a green pipeline only proves what your tests check. GitHub Actions runs your pipeline on GitHub's machines, Jenkins on machines you run yourself, and the ideas carry across both. Lesson 2 opens the files: a GitHub workflow and a Jenkinsfile, side by side, and your own Jenkins running in Docker.

Check yourself

Key terms

• CI/CD: continuous integration and continuous delivery (or deployment).
• Continuous integration (CI): merging small changes often, with each one built and tested automatically.
• Continuous delivery: every passing change is packaged and ready to release; a person decides when.
• Continuous deployment: every passing change is released automatically.
• Pipeline: the series of stages a change goes through, like lint, test, build and release.
• Gate: a stage that must pass before the next one starts.
• Linting: automatically checking code for style problems and common mistakes.
• Workflow: a GitHub Actions pipeline, written as a YAML file in the repository.
• Runner: the machine that runs a GitHub Actions job, usually a fresh virtual machine.
• Jenkinsfile: a Jenkins pipeline, written as a file in the repository.
• Controller and agent: the Jenkins server that plans the work, and the machines that do it.

Go deeper

• Martin Fowler's Continuous Integration (opens in a new tab), the long-standing explanation of what CI is and why it works. It's long, but the first few sections are enough for now.
• GitHub's Understanding GitHub Actions (opens in a new tab), a short tour of workflows, events, jobs and runners, which Lesson 2 builds on.
• The Jenkins handbook's Pipeline (opens in a new tab) chapter, which introduces the Jenkinsfile and the idea of pipeline as code.