What GitOps solves

By the end you'll know

Why kubectl apply from laptops and pipelines stops scaling
The four ideas behind GitOps, and what Argo CD does with them
What an Argo CD Application is: where to look in Git, and where to put it in the cluster
What happens when someone changes the cluster by hand
How GitOps differs from Ansible, and from a pipeline that pushes changes

Before you start

You need: the Kubernetes track, at least up to Deployments and Services, and Docker running on your computer.
Install the tools: brew install kind kubectl on a Mac. The Kubernetes track's Lesson 2 covers kind on other systems.
Get the files: git clone https://github.com/Kwamib/gitops-practice.git, then cd gitops-practice.
Memory: the practice cluster uses about 2 GB while it runs. docker stop gitops-control-plane gitops-worker pauses it.
Where commands run: your own computer's terminal.

The short version

The problem. When people deploy with kubectl apply, from laptops or pipelines, nobody can say for sure what's running, who changed it, or how to get back to yesterday.
The fix. GitOps: Git holds the description of what should run, and a controller inside the cluster keeps the cluster matching it. A deploy is a commit. A rollback is a revert.
The first look. Argo CD, installed on a local cluster, deployed an app from GitHub in under 20 seconds. Then the app was scaled up by hand, and deleted by hand, and both times Argo CD put it back within seconds.

Who deployed that?

The Kubernetes track deployed everything the same way: write some YAML, run kubectl apply. That's fine for one person and one cluster. Now picture a team. Five people can run kubectl apply, from five laptops, each with their own copy of the YAML, some of it edited and never committed. A pipeline runs it too, with credentials that can change anything in the cluster. One evening, someone runs kubectl edit to fix something during an incident, and doesn't write it down.

A month later, three questions have no good answer:

What's actually running? The YAML in Git says one thing. The cluster may say another.
Who changed it, and why? Kubernetes doesn't keep that history.
How do we get back to how it was last Tuesday? Only by remembering.

This is the same problem as the hand-built server in the Linux and Ansible tracks, one level up: a cluster nobody can describe, and so nobody can rebuild.

GitOps

GitOps is a way of running systems where Git is the only way in. Four ideas, as the OpenGitOps (opens in a new tab) project puts them:

Declarative. The system is described as the result you want, not steps to get there. Kubernetes YAML already is.
Versioned and immutable. That description lives in Git, so every change is a commit, with an author, a time and a reason, and old versions are never lost.
Pulled automatically. Software inside the cluster fetches the description from Git. Nobody pushes changes in.
Continuously reconciled. That software keeps comparing the cluster with Git, and fixes any difference, all the time, not only when someone deploys.

Think of a thermostat. You don't heat the room yourself; you set the temperature you want, and the thermostat keeps checking and correcting, all day. Open a window, and it turns the heat up. Git is where you set the temperature. Argo CD is the thermostat: a controller that runs in the cluster, watches a Git repository, and makes the cluster match it.

A first look

Here's where this lesson is heading. Everything below runs on a cluster on your own computer, made with kind, as in the Kubernetes track.

A cluster

kind-gitops.yaml describes it: one control plane and one worker.

BASH

kind create cluster --name gitops --config kind-gitops.yaml
kubectl get nodes

OUTPUT

NAME                   STATUS     ROLES           AGE   VERSION
gitops-control-plane   NotReady   control-plane   10s   v1.35.0
gitops-worker          NotReady   <none>          1s    v1.35.0

NotReady only because they're seconds old; give them a moment. Creating it took 15 seconds.

Argo CD

Argo CD installs like any other app on Kubernetes: one YAML file, applied to its own namespace. The file is pinned to a version, so you get exactly what this lesson shows:

BASH

kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.7/manifests/install.yaml

(--server-side has the cluster work out the changes, rather than kubectl. Argo CD's install instructions use it, because some of its definitions are too large for the older way.) What that one file contains, counted:

OUTPUT

   3 clusterrole.rbac.authorization.k8s.io
   3 clusterrolebinding.rbac.authorization.k8s.io
   7 configmap
   3 customresourcedefinition.apiextensions.k8s.io
   6 deployment.apps
   7 networkpolicy.networking.k8s.io
   6 role.rbac.authorization.k8s.io
   6 rolebinding.rbac.authorization.k8s.io
   2 secret
   8 service
   7 serviceaccount
   1 statefulset.apps

Mostly things you met in the Kubernetes track: Deployments, Services, ConfigMaps, and the permissions from the access-control lesson. The new ones are the three custom resource definitions (CRDs): they teach Kubernetes new kinds of object, and the one that matters here is called an Application. Then wait for it to start:

BASH

kubectl wait -n argocd --for=condition=Available deployment --all --timeout=300s
kubectl get pods -n argocd

OUTPUT

NAME                                                READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                     1/1     Running   0          34s
argocd-applicationset-controller-7f9f48c469-bfplk   1/1     Running   0          34s
argocd-dex-server-64765dc69-7h8t6                   1/1     Running   0          34s
argocd-notifications-controller-87598c956-n45jb     1/1     Running   0          34s
argocd-redis-878c58d8d-sp27d                        1/1     Running   0          34s
argocd-repo-server-86655dc5cc-9z8xw                 1/1     Running   0          34s
argocd-server-799b87744b-lfrl5                      1/1     Running   0          34s

Seven pods, ready in 34 seconds. Three do the core work:

repo-server fetches the repository from Git and turns it into plain Kubernetes YAML.
application-controller compares that YAML with the cluster, and changes the cluster to match. This is the thermostat.
server is the web interface and the API, which Lesson 2 opens.

The others handle login (dex), a cache (redis), messages to chat and email (notifications), and generating many Applications from one template (applicationset).

An Application

An Application tells Argo CD two things: where to look in Git, and where to put it in the cluster. argocd/guestbook.yaml points at the example app in Argo CD's own example repository:

YAML

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

source: the repository, which version (HEAD, the latest commit), and which folder in it.
destination: which cluster (kubernetes.default.svc is "this one") and which namespace.
syncPolicy.automated: apply changes from Git without waiting for a person. prune deletes things that were removed from Git; selfHeal puts back anything changed in the cluster by hand.
CreateNamespace=true: make the guestbook namespace if it doesn't exist.

BASH

kubectl apply -f argocd/guestbook.yaml
kubectl get application guestbook -n argocd
kubectl get deployment,service,pods -n guestbook

OUTPUT

application.argoproj.io/guestbook created

NAME        SYNC STATUS   HEALTH STATUS
guestbook   Synced        Healthy

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/guestbook-ui   1/1     1            1           18s

NAME                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/guestbook-ui   ClusterIP   10.96.3.40   <none>        80/TCP    18s

NAME                                READY   STATUS    RESTARTS   AGE
pod/guestbook-ui-6595f948db-rxtrc   1/1     Running   0          18s

You applied an Application, not a Deployment. Argo CD read the guestbook folder from GitHub, and made the Deployment and the Service itself, within 18 seconds. Two words to know:

Sync status: does the cluster match Git? Synced, or OutOfSync.
Health status: is what's running working? Healthy, Progressing while it starts, or Degraded.

And Argo CD records exactly which version of Git it deployed:

BASH

kubectl get application guestbook -n argocd -o jsonpath='{.status.sync.revision}{"\n"}'

OUTPUT

8088f4c0d970abb09e250248cc97e35623447cb5

A Git commit ID. "What's running?" now has an exact answer: that commit.

Changing it by hand

Now the reason Argo CD exists. Someone scales the app up by hand, outside Git:

BASH

kubectl scale deployment guestbook-ui -n guestbook --replicas=3

Checking every three seconds how many copies the Deployment wants:

OUTPUT

deployment.apps/guestbook-ui scaled
after 1x3s: 3 replicas wanted
after 2x3s: 1 replicas wanted
after 3x3s: 1 replicas wanted
after 4x3s: 1 replicas wanted
...

Back to 1 within six seconds, and it stays there. Git says one copy, so the cluster has one copy. Something more drastic: deleting the app outright.

BASH

kubectl delete deployment guestbook-ui -n guestbook
sleep 10
kubectl get deployment guestbook-ui -n guestbook

OUTPUT

deployment.apps "guestbook-ui" deleted

NAME           READY   UP-TO-DATE   AVAILABLE   AGE
guestbook-ui   1/1     1            1           10s

Back, and running, ten seconds later. Its age is 10 seconds: a new Deployment, made from Git. Argo CD's events show what it did each time:

OUTPUT

OperationStarted     Initiated automated sync to '8088f4c0d970abb09e250248cc97e35623447cb5'
ResourceUpdated      Updated sync status: Synced -> OutOfSync
OperationCompleted   Partial sync operation to 8088f4c0d970abb09e250248cc97e35623447cb5 succeeded
ResourceUpdated      Updated sync status: OutOfSync -> Synced
ResourceUpdated      Updated health status: Healthy -> Progressing
ResourceUpdated      Updated health status: Progressing -> Healthy

It noticed the cluster no longer matched (Synced -> OutOfSync), synced back to the same commit, and watched the new pod start (Progressing -> Healthy). "Partial sync" means it only reapplied what had changed, not the whole app. This is drift, caught and fixed, the thing Lesson 11 of the Ansible track could only find by rebuilding a server.

Push and pull

The CI/CD track's pipelines pushed changes: GitHub Actions ran a deploy step, with credentials that could change the target. GitOps pulls: the controller inside the cluster fetches from Git. Three things follow:

The cluster's credentials never leave the cluster. The pipeline needs permission to push to Git, not to change the cluster.
Nothing happens only once. A push deploys and forgets. A pull keeps checking, so a hand change is undone, as you just saw.
Git's history is the deploy history. Every change has a commit, an author and a message, and going back is git revert.

A pipeline still has a job: it builds and tests the image, then commits the new image tag to Git. Argo CD does the rest.

Argo CD and Ansible

Both describe a result and make it true, and both are idempotent. The difference is when they act. Ansible acts when you run it: between runs, a hand change stays until the next run, which is why the Ansible track suggested running --check --diff on a schedule to notice drift. Argo CD never stops: it compares and fixes all the time. They fit together, too: Ansible for the servers, Argo CD for what runs on the cluster those servers make.

In real systems

Git becomes the front door. Teams that adopt GitOps take away most people's permission to change the cluster directly. Changes go through pull requests, reviewed like code, and Argo CD applies them. The cluster's audit trail is the repository's history, which is also what auditors ask for.
Image tags are pinned in Git. A common split: the app's repository holds its code and its chart, and a separate GitOps repository holds which version runs where. Deploying a new version is a one-line change to that repository. Lesson 7 builds exactly that.
One Argo CD, many clusters. A single Argo CD can manage many clusters: development, staging and production, each with its own Applications. Lesson 8's app-of-apps pattern keeps them organized.
Self-heal is a choice. Some teams start with automated sync but without selfHeal, so Argo CD reports drift and a person decides. Lesson 4 compares the two.

Spot the problem

Work out what's wrong before you read the explanation.

The fix that kept disappearing. At 2 a.m., a production app is running out of memory. The engineer on call raises its memory limit with kubectl edit, the pods restart, and the alerts stop. Ten seconds later, the pods restart again, with the old limit, and the alerts come back. They try again; same thing. They start to suspect the cluster is broken. What's happening? Argo CD is doing its job. The app's Application has selfHeal: true, and Git still says the old limit, so every hand change is undone within seconds, exactly as with kubectl scale above. The fix that sticks is to change the limit in Git, as a small pull request, and let Argo CD apply it. If that's too slow in an emergency, Argo CD can pause automated sync for one Application while you work, and you turn it back on, with the fix in Git, afterwards. Either way, the cluster and Git end up agreeing.

Keep this

To	Run
Create the practice cluster	`kind create cluster --name gitops --config kind-gitops.yaml`
Install a pinned Argo CD	`kubectl apply -n argocd --server-side --force-conflicts -f .../v3.5.7/manifests/install.yaml`
Wait for it to be ready	`kubectl wait -n argocd --for=condition=Available deployment --all`
Add an Application	`kubectl apply -f argocd/guestbook.yaml`
See its sync and health status	`kubectl get application guestbook -n argocd`
See which Git commit it deployed	`kubectl get application guestbook -n argocd -o jsonpath='{.status.sync.revision}'`
See what Argo CD did	`kubectl get events -n argocd --field-selector involvedObject.name=guestbook`
Pause the cluster, start it again	`docker stop gitops-control-plane gitops-worker`, then `docker start`
Delete the cluster	`kind delete cluster --name gitops`

Where you are now

You have a cluster on your own computer, Argo CD running in it, and an app it deployed from Git and keeps there, undoing hand changes within seconds. Lesson 2 looks at Argo CD itself: its command-line tool, its web interface, and logging in, so you can see what it sees.

Check yourself

In GitOps, what decides what runs in the cluster?
What does an Argo CD Application tell it?
The guestbook was scaled to 3 by hand. What happened?
What does Synced mean, as opposed to Healthy?
Why is pulling from Git safer than a pipeline pushing to the cluster?
How is Argo CD different from Ansible?

Key terms

GitOps: running systems with Git as the description of what should run, and a controller that keeps the system matching it.
Argo CD: a GitOps controller for Kubernetes, which runs inside the cluster and watches Git repositories.
Application: Argo CD's object saying where to look in Git, and where to put it in the cluster.
Custom resource definition (CRD): a way to teach Kubernetes a new kind of object, like an Application.
Sync status: whether the cluster matches Git: Synced or OutOfSync.
Health status: whether what's running is working: Healthy, Progressing or Degraded.
Self-heal: Argo CD undoing changes made to the cluster outside Git.
Drift: a difference between the cluster and what Git says.
Reconcile: compare what is with what should be, and fix the difference.

Go deeper

OpenGitOps (opens in a new tab): the four GitOps principles, in their own words.
The Argo CD documentation: Core concepts (opens in a new tab), and Getting started (opens in a new tab).
Try: edit the guestbook's Service with kubectl edit service guestbook-ui -n guestbook, and change its port. How long does the change last?
Try: apply argocd/guestbook.yaml again with selfHeal: false, then scale the app by hand. What does kubectl get application guestbook -n argocd say now, and what doesn't happen?