Resources Security Security for engineers

Security for engineers · Lesson 6 of 11 · about 30 minutes

Scanning images and dependencies

Every image the practice cluster runs, scanned for known vulnerabilities: 28 images from seven registries, 61 critical and 1,598 high findings, most with a fix already published. Then what to do with a number like that: an example app nobody uses holds a quarter of them, one base image change takes the bookshop from 216 to 0, and a tag that looks current hasn't been rebuilt in six months. Plus the manifests scanned for misconfigurations, an SBOM, and the scanner that was itself the attack.

By the end you'll know

  • What a container image scan finds, and where in the image it finds it
  • How to read severity, "fixable" and end of life, and which to act on first
  • Why the base image decides most of the result
  • Why a tag isn't a version, and what a digest is
  • How to scan manifests and charts for misconfigurations
  • What an SBOM is, and why your scanner is part of your supply chain

Before you start

  • You need: the kind cluster running, and Docker. Trivy runs from its container image, so there's nothing to install.
  • This lesson changes nothing in the cluster or in Git. It downloads about thirty images to scan them.
  • Maps to Security+: domain 4 (vulnerability management, application security) and domain 2 (supply chain threats).

The short version

  • 28 images, 7 registries, 1 pinned by digest.
  • 61 critical, 1,598 high, and 1,459 of those critical and high findings have a fix already published.
  • The worst image is the guestbook, an example app from the Argo CD track: Debian 10, past end of life, built in 2024, 18 critical. It's still running.
  • The bookshop runs nginx:1.29: 7 critical, 209 high, built May 8. The same app on Chainguard's nginx: 0, and 23 packages instead of 151.
  • The manifests: 57 misconfiguration findings, most of them nginx running as root.
  • The scanner: Trivy itself was compromised in March 2026. This lesson pins a version from before.

What a scanner looks at

A container image is a stack of files: an operating system's packages (Debian, Alpine, Ubuntu), plus whatever the app adds (a Go binary, Python packages, a Java JAR). Every one of those has a version. An image scanner lists them, then looks each one up in a database of known vulnerabilities, each with a CVE number (Common Vulnerabilities and Exposures) and a severity.

So a scan answers: which known flaws are in the software in this image? It doesn't answer whether they can be reached, whether they matter for how you run it, or whether there are unknown ones. Keep that in mind when the numbers come in.

Trivy, from Aqua Security, is free and the most widely used. It scans images, folders, Git repositories and Kubernetes manifests. Grype, from Anchore, is the main alternative.

1. Every image the cluster runs

BASH
kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.initContainers[*]}{.image}{"\n"}{end}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u
  • range .items[*] walks every pod, then every init container and container in it, printing one image per line.
OUTPUT
docker.io/bitnami/sealed-secrets-controller:0.41.2
docker.io/grafana/alloy:v1.21.3
docker.io/grafana/grafana:13.3.0-distroless
docker.io/grafana/loki:3.7.9
docker.io/kindest/kindnetd:v20251212-v0.29.0-alpha-105-g20ccfc88
docker.io/kindest/local-path-provisioner:v20251212-v0.29.0-alpha-105-g20ccfc88
gcr.io/google-samples/gb-frontend:v5
ghcr.io/dexidp/dex:v2.45.1
nginx:1.29
nginx/nginx-prometheus-exporter:1.5.3
public.ecr.aws/docker/library/redis:8.2.3-alpine
python:3.13-alpine
quay.io/argoproj/argocd:v3.5.7
...
registry.k8s.io/kube-apiserver:v1.35.0
...
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0

28
OUTPUT
   9 quay.io
   7 registry.k8s.io
   6 docker.io
   3 docker.io (no registry named)
   1 public.ecr.aws
   1 ghcr.io
   1 gcr.io

1    pinned by digest
27   pinned by tag

Twenty-eight images from seven registries, for one small practice cluster. Each registry is someone you trust to serve the right image. Three images name no registry at all (nginx:1.29), which means Docker Hub by default.

2. The scanner, pinned

BASH
docker pull aquasec/trivy:0.69.3
docker image inspect --format '{{index .RepoDigests 0}}' aquasec/trivy:0.69.3
OUTPUT
aquasec/trivy@sha256:bcc376de8d77cfe086a917230e818dc9f8528e3c852f7b1aff648949b6258d1c

Why 0.69.3, and why the digest matters. In March 2026, attackers used stolen credentials to publish a malicious Trivy 0.69.4, then malicious Docker Hub images tagged 0.69.5 and 0.69.6, and moved 75 tags of the trivy-action GitHub Action to point at code that stole the secrets of every pipeline that ran it (CVE-2026-33634). The scanner people ran to find supply chain problems was the supply chain problem. 0.69.3 predates the attack; the digest above is the exact image, whatever happens to the tag later.

Then the vulnerability database, updated every few hours:

OUTPUT
Version: 0.69.3
Vulnerability DB:
  Version: 2
  UpdatedAt: 2026-11-07 18:32:58.034005789 +0000 UTC

A scan is only as current as its database. The same image scanned next month will show more findings, without anything in it changing.

3. Known vulnerabilities, image by image

BASH
trivy image --scanners vuln --format json --quiet IMAGE
  • --scanners vuln looks for known vulnerabilities only (Trivy can also look for secrets and misconfigurations).
  • --format json gives every finding with its package, versions and severity, which the script summarized one line per image.
OUTPUT
IMAGE                                             OS             EOL  CRIT  HIGH  MED  LOW  FIXABLE(C+H)  BUILT
gcr.io/google-samples/gb-frontend:v5              debian-10.13   EOL  18    344   477  90   230           2024-11-26
nginx:1.29                                        debian-13.4    -    7     209   308  247  150           2026-05-08
registry.k8s.io/etcd:3.6.6-0                      debian-12.9    -    6     136   113  9    142           2025-11-12
ghcr.io/dexidp/dex:v2.45.1                        alpine-3.23.3  -    5     151   121  44   156           2026-03-03
registry.k8s.io/kube-proxy:v1.35.0                debian-12.12   -    4     85    88   44   89            2025-12-17
docker.io/kindest/kindnetd:v20251212-...          debian-12.10   -    4     74    101  44   78            2025-12-12
registry.k8s.io/kube-apiserver:v1.35.0            debian-12.12   -    3     74    56   7    77            2025-12-17
...
quay.io/argoproj/argocd:v3.5.7                    ubuntu-26.04   -    2     127   180  14   129           2026-11-02
docker.io/kindest/local-path-provisioner:v2025... debian-11.10   EOL  2     49    86   52   49            2025-12-12
public.ecr.aws/docker/library/redis:8.2.3-alpine  alpine-3.22.3  -    2     24    24   32   26            2026-01-28
docker.io/grafana/grafana:13.3.0-distroless       debian-13.7    -    0     54    114  7    54            2026-10-26
quay.io/prometheus/node-exporter:v1.12.1-distr... debian-13.5    -    0     16    16   1    16            2026-07-14
...
python:3.13-alpine                                alpine-3.24.4  -    0     5     3    1    5             2026-11-05
docker.io/grafana/alloy:v1.21.3                   ubuntu-24.04   -    0     3     23   4    3             2026-10-28
quay.io/prometheus/alertmanager:v0.34.2           --             -    0     2     2    0    2             2026-10-29
quay.io/prometheus/prometheus:v3.15.1-distroless  debian-13.7    -    0     2     2    0    2             2026-10-21
docker.io/bitnami/sealed-secrets-controller:0.41.2 debian-13.7   -    0     1     1    0    1             2026-10-26
docker.io/grafana/loki:3.7.9                      debian-13.7    -    0     1     1    0    1             2026-10-20
...

critical 61  high 1598  medium 1917  low 619  fixable critical+high 1459

How to read a line:

  • CRIT, HIGH, MED, LOW: findings by severity, from the CVE's score. A finding is one CVE in one package, so one flaw in a library several packages share counts several times.
  • FIXABLE: critical and high findings where a fixed version of the package already exists. 1,459 of 1,659. Most of this isn't waiting on anyone but whoever builds the image.
  • EOL: the operating system is past its end of life. Nobody publishes fixes for it any more, so its findings only grow.
  • BUILT: when the image was built. An image doesn't get fixes; it gets replaced by a newer build.

What stands out:

  • Fewer packages, fewer findings. The images at the bottom are distroless (grafana:13.3.0-distroless, prometheus:v3.15.1-distroless) or close to it (alertmanager and the Prometheus operator show -- for their OS: no package list Trivy recognizes): just the program and the few files it needs, no shell or package manager. Prometheus: 2 high. The same project's images, built that way on purpose.
  • Recent builds help, base images matter more. Most images built in the last few weeks sit near the bottom, and the ones built a year or more ago sit near the top. The exception proves the point: Argo CD, built five days ago on a full Ubuntu, still has 2 critical and 127 high.
  • kind's own images aren't yours to fix. etcd, kube-apiserver and the rest come with the Kubernetes version kind installs. On a real cluster, upgrading Kubernetes is how those get fixed, and managed clusters do it for you.

4. The worst image, up close

OUTPUT
gcr.io/google-samples/gb-frontend:v5

CRITICAL  CVE-2019-8457   libdb5.3        5.3.28+dfsg1-0.5         no fix yet         sqlite: heap out-of-bound read in function rtreenode()
CRITICAL  CVE-2021-46848  libtasn1-6      4.13-3                   4.13-3+deb10u1     libtasn1: Out-of-bound access in ETYPE_OK
CRITICAL  CVE-2022-32221  curl            7.64.0-4+deb10u3         7.64.0-4+deb10u4   curl: POST following PUT confusion
CRITICAL  CVE-2022-36760  apache2         2.4.38-3+deb10u8         2.4.38-3+deb10u9   httpd: mod_proxy_ajp: Possible request smuggling
CRITICAL  CVE-2022-45063  xterm           344-1+deb10u2            no fix yet         xterm: code execution via OSC 50 input sequences
CRITICAL  CVE-2023-25690  apache2         2.4.38-3+deb10u8         2.4.38-3+deb10u10  httpd: HTTP request splitting with mod_rewrite and mod_proxy
CRITICAL  CVE-2023-25775  linux-libc-dev  4.19.260-1               4.19.304-1         kernel: irdma: Improper access control
CRITICAL  CVE-2023-42117  exim4-base      4.92-8+deb10u9           no fix yet         Exim: Improper Neutralization of Special Elements Remote Cod
CRITICAL  CVE-2023-45853  zlib1g          1:1.2.11.dfsg-1+deb10u2  no fix yet         zlib: integer overflow and resultant heap-based buffer overf
...

os-pkgs    debian           gcr.io/google-samples/gb-frontend:v5 (debian 10.13)  934
lang-pkgs  composer-vendor  var/www/html/vendor/composer/installed.json          0

The guestbook: the example app the Argo CD track deployed to show what an Application is. All 934 findings are in the operating system's packages, none in the app's own PHP libraries. Look at what's installed: an email server (exim4), a terminal emulator (xterm), the Linux kernel's headers, part of a C compiler (cpp-8). A small PHP web page doesn't need any of that; it's there because the image was built on a full Debian. And Debian 10 reached end of life in 2024, so "no fix yet" there means "no fix ever".

It's also not doing anything. Lesson 2 listed removing it as the first quick fix. One git rm removes 18 of the cluster's 61 critical findings. The cheapest vulnerability to fix is the one in software you don't need.

5. The bookshop: one app, four base images

The bookshop is nginx serving a page. The same nginx, built four ways:

OUTPUT
IMAGE                            OS              EOL  CRIT  HIGH  MED  LOW  FIXABLE(C+H)  BUILT       PACKAGES
nginx:1.28                       debian-13.4     -    10    222   314  247  166           2026-04-07  151
nginx:1.29                       debian-13.4     -    7     209   308  247  150           2026-05-08  151
nginx:1.29-alpine                alpine-3.23.4   -    1     54    84   96   55            2026-04-15  72
cgr.dev/chainguard/nginx:latest  wolfi-20230201  -    0     0     0    0    0             2026-11-07  23

nginx:1.28                         255 MB
nginx:1.29-alpine                  92 MB
cgr.dev/chainguard/nginx:latest    26 MB
  • The Debian image: 151 packages, 255 MB. The Chainguard image: 23 packages, 26 MB, no known vulnerabilities, built today. Chainguard builds minimal images on its own Linux, Wolfi (the 20230201 is a fixed label, not a date), and rebuilds them every day.
  • Alpine is in between: a smaller Linux, so fewer packages, fewer findings.
  • nginx:1.29, the tag the bookshop runs, was built on May 8. A tag like 1.29 is supposed to move to newer builds as fixes come out; this one hasn't moved in six months, so it carries every Debian fix released since. 150 of its findings are fixable by a rebuild nobody did.

The trade-off with minimal images is real: no shell, no package manager, often a non-root user and a different port by default. That's harder to debug (kubectl exec ... sh won't work) and the app may need small changes. It's also the point: an attacker who gets in finds no tools either. And Chainguard's free images are the latest tag only; pinned versions are a paid product.

6. Misconfigurations

Trivy also checks Kubernetes manifests and Helm charts against good practice, the same things Lesson 1 found by hand:

BASH
trivy config /repo
OUTPUT
57

HIGH    KSV-0014  4  Root file system is not read-only                          apps/bookshop/deployment.yaml, apps/monitoring/alert-receive
HIGH    KSV-0118  7  Default security context configured                        apps/bookshop/deployment.yaml, apps/monitoring/alert-receive
LOW     KSV-0003  4  Default capabilities: some containers do not drop all      apps/bookshop/deployment.yaml, apps/monitoring/alert-receive
...
MEDIUM  KSV-0001  4  Can elevate its own privileges                             apps/bookshop/deployment.yaml, apps/monitoring/alert-receive
MEDIUM  KSV-0012  4  Runs as root user                                          apps/bookshop/deployment.yaml, apps/monitoring/alert-receive
MEDIUM  KSV-0104  4  Seccomp policies disabled                                  apps/bookshop/deployment.yaml, apps/monitoring/alert-receive
MEDIUM  KSV-0117  2  Prevent binding to privileged ports                        apps/bookshop/deployment.yaml, charts/bookshop/templates/dep
MEDIUM  KSV-0125  1  Restrict container images to trusted registries            charts/bookshop/templates/deployment.yaml

57 findings in three files, and nearly all of them say one thing: these containers run with the defaults. As root, with a writable file system, every Linux capability, and nothing stopping them gaining more. That's Lesson 1's "nginx runs as root", by rule number. Fixing it is a securityContext block, which Lesson 9 adds.

Two to read with care. KSV-0110, "Workloads in the default namespace", fires because the files don't name a namespace; Argo CD sets it when it applies them, so it's a false alarm here. And KSV-0117, binding to privileged ports: nginx listens on port 80, below 1024, which non-root processes traditionally can't use. That's exactly what makes moving nginx off root more than a one-line change.

Scanning manifests is cheap and catches problems before anything runs. It's one of the most useful checks to put in a pipeline.

7. Tags move; digests don't

OUTPUT
docker.io/library/nginx:1.29  ->  docker.io/library/nginx@sha256:1881968aff6f7cdcc4b888c00a11f4ce241ad7ec957e0cb4a9e19e93a3ff87ea
docker.io/nginx/nginx-prometheus-exporter:1.5.3  ->  docker.io/nginx/nginx-prometheus-exporter@sha256:85666e7fde7e253f1c6d978f9a961245df4c9cdb7677e1866d9bb28e8f9ee41d

A tag (1.29) is a label the publisher can move to a different image at any time. A digest (sha256:1881...) is a fingerprint of the image's contents: it can only ever mean that exact image. Pulling by tag on Monday and Friday can give two different images; pulling by digest can't.

That cuts both ways, as this lesson shows. A tag that moves picks up fixes, and a tag that moves can pick up an attacker's image, as Trivy's did. Pinning by digest stops both, so teams pin by digest and use a bot (Renovate or Dependabot) that opens a pull request whenever a newer digest appears. Every change gets reviewed, and nothing goes stale silently.

Only one image here is pinned by digest: the Prometheus operator pins its own config reloader, which is why it appears twice in the list (once by tag, once by tag and digest).

8. An SBOM: what's inside one image

A software bill of materials (SBOM) is a list of every component in a piece of software, with versions, in a standard format (CycloneDX or SPDX).

BASH
trivy image --format cyclonedx nginx:1.28 > nginx-1.28.cdx.json
OUTPUT
152

 151 library
   1 operating-system

libssl3t64	3.5.5-1~deb13u1
nginx	1.28.3-1~trixie
openssl	3.5.5-1~deb13u1
zlib1g	1:1.3.dfsg+really1.3.1-1+b1

152 components, each with an exact version. Why it's worth having:

  • When the next big vulnerability lands (a Log4Shell, an OpenSSL bug), "are we affected?" becomes a search across SBOMs you already have, not a scan of everything you run.
  • It can be scanned without the image. Trivy scanned the SBOM file directly: 917 findings, against 915 for the image itself (the extra two are low-severity, from how the two scans match packages). No download needed.
  • It's increasingly asked for. US federal agencies can require one from software suppliers, and registries and build tools can attach one to every image.

In real systems

  • Scan in the pipeline and in the registry. Fail the build on new critical findings with a fix available; scan running images daily, because the database changes even when the image doesn't. Harbor, ECR, GCR and GitHub all scan images on push.
  • Choose base images on purpose. Distroless, Chainguard, or Docker Hardened Images, with a policy that says which are allowed (Kyverno or Gatekeeper can enforce it, like the KSV-0125 rule above).
  • Pin by digest, and let a bot keep it current. Renovate and Dependabot do both for images and Helm charts.
  • Sign and verify. Cosign signs images; a policy at the cluster checks the signature before anything runs. A stolen tag can't fake a signature.
  • Prioritize by risk, not count. Is it reachable from outside? Is it being exploited (CISA's Known Exploited Vulnerabilities list, EPSS scores)? Is there a fix? A critical in a package the program never loads matters less than a high in its web server.
  • Your tools are dependencies. Pin scanners and CI actions by digest or commit, not by tag. The Trivy compromise worked because thousands of pipelines trusted a tag.

Spot the problem

Work out what's wrong before you read the explanation.

1,459 tickets. A security team runs a scanner across the cluster for the first time and files one ticket per fixable critical or high finding: 1,459 of them, split between three teams, with a deadline. A month later, about 40 are closed, the teams have stopped reading the tickets, and the scanner's next run shows more than before. What went wrong? They treated findings as the unit of work, when the unit of work is the image. In this lesson's own numbers: deleting one unused example app (the guestbook) takes out 18 critical and 344 high. Moving the bookshop to a minimal base image takes it from 216 critical and high to 0. Rebuilding or upgrading each remaining image to its newest version clears most of the rest at once, because "fixable" means "a newer package exists". What's left after that is the short list worth a ticket each: findings with no fix, in images you can't replace (like kind's own), judged by whether they're reachable and being exploited. Three fixes: remove what you don't use, then change base images, then update, in that order; count images that need work, not findings; and gate new problems in the pipeline so the number stops growing while the old ones get fixed.

Keep this

ToUse
List every image a cluster runskubectl get pods -A -o jsonpath=... over containers and init containers, sort -u
Run Trivy without installing itdocker run --rm -v trivy-cache:/root/.cache/ aquasec/trivy:VERSION ...
Scan an image for known vulnerabilitiestrivy image --scanners vuln IMAGE
Only what's serious and fixabletrivy image --severity CRITICAL,HIGH --ignore-unfixed IMAGE
Scan manifests and chartstrivy config PATH
Make an SBOMtrivy image --format cyclonedx IMAGE > sbom.json
Scan an SBOMtrivy sbom sbom.json
See an image's digestdocker image inspect --format '{{index .RepoDigests 0}}' IMAGE
See what a pod is really running.status.containerStatuses[*].imageID

Where you are now

You can list everything a cluster runs, find what's known to be wrong with it, and decide what to do first: remove, rebase, update, then triage what's left. You know why a tag isn't a version, and why the scanner needs the same care as everything it scans. The next lesson moves from what's inside the containers to what they serve: the OWASP Top 10, on a deliberately insecure web shop.

Check yourself

  1. What does FIXABLE mean in a scan result?

  2. Why does Debian 10 in the guestbook image make its findings worse?

  3. Why did Chainguard's nginx have no findings?

  4. What's the difference between a tag and a digest?

  5. What does an SBOM let you do without the image?

  6. Why does this lesson pin Trivy to 0.69.3 by digest?

Key terms

  • CVE (Common Vulnerabilities and Exposures): the public catalog of known vulnerabilities, one ID each.
  • Severity: critical, high, medium or low, from the vulnerability's score (CVSS).
  • Fixable: a version of the package with the fix has been published.
  • End of life (EOL): a release that no longer gets fixes.
  • Base image: the image an app's image is built on, usually an operating system.
  • Distroless image: an image with only the app and what it needs to run: no shell, no package manager.
  • Tag and digest: a movable name for an image, and a fingerprint that can only mean one image.
  • SBOM (software bill of materials): a list of every component in a piece of software, with versions.
  • Supply chain attack: attacking the software, tools or services others depend on, to reach everyone who uses them.

Go deeper

From www.kwamib.dev/resources/security/scanning-images-and-dependencies