Policy as code: checking the rules automatically

Where you are

Lesson 4 showed how Git, pull requests and Terraform record every change. That's evidence after the fact. This lesson moves earlier: checking every change against the rules automatically, before it reaches production, and turning some controls from "please remember to" into "the system won't let you".

This one is hands-on throughout. You'll run three real tools, watch them fail on purpose, and fix what they find.

Spell-check, not proofreading

When you write a document, you could ask a friend to read every word for spelling mistakes, every time. Or you could let the spell-checker underline them as you type. The friend is still useful for whether the writing makes sense; but nobody uses a friend to find typos any more.

Policy as code is the spell-checker for infrastructure. Instead of a written rule like "SSH must never be open to the internet" that a reviewer has to remember to check, you write the rule as code, and a tool checks every change against it, every time, in seconds. Reviewers are still needed for judgement. They're just not needed to catch the same ten mistakes over and over.

The written rule becomes a test, the test result becomes evidence, and a failing test can block the change. That's three controls from earlier lessons in one: preventive, detective and evidence-producing.

Two places to check

Policy checks run at two points, and good setups use both:

• Before deployment, on the code: scanning Terraform and Kubernetes files on every pull request. A problem found here costs a minute to fix and never reaches production. People call this shifting left, because it moves the check earlier, to the left of a timeline.
• At the door, and after deployment, on what's actually running: blocking bad resources as they're created, and scanning the live system for anything that slipped through, or drifted, as Lesson 4 described.

Checkov: scan Terraform before it's applied

Checkov is a free, open-source scanner that reads Terraform, Kubernetes files and more, and checks them against hundreds of built-in security rules. It doesn't need a cloud account: it reads the files, so you can try it on Terraform you've never applied. Install it with Homebrew on a Mac, or with pip:

brew install checkov

In a new folder, create main.tf with a security group from Lesson 8 of the Networking track, with a mistake in it:

resource "aws_security_group" "web" {
  name        = "web"
  description = "Web servers"

  ingress {
    description = "HTTPS from anywhere"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "SSH for admins"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Scan it. --compact leaves out the code snippets, and --quiet shows only what failed:

checkov -f main.tf --compact --quiet

terraform scan results:

Passed checks: 5, Failed checks: 2, Skipped checks: 0

Check: CKV_AWS_24: "Ensure no security groups allow ingress from 0.0.0.0:0 to port 22"
	FAILED for resource: aws_security_group.web
	File: /main.tf:1-20
Check: CKV2_AWS_5: "Ensure that Security Groups are attached to another resource"
	FAILED for resource: aws_security_group.web
	File: /main.tf:1-20

Two failures, each with an ID and a plain description:

• CKV_AWS_24 is the real problem: SSH open to the whole internet, exactly what Networking Lesson 8 warned against. It's a breach of least privilege, AC-6, and of boundary protection, SC-7.
• CKV2_AWS_5 says the security group isn't attached to anything. In this file it isn't, because the servers would be defined somewhere else. That's a false positive here: the rule is sensible, but it doesn't apply.

Fix the first, and record a reason for the second. Checkov reads special comments to skip a check, and the reason you write is kept in the report, which makes the exception itself evidence:

resource "aws_security_group" "web" {
  #checkov:skip=CKV2_AWS_5:Attached to the web servers in servers.tf
  name        = "web"
  description = "Web servers"

  ingress {
    description = "HTTPS from anywhere"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "SSH from the office only"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["198.51.100.7/32"]
  }
}

SSH now only accepts the office's one address, a /32 from the Networking track. Scan again:

terraform scan results:

Passed checks: 6, Failed checks: 0, Skipped checks: 1

The part that makes it a control is the exit code. Checkov exits with 1 when any check fails, and 0 when everything passes, the same idea as Terraform's exit codes in Lesson 4. Run it as a required check on every pull request, and a change that opens SSH to the internet simply can't be merged. That's a preventive control, enforced by a machine, with the scan results as evidence.

kube-bench: check a cluster against the CIS Benchmark

Lesson 2 introduced the CIS Benchmarks, the community's hardening guides. There's one for Kubernetes, with checks like "the API server should not allow anonymous access" and "the kubelet's config file should have restrictive permissions". kube-bench is a free tool that runs those checks on a real cluster and reports each one as PASS, FAIL or WARN.

You'll need a kind cluster, from the Kubernetes track's first cluster lesson. Create a throwaway one:

kind create cluster --name policy

kube-bench runs as a short job inside the cluster, so it can see the node's files. Apply the job from the project's own release, wait for it to finish, and read the summary:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/v0.16.0/job.yaml
kubectl wait --for=condition=complete job/kube-bench --timeout=180s
kubectl logs job/kube-bench | tail -8

== Summary total ==
63 checks PASS
12 checks FAIL
56 checks WARN
0 checks INFO

That's a real run on a brand new kind cluster: 63 checks pass, 12 fail, and 56 are warnings, mostly checks kube-bench can't decide automatically and leaves for a person. kind is built for learning on a laptop, not for production, so failures are expected. Look at a few of them:

kubectl logs job/kube-bench | grep "^\[FAIL\]"

[FAIL] 1.2.15 Ensure that the --profiling argument is set to false (Automated)
[FAIL] 1.2.16 Ensure that the --audit-log-path argument is set (Automated)
[FAIL] 1.2.17 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)
[FAIL] 1.2.18 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)
[FAIL] 1.2.19 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)

Those are a few of the 12. Four of them say the same thing: the cluster's API server isn't keeping an audit log, the record of who asked the cluster to do what. That's event logging, AU-2 from Lesson 3, missing on this cluster, found in seconds by a tool rather than by an auditor months later.

On a cloud provider's managed cluster, many checks don't apply, because the provider runs the control plane you can't see, which is why providers publish their own versions of the benchmark. The value isn't a perfect score; it's knowing which settings differ from the benchmark, and having a reason for each one. That list of reasons is exactly what an auditor asks for.

Kyverno: block bad deployments at the door

Checkov checks code, and kube-bench checks the running cluster. Kyverno checks at the door: it plugs into Kubernetes's admission control, the step where the cluster decides whether to accept a new or changed resource, and refuses anything that breaks a policy.

Install it on the same cluster, from its own release, and wait for it to be ready:

kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.19.1/install.yaml
kubectl -n kyverno wait --for=condition=Available deployment --all --timeout=240s

Here's a policy for a real control. CM-2 asks for a known, recorded baseline. An image tagged latest, or with no tag at all, could be a different version every time it's pulled, so nobody can say exactly what's running. This policy refuses both. Create no-latest.yaml:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  rules:
  - name: require-image-tag
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      failureAction: Enforce
      message: "Pin every image to a version."
      pattern:
        spec:
          containers:
          - image: "*:*"
  - name: no-latest-tag
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      failureAction: Enforce
      message: "The latest tag is not allowed. Pin the image to a version."
      pattern:
        spec:
          containers:
          - image: "!*:latest"

It has two rules: every image must have a tag (*:*), and that tag can't be latest (!*:latest, where ! means not). failureAction: Enforce means refuse; the other choice, Audit, lets the resource in but records the problem, which is a good way to roll a new policy out gently. Apply it, and try it:

kubectl apply -f no-latest.yaml

Warning: kyverno.io/v1 ClusterPolicy is deprecated and will be removed in a future release; migrate to ValidatingPolicy, MutatingPolicy, GeneratingPolicy or ImageValidatingPolicy (policies.kyverno.io), see https://kyverno.io/docs/guides/migration-to-cel/
clusterpolicy.kyverno.io/disallow-latest-tag created

The policy is in, with a warning worth understanding. Kyverno is moving to a newer kind of policy, ValidatingPolicy, written in CEL (Common Expression Language), the same expression language Kubernetes uses for its own built-in policies. The ClusterPolicy style here still works, and it's what most existing policies and examples you'll find use, so it's the clearer one to learn first. New policies in a real cluster are worth writing in the new style; the link at the end of the warning shows how. Now try to break the rule:

kubectl run bad --image=nginx:latest

Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/default/bad was blocked due to the following policies 

disallow-latest-tag:
  no-latest-tag: 'validation error: The latest tag is not allowed. Pin the image to a version. rule no-latest-tag failed at path /spec/containers/0/image/'

Now an image with no tag at all, which means latest without saying so:

kubectl run bad2 --image=nginx

Error from server: admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Pod/default/bad2 was blocked due to the following policies 

disallow-latest-tag:
  require-image-tag: 'validation error: Pin every image to a version. rule require-image-tag failed at path /spec/containers/0/image/'

Each refusal names the policy, the rule that failed and your message. And a pinned image:

kubectl run good --image=nginx:1.27

pod/good created

The first pod never existed: the cluster refused it at the door, with your message. The second, pinned to a version, went straight in. That's a preventive control enforced on every pod anyone creates, by any route: kubectl, a pipeline or Argo CD.

When you're done, delete the cluster:

kind delete cluster --name policy

Break it on purpose

Three ways policy as code goes wrong. You've seen the first already.

The false positive. Checkov failed CKV2_AWS_5, "security group not attached", on a group that was attached in another file. If people get too many false positives, they stop reading the results, and real failures hide among them. The fix isn't to turn the scanner off; it's to skip specific checks with a recorded reason, as you did, and to review those skips now and then. A skip with a reason is a decision; a skip without one is a gap.

Audit forever. A team rolls out Kyverno with every policy on Audit, to see what would break. Six months later they're still on Audit, and the policy reports list hundreds of violations nobody looks at. What went wrong? Audit mode is a detective control, and, like the drift check in Lesson 4, a detective control nobody acts on is no control at all. Audit is the right way to start; the plan should always end with switching to Enforce, policy by policy, with a date.

The scan that doesn't block. A team adds Checkov to its pipeline, and it runs on every pull request. But it's not a required check, so when it fails, people merge anyway "to fix it later". Is this a control? Barely. It produces evidence that problems were known and ignored, which is arguably worse than no evidence. A policy check only prevents anything if a failure stops the merge, which is the branch protection setting from Lesson 4.

Where you are now

You've turned controls into code: Checkov blocking an open SSH port before it's applied, kube-bench measuring a cluster against the CIS Benchmark, and Kyverno refusing an unpinned image at the cluster's door. You know where each one checks, how exit codes and enforcement make them preventive, and why every exception needs a reason. Lesson 6 steps back from tools to process: the federal path, RMF, FedRAMP and the ATO, where all of this evidence ends up.

Check yourself

Key terms

• Policy as code: writing security rules as code that tools check automatically.
• Shift left: moving checks earlier, to the code and the pull request, where fixes are cheapest.
• Checkov: an open-source scanner for Terraform, Kubernetes files and more.
• False positive: a check that fails on something that isn't actually a problem.
• CIS Kubernetes Benchmark: the community's hardening checks for Kubernetes clusters.
• kube-bench: a tool that runs the CIS Kubernetes Benchmark checks on a real cluster.
• Admission control: the step where Kubernetes decides whether to accept a new or changed resource.
• Kyverno: a policy engine that checks resources at admission, and can enforce or audit.
• Enforce and Audit: refusing what breaks a policy, or allowing it and recording it.

Go deeper

• Checkov's documentation (opens in a new tab), including how to write your own checks.
• kube-bench on GitHub (opens in a new tab), with notes on running it on managed clusters.
• Kyverno's policy library (opens in a new tab), hundreds of ready-made policies to read and adapt.