Firewalls and security groups

Where you are

Lesson 7 built a cloud network with a public subnet and a route to the internet. But a route only says where traffic can go. On its own, it would carry anything anyone sends: a visitor asking for your website, and a stranger trying passwords on your server's remote login, alike.

You've already met one accidental protection. Lesson 6's NAT drops anything nobody inside asked for. But servers in a public subnet have no NAT in front of them, and even inside a private network you don't want every machine able to talk to every other. That's what firewalls are for.

A guard with a list

A building with a security guard at the door doesn't let in everyone who turns up. The guard has a list: delivery drivers at the loading bay, staff with a badge at the main door, visitors only if someone's expecting them. Anyone not on the list doesn't get in.

A firewall is that guard for network traffic. It sits between a network, or a single machine, and everything else, and checks each connection against a list of rules. Each rule describes traffic by up to four things:

• Protocol: TCP or UDP from Lesson 4, or ICMP from Lesson 6.
• Port: which door, like 443 for HTTPS or 22 for SSH (Secure Shell).
• Source: who it's from, as a CIDR range from Lesson 3, like 203.0.113.45/32 for one address or 0.0.0.0/0 for anyone.
• Direction: inbound, arriving at your servers, or outbound, leaving them. Cloud consoles often call these ingress and egress.

So a rule reads like a line on the guard's list: "TCP, port 443, from anyone, inbound: allow".

Default deny

Good firewalls start from the safe end: everything is blocked unless a rule allows it. That's called default deny, and it means your rules are an allow list: a short list of what's welcome, rather than a long, never-finished list of what's banned. If you forget a rule, something doesn't work, and you notice. With a ban list, if you forget a rule, something's exposed, and you might not notice until someone else does.

This lesson's rule of thumb follows from it: allow the smallest thing that works. The right port, from the narrowest source, and nothing else.

Stateful: replies are let back in

When your server calls out to an API, the reply comes back in. Does the firewall need an inbound rule for every reply from every API you'll ever call? With most modern firewalls, no, because they're stateful: they remember connections that were allowed to start, and automatically let their replies through. It's the same trick as the NAT table in Lesson 6.

So in practice, you write inbound rules for conversations that others start with you, and the replies to conversations you start are handled for you.

Drop or refuse

When a rule blocks something, the firewall has a choice. It can refuse it, sending back a "no", or drop it, throwing it away silently and saying nothing. Cloud firewalls almost always drop, so that strangers scanning for open doors learn nothing at all.

That gives you a tool for working out what's wrong, which builds on Lesson 4's three ways to fail:

• Refused, instantly: the machine was reached, and nothing is listening on that port. The firewall let it through.
• Timed out: nothing answered at all. It might be a wrong address or a broken route, or, very often, a firewall dropping it.

Try it from your own computer. nc from Lesson 4 can test whether a port accepts connections without sending anything: -z means just check, -v means say what happened, and -w 3 means give up after 3 seconds.

nc -zv -w 3 api.civicgrid.org 443
nc -zv -w 3 api.civicgrid.org 22

On a Mac you'll see something like:

Connection to api.civicgrid.org port 443 [tcp/https] succeeded!
nc: connectx to api.civicgrid.org port 22 (tcp) failed: Operation timed out

The exact wording differs on Linux. Port 443 is open to everyone, as it has to be for a public API. Port 22 just goes silent: no "refused", no answer at all. Nothing in front of the API accepts remote logins from the internet, and it doesn't tell you so.

Security groups

In AWS, the firewall you'll use most is the security group. It's attached to each server, database or load balancer, not to the subnet, so two servers in the same subnet can have different rules. A security group:

• has allow rules only: there's no "deny" rule, because anything not allowed is already denied
• is stateful, so replies are allowed automatically
• starts out, when you create one, with no inbound rules and all outbound allowed

Their best feature is that a rule's source can be another security group instead of an address range. "Allow port 5432 from the app servers' group" means "from any server in that group", whatever its address is today. Add a server to the group, and it's allowed; remove it, and it isn't. Addresses can change; the group stays true.

Here's a typical setup for a website with a database, built on Lesson 7's VPC. Each layer only accepts traffic from the layer in front of it:

Nothing on the internet can reach the app or the database directly, even if someone found their addresses. And notice what's missing: no SSH from anywhere. If you do need to log in to servers, allow port 22 only from your own address, as a /32, never from 0.0.0.0/0. Better still, AWS and other clouds offer ways to reach a server's shell through their own consoles, with no open port at all.

Network ACLs: a second line, at the subnet

AWS has a second, older kind of firewall, the network ACL (access control list), which sits at the edge of a subnet rather than on each server. It works differently in two ways that catch people out: its rules are numbered and checked in order, with both "allow" and "deny" rules, and it's stateless, so replies are not let back in automatically. You have to allow them yourself, in the other direction.

Most teams leave network ACLs at their default, which allows everything, and do their real work in security groups. It's worth knowing they exist, because when something is blocked and every security group looks right, a network ACL is the next place to look.

Firewalls everywhere

Security groups are one kind among many, and the idea is the same everywhere:

• Other clouds: Google Cloud has VPC firewall rules, Azure has network security groups, and DigitalOcean has cloud firewalls.
• On a server itself: Linux has its own firewall, often managed with ufw (uncomplicated firewall) on Ubuntu. Your Mac has one too, in System Settings, under Network.
• Inside Kubernetes: network policies decide which pods can talk to which, which comes up in Lesson 11.

Using more than one at once, like a cloud firewall and a server's own firewall, is common and sensible: if one is set up wrong, the other still stands. It's also why "check the firewall" can mean checking several places.

Build a rule checker

Firewall rules are just matching: does this connection's port match, and is its source inside the rule's range? That's ipaddress from Lesson 3. Create firewall.py:

import sys
import ipaddress

rules = [
    {"name": "web from anywhere", "port": 443, "source": "0.0.0.0/0"},
    {"name": "ssh from the office", "port": 22, "source": "198.51.100.7/32"},
    {"name": "database from app subnet a", "port": 5432, "source": "10.0.10.0/24"},
]

source = ipaddress.ip_address(sys.argv[1])
port = int(sys.argv[2])
matched = None

for rule in rules:
    if port == rule["port"] and source in ipaddress.ip_network(rule["source"]):
        matched = rule
        break

if matched:
    print(f"ALLOW {source} to port {port}, by rule: {matched['name']}")
else:
    print(f"DROP  {source} to port {port}, no rule allows it")

The rules are a list of dictionaries, one per rule. The loop checks each one, and break stops at the first rule that matches. If none does, matched is still None, and the connection is dropped: default deny, in one line. Try some connections, with an address and a port:

python3 firewall.py 203.0.113.45 443
python3 firewall.py 203.0.113.45 22
python3 firewall.py 198.51.100.7 22
python3 firewall.py 10.0.10.25 5432

ALLOW 203.0.113.45 to port 443, by rule: web from anywhere
DROP  203.0.113.45 to port 22, no rule allows it
ALLOW 198.51.100.7 to port 22, by rule: ssh from the office
ALLOW 10.0.10.25 to port 5432, by rule: database from app subnet a

Anyone can reach the website. SSH is allowed only from the office's one address, so the same visitor who reached port 443 is dropped on port 22. And an app server in 10.0.10.0/24 can reach the database. Real firewalls do the same checks, with more detail, millions of times a second.

The same thing, in Terraform

Here's the database rule from the table above as Terraform, to read rather than run. The source is the app's security group, not an address range:

resource "aws_security_group" "database" {
  name   = "database"
  vpc_id = aws_vpc.main.id

  ingress {
    description     = "Postgres from the app servers"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.app.id]
  }
}

ingress is an inbound rule. from_port and to_port are a range of ports, here just the one. security_groups is where you'd otherwise put cidr_blocks = ["10.0.10.0/24"], and the next section shows why the group is the better choice. With no egress block, outbound traffic is open by default for security groups made in the AWS console, but Terraform removes that default, so a real configuration adds an egress rule too when the database needs to reach out.

Break it on purpose

Three cases. Work out the cause before you read the explanation.

SSH that worked yesterday. Your server's security group allows port 22 from your home address, as a /32, the way this lesson recommends. Yesterday you logged in fine. Today ssh just hangs, then times out. Nothing changed on the server. What happened? Your home's public address changed: most internet providers can change it, after a router restart for example. Run curl -s https://ifconfig.me && echo from Lesson 2, and you'll likely find a new address that the rule doesn't match, so the firewall drops you, silently, as always. Timeout plus a rule based on your address is a big hint. Update the rule to your new /32, and consider a console-based login that doesn't depend on your address at all.

The app that lost its database. The app servers ran in subnet 10.0.10.0/24. To survive a zone failure, as in Lesson 7, you add copies in zone b, in 10.0.11.0/24. The new copies can't reach the database. Check one with the rule checker:

python3 firewall.py 10.0.11.25 5432

DROP  10.0.11.25 to port 5432, no rule allows it

The database rule allows 10.0.10.0/24, and the new servers are in a different range. Nothing is wrong with them; the rule was written for where the servers were, not for what they are. This is exactly why the security-group-as-source rule is better: "from the app group" would have covered the new servers the moment they joined the group. Fix the rule that way, not by adding another range.

It works, so it must be fine. A teammate can't get their laptop to connect to the database, so they add a rule: TCP 5432 from 0.0.0.0/0. Now it works. Is this fine? Try it in the checker by adding that rule to the list, then checking any stranger's address:

python3 firewall.py 203.0.113.45 5432

Before the change it's DROP; after, it's ALLOW, for every address on the internet. Automated programs scan every public IPv4 address all day long, looking for open database ports, so an open port can be found and attacked surprisingly fast. Nothing about the database looks broken, which is what makes this one dangerous. The fix is to allow only the teammate's own address as a /32, or better, have them connect through a VPN or a bastion server instead. When a firewall rule says 0.0.0.0/0, it should be on purpose, and it should almost always be port 443.

Where you are now

You can read and write firewall rules, explain default deny and stateful filtering, tell a drop from a refusal, and design security groups so each layer only trusts the one in front. Your traffic now reaches the right place, and only the traffic you meant to allow. What it isn't yet is private: anyone in the middle could read it. Lesson 9 is TLS and HTTPS, how traffic is encrypted, and what that *.civicgrid.org certificate from Lesson 1 actually proves.

Check yourself

Key terms

• Firewall: something that checks connections against rules, and blocks those not allowed.
• Rule: a description of traffic by protocol, port, source and direction, and whether to allow it.
• Inbound and outbound (ingress and egress): traffic arriving at, and leaving, your servers.
• Default deny: block everything unless a rule allows it.
• Stateful: remembers allowed connections, and lets their replies back in automatically.
• Drop and refuse: blocking silently, which causes a timeout, or with a "no", which is refused.
• Security group: AWS's stateful, allow-only firewall, attached to each resource.
• Network ACL (access control list): AWS's stateless firewall at the edge of a subnet.

Go deeper

• AWS's guide to security groups (opens in a new tab), and how they compare with network ACLs (opens in a new tab).