SSH and keys

By the end you'll know

How SSH proves who you are with a key pair, and how it proves who the server is
How to make a key, and put its public half on a server
How to log in, copy files and give a server a short name, without Vagrant
What SSH's permission checks are, on your computer and on the server
How to switch off password and root logins, and check it really worked

The problem: someone else has been logging you in

Since Lesson 7, vagrant ssh has logged you in to the server. A real cloud server gives you an address and a key, and nothing else. SSH (Secure Shell) is how almost every server in the world is managed, and it's also the front door attackers try first: every server on the internet gets password guesses within minutes of starting. This lesson does SSH by hand, then locks it down.

Two keys, two questions. SSH uses key pairs: two linked files, made together. The private key stays on your computer and is never shared with anyone. The public key can go anywhere. Something locked with one can only be unlocked with the other, which lets SSH answer two questions without sending a secret across the network:

Question	Answered by	Kept in
Is this really the server I meant?	The server's own host key	Your computer remembers it, in `~/.ssh/known_hosts`
Is this really someone allowed in?	Your key pair	The server keeps your public key, in `~/.ssh/authorized_keys` of the user you log in as

What vagrant ssh was doing

On your own computer, in your practice folder:

BASH

cd linux-practice
vagrant ssh-config

OUTPUT

Host default
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/you/linux-practice/.vagrant/machines/default/vmware_desktop/private_key
  IdentitiesOnly yes
  LogLevel FATAL
  PubkeyAcceptedKeyTypes +ssh-rsa
  HostKeyAlgorithms +ssh-rsa

Nothing magic: vagrant ssh is plain ssh with these settings. It connects as vagrant to 127.0.0.1 port 2222, which Vagrant forwards to the server's port 22, using a private key Vagrant made and keeps in the hidden .vagrant folder (the one .gitignore keeps out of Git). Two settings are worth noticing, because you should never use them on a real server: StrictHostKeyChecking no and UserKnownHostsFile /dev/null mean "don't check which server this is, and don't remember". Fine for a throwaway VM that gets a new identity every rebuild; on a real server, they'd switch off half of SSH's protection.

Make your own key

Everything in this lesson runs on your own computer unless it says otherwise. Make a key pair just for this server, with a name of its own, so it can't overwrite a key you already use. The test -e stops if the file already exists:

BASH

test -e ~/.ssh/bookshop_ed25519 && echo "already exists, stopping" || ssh-keygen -t ed25519 -f ~/.ssh/bookshop_ed25519 -C "bookshop practice" -N ""
ls -l ~/.ssh/bookshop_ed25519 ~/.ssh/bookshop_ed25519.pub
cat ~/.ssh/bookshop_ed25519.pub

OUTPUT

Generating public/private ed25519 key pair.
Your identification has been saved in /Users/you/.ssh/bookshop_ed25519
Your public key has been saved in /Users/you/.ssh/bookshop_ed25519.pub
The key fingerprint is:
SHA256:6eEl/RZPItwRPmFFjC7Mab96dVRqcTk4fIL8hiYKZtM bookshop practice
The key's randomart image is:
+--[ED25519 256]--+
|          . *=+ .|
|           =.Bo+o|
|      .  o o* ++o|
|     = E =*+.=o .|
|    o o S.Bo=... |
|       + + o.=. .|
|        o   oo.. |
|           .o    |
|          .o     |
+----[SHA256]-----+
-rw-------  1 you  staff  411 Oct 30 10:40 /Users/you/.ssh/bookshop_ed25519
-rw-r--r--  1 you  staff   99 Oct 30 10:40 /Users/you/.ssh/bookshop_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKtT73fqtSBYIS+fuQbwB9nRgmCbTmOqT/0Es1uN4n1k bookshop practice

-t ed25519 is the kind of key: modern, short and strong. You'll still see rsa keys, the older kind.
-f names the files: the private key, and the same name ending in .pub for the public key.
-C is a comment, to tell keys apart. It ends up at the end of the public key.
-N "" sets an empty passphrase, so the key works without typing anything. That's for practice only. A real key should have a passphrase, which encrypts the private key file, so a stolen copy is useless on its own. Your computer's SSH agent remembers it once per session, so you don't type it every time.

Look at the permissions: the private key is -rw------- (600), readable by you alone, and the public key is -rw-r--r-- (644), since anyone may see it. The fingerprint is a short summary of the key, the easy way to compare keys by eye. The public key is safe to show, as here; never share the private key, in a chat, a ticket or a Git repository. Anyone who has it is you.

Give the server your public key

The server needs a user to log in as, and your public key in that user's ~/.ssh/authorized_keys. Sam, from Lesson 5, gets an account with admin rights (the sudo group). There's no way in except vagrant ssh yet, so use it to run the commands for you; -c runs one command on the server and comes straight back:

BASH

vagrant ssh -c 'sudo useradd --create-home --shell /bin/bash --groups sudo sam && sudo install -d -m 700 -o sam -g sam /home/sam/.ssh'
cat ~/.ssh/bookshop_ed25519.pub | vagrant ssh -c 'sudo tee /home/sam/.ssh/authorized_keys > /dev/null'
vagrant ssh -c 'sudo chown sam:sam /home/sam/.ssh/authorized_keys && sudo chmod 600 /home/sam/.ssh/authorized_keys && sudo ls -la /home/sam/.ssh && sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub'

OUTPUT

total 12
drwx------ 2 sam sam 4096 Oct 30 14:40 .
drwxr-x--- 3 sam sam 4096 Oct 30 14:40 ..
-rw------- 1 sam sam   99 Oct 30 14:40 authorized_keys
256 SHA256:IuwW2RLSQSodKvu+ltnSnNarLcslQAT8LLJei8g3hI0 root@vagrant (ED25519)

install -d makes a folder with its owner and permissions set in one go. The second line pipes the public key from your computer into tee on the server (Lessons 4 and 7): a pipe works across SSH, too. And the permissions matter, as you'll see in "Break it on purpose": .ssh is 700 and authorized_keys 600, both owned by Sam. The last command printed the server's host key fingerprint. Keep it on screen.

On a cloud server, you'd usually skip all this: you give the cloud provider your public key when you create the server, and it puts it in place for you.

Log in

Straight to the server's address, port 22, the way you'd reach any server. -i says which private key to use:

BASH

ssh -i ~/.ssh/bookshop_ed25519 sam@192.168.34.130 'whoami; hostname; id'

OUTPUT

The authenticity of host '192.168.34.130 (192.168.34.130)' can't be established.
ED25519 key fingerprint is: SHA256:IuwW2RLSQSodKvu+ltnSnNarLcslQAT8LLJei8g3hI0
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.34.130' (ED25519) to the list of known hosts.
sam
bookshop
uid=1001(sam) gid=1001(sam) groups=1001(sam),27(sudo)

The first connection asks you to trust the server. Your computer has never seen it, so it shows the server's host key fingerprint and asks. Compare it with the one the server printed: SHA256:IuwW2...hI0 on both. Only then type yes. If someone were pretending to be the server, the fingerprints wouldn't match. Your computer saves it in ~/.ssh/known_hosts, and checks it on every connection from now on. If it ever changes, SSH stops with a loud WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! That's either a rebuilt server, or someone in the middle: find out which before you connect.

Then Sam's commands ran, and came back: logged in as sam, on bookshop, in the sudo group. No password was asked for, because the key answered "is this someone allowed in?".

A short name: ~/.ssh/config

Typing the address, the user and the key every time gets old. ~/.ssh/config on your computer gives a server a name. Add to the end of it; >> never replaces what's already there:

BASH

cat >> ~/.ssh/config <<'EOF'

Host bookshop
    HostName 192.168.34.130
    User sam
    IdentityFile ~/.ssh/bookshop_ed25519
    IdentitiesOnly yes
EOF
ssh bookshop 'whoami; hostname'

OUTPUT

sam
bookshop

IdentitiesOnly yes means "only try this key". Without it, SSH offers every key you have, one after another, and servers often give up after a few. Every SSH tool reads this file, including scp (secure copy), which copies files over SSH. bookshop: with nothing after the colon means Sam's home folder:

BASH

echo "a note from the Mac" > /tmp/note.txt
scp /tmp/note.txt bookshop:
ssh bookshop 'cat note.txt'

OUTPUT

note.txt                                                                                                                 100%   20    69.5KB/s   00:00    
a note from the Mac

Break it on purpose: the permission checks

SSH refuses to use keys that might have been tampered with, on both ends. Make the private key readable by everyone:

BASH

chmod 644 ~/.ssh/bookshop_ed25519
ssh bookshop 'whoami'

OUTPUT

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/Users/you/.ssh/bookshop_ed25519' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/Users/you/.ssh/bookshop_ed25519": bad permissions
sam@192.168.34.130's password:

Your own computer refused to use the key, and then something worse: it fell back to asking for Sam's password. Press Ctrl+C here. While preparing this lesson, the lines pasted after this command were typed into that password prompt, one after another, as password guesses. Don't paste a block of commands that contains one which might prompt: everything after it goes into the prompt. ssh -o BatchMode=yes never prompts, so it's the safe way to test. Fix the key:

BASH

chmod 600 ~/.ssh/bookshop_ed25519
ssh bookshop 'whoami'

OUTPUT

sam

The server checks too. Make Sam's home folder writable by anyone, and try again, without prompts:

BASH

vagrant ssh -c 'sudo chmod 777 /home/sam'
ssh -o BatchMode=yes bookshop 'whoami'

OUTPUT

sam@192.168.34.130: Permission denied (publickey,password).

Denied, and your computer isn't told why. The reason is in the server's log:

BASH

vagrant ssh -c 'sudo journalctl -u ssh --no-pager -n 20 | grep -i -E "refused|bad ownership" | tail -2; sudo chmod 750 /home/sam'
ssh bookshop 'whoami'

OUTPUT

Oct 30 14:46:34 bookshop sshd[10892]: Authentication refused: bad ownership or modes for directory /home/sam
sam

bad ownership or modes: if anyone could write to Sam's home folder, anyone could have replaced Sam's authorized_keys, so the server won't trust it. The same check covers .ssh and authorized_keys themselves. The command fixed the folder back to 750, and Sam's in again. When a key login fails and the key looks right, it's almost always permissions or ownership on the server, and the server's SSH log says so. Notice (publickey,password) in the denial, too: the server still offers password logins. That's next.

Try the combinations:

A sudo setting worth checking

Sam is in the sudo group, and has no password at all yet. So Sam can't use sudo... except that this server lets Sam do it anyway:

BASH

ssh bookshop 'sudo -n whoami'
vagrant ssh -c 'sudo grep "^sam:" /etc/shadow | cut -d: -f1,2'

OUTPUT

root
sam:!

sudo -n means "never ask for a password; fail instead", and it didn't fail: Sam became root. sam:! in /etc/shadow (Lesson 5) means Sam has no usable password at all. So Sam's SSH key alone is enough to become root. Why? Ask sudo what Sam may do, and read its rules, with comments and blank lines left out:

BASH

vagrant ssh -c 'sudo -l -U sam'
vagrant ssh -c 'sudo grep -rn -v "^#" /etc/sudoers /etc/sudoers.d/ | grep -v -E ":[0-9]+:$"'

OUTPUT

Matching Defaults entries for sam on bookshop:
    env_reset, exempt_group=sudo, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin\:/snap/bin, use_pty

User sam may run the following commands on bookshop:
    (ALL : ALL) ALL
/etc/sudoers:9:Defaults	env_reset
/etc/sudoers:10:Defaults	exempt_group=sudo
/etc/sudoers:11:Defaults	mail_badpass
/etc/sudoers:12:Defaults	secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
/etc/sudoers:16:Defaults	use_pty
/etc/sudoers:48:root	ALL=(ALL:ALL) ALL
/etc/sudoers:51:%admin ALL=(ALL) ALL
/etc/sudoers:54:%sudo	ALL=(ALL:ALL) ALL
/etc/sudoers:58:@includedir /etc/sudoers.d
/etc/sudoers.d/vagrant:1:vagrant ALL=(ALL) NOPASSWD:ALL
/etc/sudoers.d/99_vagrant:1:vagrant ALL=(ALL) NOPASSWD:ALL

Line 54, %sudo ALL=(ALL:ALL) ALL, is Ubuntu's normal rule: the sudo group may run anything, with a password. But line 10, Defaults exempt_group=sudo, says the sudo group never needs one. Ubuntu doesn't ship that line; this Vagrant box adds it for convenience. On a practice machine, harmless. On a real server, it means one stolen key gives away the whole machine. (The two vagrant lines give Vagrant's own user password-free root, which Vagrant needs, and keep.)

Fix it safely. A mistake in sudo's rules can take sudo away from everyone, including you, so never edit them directly. Write the change to a temporary file, check it with visudo -cf, and only then install it:

BASH

vagrant ssh -c 'echo "Defaults !exempt_group" > /tmp/10-bookshop && sudo visudo -cf /tmp/10-bookshop && sudo install -m 440 -o root -g root /tmp/10-bookshop /etc/sudoers.d/10-bookshop && sudo visudo -c'
ssh bookshop 'sudo -n whoami'

OUTPUT

/tmp/10-bookshop: parsed OK
/etc/sudoers: parsed OK
/etc/sudoers.d/10-bookshop: parsed OK
/etc/sudoers.d/99_vagrant: parsed OK
/etc/sudoers.d/README: parsed OK
/etc/sudoers.d/vagrant: bad permissions, should be mode 0440
sudo: a password is required

!exempt_group switches the exemption off. visudo -c checked every sudo file, and found one more thing: the box's /etc/sudoers.d/vagrant should be readable only by root, mode 440, and isn't. Fix it with vagrant ssh -c 'sudo chmod 440 /etc/sudoers.d/vagrant && sudo visudo -c'. And now sudo -n fails: a password is required. Give Sam one. passwd asks you to type it, so log in properly first, with vagrant ssh, then run, on the server:

BASH

sudo passwd sam

Then exit, and on your own computer:

BASH

ssh -t bookshop 'sudo whoami'

OUTPUT

[sudo] password for sam: 
root
Connection to 192.168.34.130 closed.

-t gives SSH a proper terminal, which sudo needs to ask for a password. Now it takes both: Sam's key to log in, and Sam's password to become root. Someone with only one of them gets nowhere near root.

Lock SSH down

Two settings close the doors attackers try most: password logins, which they guess at all day, and root logins, the one user that exists on every server. First, test what the server accepts now, asking only for a password and without waiting at a prompt:

BASH

ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password -o NumberOfPasswordPrompts=0 vagrant@192.168.34.130

OUTPUT

vagrant@192.168.34.130: Permission denied (publickey,password).

The list in brackets is what the server would accept: publickey, password. Passwords are on. Where's that set? SSH's server is sshd, and its settings are /etc/ssh/sshd_config, plus any files in /etc/ssh/sshd_config.d/:

BASH

ssh bookshop 'ls /etc/ssh/sshd_config.d/'
ssh -t bookshop 'sudo grep -rn -i -E "PasswordAuthentication|PermitRootLogin" /etc/ssh/sshd_config.d/ /etc/ssh/sshd_config | grep -v "#"'

OUTPUT

50-cloud-init.conf
[sudo] password for sam: 
/etc/ssh/sshd_config.d/50-cloud-init.conf:1:PasswordAuthentication yes

50-cloud-init.conf, from the tool that set the server up, switches passwords on. For sshd, the first value it reads wins, and it reads the .d files in name order. So a file starting with a lower number, 10-, is read first, and wins. Write it, check the settings with sshd -t, reload, and ask sshd -T what it's really using:

BASH

ssh -t bookshop 'printf "PasswordAuthentication no\nPermitRootLogin no\n" | sudo tee /etc/ssh/sshd_config.d/10-bookshop.conf && sudo sshd -t && sudo systemctl reload ssh && sudo sshd -T | grep -E "^(passwordauthentication|permitrootlogin)"'

OUTPUT

[sudo] password for sam: 
Sorry, try again.
[sudo] password for sam: 
PasswordAuthentication no
PermitRootLogin no
permitrootlogin no
passwordauthentication no
Connection to 192.168.34.130 closed.

(Sorry, try again was just a mistyped password.) sshd -t checks the settings and prints nothing when they're fine; && means the reload only happens if they are. sshd -T prints the settings in effect, and it's the only way to be sure which file won. reload, not restart: sessions already open stay open. Test again:

BASH

ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password -o NumberOfPasswordPrompts=0 vagrant@192.168.34.130

OUTPUT

vagrant@192.168.34.130: Permission denied (publickey).

(publickey) alone. The server no longer even offers to take a password. vagrant ssh and ssh bookshop still work, because both use keys. The server's log shows who got in, and how:

BASH

ssh -t bookshop 'sudo journalctl -u ssh --no-pager -n 30 | grep -E "Accepted|Failed|Connection closed" | tail -4'

OUTPUT

Oct 30 14:52:50 bookshop sshd[11470]: Accepted publickey for sam from 192.168.34.1 port 60775 ssh2: ED25519 SHA256:6eEl/RZPItwRPmFFjC7Mab96dVRqcTk4fIL8hiYKZtM
Oct 30 14:53:21 bookshop sshd[11522]: Accepted publickey for sam from 192.168.34.1 port 60841 ssh2: ED25519 SHA256:6eEl/RZPItwRPmFFjC7Mab96dVRqcTk4fIL8hiYKZtM
Oct 30 14:53:47 bookshop sshd[11588]: Connection closed by authenticating user vagrant 192.168.34.1 port 60898 [preauth]
Oct 30 14:53:57 bookshop sshd[11591]: Accepted publickey for sam from 192.168.34.1 port 60919 ssh2: ED25519 SHA256:6eEl/RZPItwRPmFFjC7Mab96dVRqcTk4fIL8hiYKZtM

Every login by Sam, with the fingerprint of the exact key used: the 6eEl/... your ssh-keygen printed. On a server several people use, that's how you know whose key it was. The [preauth] line is the password test, turned away before it got anywhere.

On a real server

Keep a second session open while you change SSH settings. Change, reload, then test from a new connection. If the new one fails, the old one is still in, to fix it. Close it only when the new one works.
Watch who's trying. sudo journalctl -u ssh | grep "Invalid user" | tail on any server facing the internet shows the guessing. With passwords off, it's noise. Tools like fail2ban block addresses that keep trying.
One key per person, per computer. When someone leaves, or a laptop is lost, delete that one line from authorized_keys. Shared keys can't be taken back from one person.
Limit who may log in at all with AllowUsers sam in the same settings file.
Many clouds avoid open SSH altogether, with tools like AWS Systems Manager, which reach the server without port 22 being open to the internet. The AWS track covers it.

Spot the problem

Work out what's wrong before you read the explanation.

The key that wouldn't work. A new team member's public key is added to the server as root: sudo nano /home/ana/.ssh/authorized_keys. Her login fails with Permission denied (publickey), though the key is definitely right. What do you check? Who owns the file. Made by root, it belongs to root, and Ana's SSH won't trust an authorized_keys she doesn't own. sudo journalctl -u ssh says bad ownership or modes; sudo chown -R ana:ana /home/ana/.ssh fixes it. Lesson 5's rule again: a new file belongs to whoever made it.

Keep this

To	Run
Make a key pair, with a passphrase	`ssh-keygen -t ed25519 -f ~/.ssh/NAME -C "comment"`
A key's fingerprint	`ssh-keygen -lf ~/.ssh/NAME.pub`
A server's host key fingerprint (on the server)	`sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub`
Log in with a key	`ssh -i ~/.ssh/NAME user@HOST`
Run one command, and come back	`ssh HOST 'command'`
A command that needs a terminal, like sudo	`ssh -t HOST 'sudo command'`
Never prompt, just fail	`ssh -o BatchMode=yes HOST`
Copy a file to a server	`scp FILE HOST:`
A short name for a server	A `Host` block in `~/.ssh/config`
Permissions SSH expects	Private key 600, `~/.ssh` 700, `authorized_keys` 600, home folder not writable by others
Why a login failed (on the server)	`sudo journalctl -u ssh -n 50`
What a user may do with sudo	`sudo -l -U NAME`
Check sudo's rules	`sudo visudo -c`, a new file first: `sudo visudo -cf FILE`
Keys only, no root	`PasswordAuthentication no`, `PermitRootLogin no` in `/etc/ssh/sshd_config.d/10-NAME.conf`
Check, apply, and confirm SSH's settings	`sudo sshd -t`, `sudo systemctl reload ssh`, `sudo sshd -T`

Where you are now

You can reach the server like any real one: your own key, a checked host key, a short name, files copied over SSH. And it's locked down: keys only, no root logins, and sudo needs a password again. Lesson 12 goes back to the shell itself: loops, conditions and one-liners, for doing the same thing to many files or servers at once, and waiting for something to happen.

Check yourself

Which part of a key pair goes on the server?
The first time you connect, SSH shows a fingerprint and asks yes or no. What should you do?
SSH says UNPROTECTED PRIVATE KEY FILE. What's the fix?
A key login fails with Permission denied, and the key is right. Where is the reason?
After the lock-down, a password-only login says Permission denied (publickey). What does the bracket mean?
Why was the new SSH settings file named 10-bookshop.conf?

Key terms

SSH: Secure Shell, the encrypted way to log in to and run commands on another machine.
Key pair: a private key you keep, and a public key you share, made together.
authorized_keys: a user's list, on the server, of public keys allowed to log in as them.
Host key and known_hosts: the server's own key pair, and your computer's record of the servers it has met.
Fingerprint: a short summary of a key, for comparing keys by eye.
Passphrase: a password that encrypts a private key file.
sshd: the SSH server program, set up in /etc/ssh/sshd_config and sshd_config.d.

Go deeper

cat ~/.ssh/known_hosts | tail -2 on your computer: the bookshop's host key, saved the moment you typed yes.
ssh -v bookshop 'true' shows every step of a connection, including which keys were offered and which was accepted.
Make a key with a passphrase, ssh-keygen -t ed25519 -f ~/.ssh/bookshop2, and see ssh-add remember it.
Lesson 5, Permissions and users, for the permissions and ownership SSH checks.