Files and data

Where your checker is

At the end of Lesson 4, your checker has a function for each job, and one bad value no longer stops the run. But the list of sites is typed into the script itself. To add a site, you have to edit Python code, and anyone else who wants to change the list has to as well. And when the checker finishes, its results are gone; they were only ever printed on screen.

This lesson moves the list of sites into a file anyone can edit, even in a spreadsheet, and saves the results to another file. You'll change checker.py in two steps, and add one new file next to it.

Modules: Python's toolbox

You don't carry every tool you own around the house; they live in a toolbox, and you take out the one you need for the job. Python works the same way. It comes with a large set of ready-made tools, called the standard library, organised into modules: one for CSV files, one for JSON, one for dates, and hundreds more. A script only loads the ones it asks for, with import:

>>> import csv
>>> import json

import csv loads the csv module, and from then on you use its tools by writing the module's name, a dot, and the tool's name, like csv.DictReader. Imports go at the top of a script, so anyone reading it can see straight away which tools it uses.

The standard library is already on your computer; it came with Python. Lesson 6 shows how to install modules other people have written, which don't.

Reading a CSV file

If you've ever chosen Save As in a spreadsheet and picked CSV, you've made one of these. CSV (comma-separated values) is a plain text file with one row per line, and commas between the columns. The first line is usually the column names, called the header.

In your site-checker folder, next to checker.py, create a file called sites.csv in your editor, and put this in it:

name,status_code,response_seconds
example.com,200,0.25
example.org,503,1.8
example.net,301,0.4
example.edu,n/a,0

The status codes and times are still made up, the same as before. In Lesson 6, the checker gets them from real websites, and those two columns go. The point for now is that the list lives in a file: open it in a spreadsheet, add a row, save it, and the checker picks it up without anyone touching Python.

To read a file, Python first has to open it. Borrowing a library book works best when the library makes sure it comes back, even if you forget. with does that for files: it opens the file, runs the indented lines, and closes it again afterwards, even if something goes wrong in the middle. Try it at the prompt, from inside the site-checker folder:

>>> with open("sites.csv", newline="") as file:
...     rows = list(csv.DictReader(file))
...
>>> rows[0]
{'name': 'example.com', 'status_code': '200', 'response_seconds': '0.25'}
>>> len(rows)
4

(After the second line, press Enter on the empty ... line to finish the with block.) Here's what each piece does:

• open("sites.csv", newline="") opens the file. The newline="" part is what the csv module's documentation asks for: it lets the module handle line endings itself, so rows are never split wrongly.
• as file gives the open file a name to use inside the block.
• csv.DictReader(file) reads the rows, using the header line as the keys. Each row comes back as a dictionary, exactly the shape the checker already uses.
• list() collects all the rows into a list, so you can count them and loop over them after the file is closed.

Look closely at rows[0]: every value is in quotes, even '200'. Anything read from a file is text, which is exactly why Lesson 4's check_site() converts with int().

Now the first change to the checker. In checker.py:

1. Add import csv on the line under the comment at the top.
2. Delete the whole sites = [ ... ] list. The file holds it now.
3. Add this function below to_ms():

def load_sites(path):
    with open(path, newline="") as file:
        return list(csv.DictReader(file))

4. Add this line just above print("Checking sites..."), so the sites are loaded before the checker uses them:

sites = load_sites("sites.csv")

5. The times come from the file as text too, so check_site() needs to convert them. Change its try block, and the line just after it, to this:

    try:
        status_code = int(site["status_code"])
        response_seconds = float(site["response_seconds"])
    except ValueError:
        print(f"SKIP  {name}: status code or time isn't a number")
        return "skipped"

    response_ms = to_ms(response_seconds)

float() turns the text into a decimal, the way int() does for whole numbers. Both conversions are inside the try, so a bad value in either column skips the site instead of crashing. Run the checker:

python3 checker.py

You'll see the same results as in Lesson 4. The only difference is the SKIP message, which now covers both columns. Now add a row to sites.csv, such as example.io,200,0.3, save it, and run the checker again: five sites, and you didn't touch the code.

Saving results as JSON

When your weather app asks a server for the forecast, the answer usually comes back as JSON (JavaScript Object Notation). It's the most common format for programs passing data to each other, and it looks almost exactly like Python's lists and dictionaries:

>>> json.dumps({"name": "example.com", "result": "up"})
'{"name": "example.com", "result": "up"}'

json.dumps() turns a Python value into JSON text. To write it straight into a file instead, use json.dump(), without the s, and give it an open file. That's the second change:

1. Add import json under import csv.
2. Add this function below check_site():

def save_results(path, results):
    with open(path, "w") as file:
        json.dump(results, file, indent=2)

The "w" means open the file for writing. If the file doesn't exist, Python creates it. If it does, "w" wipes it and starts again, the same as saving over a document, so each run replaces the last run's results. indent=2 spreads the JSON over several lines, so a person can read it.

3. Collect the results as the loop runs. Above the loop, next to down = 0 and skipped = 0, add:

results = []

and inside the loop, straight after result = check_site(site), add:

    results.append({"name": site["name"], "result": result})

4. After the loop, just above the summary print(), save them:

save_results("results.json", results)

and at the very end, add a line to say where they went:

print("Results saved to results.json")

Checkpoint: your checker so far

Your checker.py should now look like this. If your blank lines differ a little, that's fine, but the line numbers in the break-it section assume this layout:

# A website checker, built one piece at a time.
import csv
import json


def is_up(status_code):
    return status_code >= 200 and status_code < 400


def to_ms(seconds):
    return round(seconds * 1000)


def load_sites(path):
    with open(path, newline="") as file:
        return list(csv.DictReader(file))


def check_site(site):
    name = site["name"]
    try:
        status_code = int(site["status_code"])
        response_seconds = float(site["response_seconds"])
    except ValueError:
        print(f"SKIP  {name}: status code or time isn't a number")
        return "skipped"

    response_ms = to_ms(response_seconds)
    if is_up(status_code):
        print(f"UP    {name}: {status_code} in {response_ms} ms")
        return "up"
    print(f"DOWN  {name}: {status_code} in {response_ms} ms")
    return "down"


def save_results(path, results):
    with open(path, "w") as file:
        json.dump(results, file, indent=2)


sites = load_sites("sites.csv")
print("Checking sites...")
print(f"{len(sites)} sites to check")

down = 0
skipped = 0
results = []
for site in sites:
    result = check_site(site)
    results.append({"name": site["name"], "result": result})
    if result == "down":
        down = down + 1
    elif result == "skipped":
        skipped = skipped + 1

save_results("results.json", results)
print(f"{down} of {len(sites)} sites down, {skipped} skipped")
print("Results saved to results.json")

If you added example.io to sites.csv earlier, take it back out, so your output matches. Then run it:

python3 checker.py

Checking sites...
4 sites to check
UP    example.com: 200 in 250 ms
DOWN  example.org: 503 in 1800 ms
UP    example.net: 301 in 400 ms
SKIP  example.edu: status code or time isn't a number
1 of 4 sites down, 1 skipped
Results saved to results.json

A new file, results.json, has appeared next to checker.py. Open it in your editor:

[
  {
    "name": "example.com",
    "result": "up"
  },
  {
    "name": "example.org",
    "result": "down"
  },
  {
    "name": "example.net",
    "result": "up"
  },
  {
    "name": "example.edu",
    "result": "skipped"
  }
]

That's the results, saved where another program, or another run of this one, can read them later. Notice that the checker itself got simpler to change: the list of sites is in one file, the results are in another, and the Python in between is the same whatever is in them.

Break it on purpose

Make each mistake, run the checker, and put it back before the next.

Run it from the wrong folder. Go up one folder, out of site-checker, and run the checker from there:

cd ..
python3 site-checker/checker.py

Traceback (most recent call last):
  File "site-checker/checker.py", line 41, in <module>
    sites = load_sites("sites.csv")
  File "site-checker/checker.py", line 15, in load_sites
    with open(path, newline="") as file:
         ~~~~^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'sites.csv'

sites.csv is right there next to checker.py, so why can't Python find it? Because a file name on its own, like "sites.csv", is looked up in the folder you ran the script from, not the folder the script is in. You ran it from the folder above, and there's no sites.csv there. This catches almost everyone eventually, and it'll matter in Lesson 8, when the checker runs on a schedule from a folder you didn't choose. For now, go back with cd site-checker.

Rename a column. In sites.csv, change the header's status_code to status, save, and run the checker:

Checking sites...
4 sites to check
Traceback (most recent call last):
  File "checker.py", line 49, in <module>
    result = check_site(site)
  File "checker.py", line 22, in check_site
    status_code = int(site["status_code"])
                      ~~~~^^^^^^^^^^^^^^^
KeyError: 'status_code'

The KeyError from Lesson 3, coming from a file this time. DictReader made the keys from the header line, so the dictionary now has a status key and no status_code key. The code didn't change, but the data it depends on did. Whoever edits the file needs to know which column names the script expects. Change the header back.

Forget the import. Delete the import csv line, and run the checker:

Traceback (most recent call last):
  File "checker.py", line 40, in <module>
    sites = load_sites("sites.csv")
  File "checker.py", line 15, in load_sites
    return list(csv.DictReader(file))
                ^^^
NameError: name 'csv' is not defined. Did you forget to import 'csv'?

(The line numbers are one lower than before, because you deleted a line.) Python doesn't know the name csv until you import it, and the message even says so. Put import csv back.

Where you are now

Your checker reads its list of sites from a file, and saves what it found to another. But the status codes and times in that file are still made up. Lesson 6 removes those two columns, and has the checker visit each site over the internet and get the real answers.

Check yourself

Key terms

• Module: a set of ready-made tools you load with import, like csv or json.
• Standard library: the modules that come with Python, already on your computer.
• CSV (comma-separated values): a text file of rows, with commas between the columns. Spreadsheets can save as CSV.
• Header: the first line of a CSV file, holding the column names.
• with open(...) as file: opens a file, and closes it again when the block ends.
• "w" (write mode): opens a file for writing, creating it, or wiping it if it already exists.
• JSON (JavaScript Object Notation): a text format for data that programs swap, shaped like lists and dictionaries.
• FileNotFoundError: Python couldn't find the file, often because the script ran from a different folder.

Go deeper

• Reading and writing files (opens in a new tab) in the official tutorial covers open(), with and JSON.
• The csv module (opens in a new tab) documentation has every option for reading and writing CSV files.