Talking to APIs

Where your checker is

At the end of Lesson 5, your checker reads its sites from sites.csv and saves results to results.json. But the status codes and times in that file are made up. The checker has never actually visited a website.

This lesson fixes that. The made-up columns go, and the checker asks each site for real, over the internet. Along the way you'll install your first package, read data from a real API, and give the checker the retry loop Lesson 3 promised.

Requests and responses

When you type an address into your browser, it sends a request to that website, and the site sends back a response: the page, plus a three-digit status code saying how it went. You rarely see the code, except when it's bad news, like "404 Page not found".

That's the code the checker has been pretending to know since Lesson 2. The common ones:

• 200: OK, here's what you asked for.
• 301 or 302: it's moved; go to this other address. Browsers, and the requests package you're about to install, follow these automatically.
• 404: not found. The site is up, but that page isn't there.
• 500 or 503: the site itself has a problem.

An API (application programming interface) works the same way, but for programs instead of people. When your weather app shows tomorrow's forecast, it sent a request to a weather service's API, and the response came back as JSON, the format you saved results in last lesson. Same requests, same status codes; the answer is just data instead of a page.

Installing a package in a virtual environment

Python's standard library can make web requests, but it's fiddly. Almost everyone uses requests, a package written by other people, which isn't on your computer yet. You install packages with pip, Python's package installer.

First, though, give the project its own space for packages. The same way each child in a family might have their own pencil case, so nobody's pens get mixed up, a virtual environment is a folder of packages that belongs to one project. Packages you install for the checker don't touch any other project, and vice versa.

From inside your site-checker folder, create one:

python3 -m venv .venv

-m venv runs Python's built-in venv module, and .venv is the name of the folder it creates. (The dot at the start makes it a hidden folder, so ls won't show it, but ls -a will.) Now activate it, which tells this terminal to use it.

On macOS and Linux:

source .venv/bin/activate

On Windows:

.venv\Scripts\activate

source runs the activation script inside your current terminal. Your prompt now starts with (.venv), which is how you know it's active. Now install requests:

pip install requests

pip downloads it from PyPI (the Python Package Index), the public library of Python packages, along with the few packages it depends on. The last line should start with Successfully installed.

Activation only lasts for the terminal you ran it in. Open a new terminal tomorrow, and you'll need to cd into the folder and run the source line again. (If you ever want to leave it, type deactivate.) And if you're using Git, add .venv/ to your .gitignore file: the folder can always be rebuilt, so it doesn't belong in the repository.

Your first request

With the virtual environment active, start python3 and try it at the prompt:

>>> import requests
>>> response = requests.get("https://github.com", timeout=5)
>>> response.status_code
200
>>> response.elapsed.total_seconds()
0.097

requests.get() sends a request and gives back a response. response.status_code is the code, already an int, so no converting this time. response.elapsed is how long the site took to answer, and .total_seconds() turns that into a float of seconds, like the times in your CSV file. (Your time will be different; it depends on your connection.) timeout=5 is covered in a moment.

Now an API. GitHub has one, and you can ask it about any public project. Ask about Python itself:

>>> response = requests.get("https://api.github.com/repos/python/cpython", timeout=5)
>>> data = response.json()
>>> data["full_name"]
'python/cpython'
>>> data["description"]
'The Python programming language'
>>> data["stargazers_count"]
77758

response.json() reads the JSON in the response and turns it into Python: here, a dictionary. From there it's the same data["key"] you learned in Lesson 3. (stargazers_count is how many people have starred the project on GitHub, so yours will be higher.) That's what talking to an API looks like, almost every time: send a request, check the status code, read the JSON.

When nobody answers

If you call a shop and it rings and rings, you don't stay on the line forever: after a while you hang up. A timeout does that for a request. timeout=5 means "if the site hasn't answered within 5 seconds, give up". Without one, a site that never answers could leave the checker waiting forever. Always set one.

When a request fails, because the site timed out, the address doesn't exist, or the network dropped, requests raises an exception. They all belong to one family, requests.RequestException, so catching that one name catches them all, exactly like except ValueError: in Lesson 4.

And if the call didn't connect, you'd probably try again once or twice before giving up. That's the while loop from Lesson 3, now for real:

def fetch(url):
    attempts = 0
    while attempts < 3:
        attempts = attempts + 1
        try:
            return requests.get(url, timeout=5)
        except requests.RequestException:
            print(f"      attempt {attempts} failed")
            time.sleep(1)
    return None

Read it from the top. Try up to 3 times. If requests.get() works, return hands back the response, and the function ends right there, loop and all. If it raises, print a note, wait a second with time.sleep(1) (from the standard library's time module), and go round again. If all 3 tries fail, the loop ends, and the function returns None: "no answer".

Change the checker

Three changes, then a checkpoint.

1. The sites file. Open sites.csv, the file you made in Lesson 5, next to checker.py in your site-checker folder. The made-up columns go, so replace everything in it with this:

name,url
github.com,https://github.com
pypi.org,https://pypi.org
missing page,https://github.com/python/no-such-repo
no such site,https://example.invalid

Two real sites; a real site with a page that doesn't exist; and an address that can never work, because .invalid is reserved and will never belong to anyone. That way you'll see every kind of result.

2. Imports and fetch(). At the top of checker.py, make the imports look like this:

import csv
import json
import time

import requests

The blank line is a Python habit: standard library modules first, then installed packages, so readers can tell at a glance what needs installing. Then add the fetch() function from the last section below load_sites().

3. check_site() and the loop. Replace the whole check_site() function with this version:

def check_site(site):
    name = site["name"]
    response = fetch(site["url"])
    if response is None:
        print(f"ERROR {name}: no answer after 3 tries")
        return "error"

    status_code = response.status_code
    response_ms = to_ms(response.elapsed.total_seconds())
    if is_up(status_code):
        print(f"UP    {name}: {status_code} in {response_ms} ms")
        return "up"
    print(f"DOWN  {name}: {status_code} in {response_ms} ms")
    return "down"

response is None asks "did fetch() hand back no answer?"; is is how you check for None. The try and int() from Lesson 4 are gone, because the status code now comes from requests as a real number. Then replace everything from down = 0 to the end of the file, so the checker counts errors instead of skipped sites:

down = 0
errors = 0
results = []
for site in sites:
    result = check_site(site)
    results.append({"name": site["name"], "result": result})
    if result == "down":
        down = down + 1
    elif result == "error":
        errors = errors + 1

save_results("results.json", results)
print(f"{down} of {len(sites)} sites down, {errors} unreachable")
print("Results saved to results.json")

Checkpoint: your checker so far

Your checker.py should now look like this. If your blank lines differ a little, that's fine, but the line numbers in the break-it section assume this layout:

# A website checker, built one piece at a time.
import csv
import json
import time

import requests


def is_up(status_code):
    return status_code >= 200 and status_code < 400


def to_ms(seconds):
    return round(seconds * 1000)


def load_sites(path):
    with open(path, newline="") as file:
        return list(csv.DictReader(file))


def fetch(url):
    attempts = 0
    while attempts < 3:
        attempts = attempts + 1
        try:
            return requests.get(url, timeout=5)
        except requests.RequestException:
            print(f"      attempt {attempts} failed")
            time.sleep(1)
    return None


def check_site(site):
    name = site["name"]
    response = fetch(site["url"])
    if response is None:
        print(f"ERROR {name}: no answer after 3 tries")
        return "error"

    status_code = response.status_code
    response_ms = to_ms(response.elapsed.total_seconds())
    if is_up(status_code):
        print(f"UP    {name}: {status_code} in {response_ms} ms")
        return "up"
    print(f"DOWN  {name}: {status_code} in {response_ms} ms")
    return "down"


def save_results(path, results):
    with open(path, "w") as file:
        json.dump(results, file, indent=2)


sites = load_sites("sites.csv")
print("Checking sites...")
print(f"{len(sites)} sites to check")

down = 0
errors = 0
results = []
for site in sites:
    result = check_site(site)
    results.append({"name": site["name"], "result": result})
    if result == "down":
        down = down + 1
    elif result == "error":
        errors = errors + 1

save_results("results.json", results)
print(f"{down} of {len(sites)} sites down, {errors} unreachable")
print("Results saved to results.json")

Run it, with the virtual environment active:

python3 checker.py

Checking sites...
4 sites to check
UP    github.com: 200 in 97 ms
UP    pypi.org: 200 in 109 ms
DOWN  missing page: 404 in 77 ms
      attempt 1 failed
      attempt 2 failed
      attempt 3 failed
ERROR no such site: no answer after 3 tries
1 of 4 sites down, 1 unreachable
Results saved to results.json

Your times will differ; everything else should match. Those are real answers from real websites. The 404 is GitHub saying "that page isn't here", and the last site failed three times and was reported, while the rest of the run carried on. Add a site you use to sites.csv and run it again.

Break it on purpose

Make each mistake, run the checker, and put it back before the next.

Forget the virtual environment. Open a new terminal, cd into site-checker, and run the checker without activating anything:

Traceback (most recent call last):
  File "checker.py", line 6, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

You installed requests, so why can't Python find it? Because you installed it into .venv, and this terminal isn't using .venv. This is the most common package problem there is. When you see ModuleNotFoundError for something you know you installed, check your prompt for (.venv) first. Run source .venv/bin/activate (or the Windows version), and it works again.

A timeout that's far too short. On line 27, change timeout=5 to timeout=0.001, a thousandth of a second:

Checking sites...
4 sites to check
      attempt 1 failed
      attempt 2 failed
      attempt 3 failed
ERROR github.com: no answer after 3 tries

and the same for every other site, ending with 0 of 4 sites down, 4 unreachable. No real site answers that fast, so every request times out, and every site is reported unreachable, even the ones that are fine. A timeout has to be long enough for a healthy site, and short enough that a stuck one doesn't hold everything up. Put 5 back.

An address without https://. In sites.csv, change the first site's URL from https://github.com to just github.com, and run it:

Checking sites...
4 sites to check
      attempt 1 failed
      attempt 2 failed
      attempt 3 failed
ERROR github.com: no answer after 3 tries

GitHub is fine, so what happened? requests needs the full address, including https://, and without it the request fails before it's even sent. The checker caught that as a RequestException like any other and retried, but a wrong address fails every time. Retrying only helps with problems that come and go, like a slow network. If a site fails every try, look at its address first. Put https:// back.

Where you are now

Your checker visits real websites, reads real status codes, and keeps going when one doesn't answer. But you still have to run it yourself and read the output to know whether anything's wrong. Lesson 7 makes it a script you can trust when you're not watching.

Check yourself

Key terms

• Request and response: a program asks a site or API for something, and gets an answer back.
• Status code: the three-digit result of a request, like 200 (OK), 404 (not found) or 503 (the site has a problem).
• API (application programming interface): a way for programs to ask another service for data, usually answered in JSON.
• Package: code written by other people that you install, like requests.
• pip: Python's package installer. It downloads packages from PyPI (the Python Package Index).
• Virtual environment: a project's own folder of packages, made with python3 -m venv .venv and switched on with source .venv/bin/activate.
• Timeout: how long to wait for an answer before giving up.
• Retry: trying a failed request again, a limited number of times.
• None check: if response is None: asks whether a function handed back no value.

Go deeper

• The requests quickstart (opens in a new tab) covers status codes, JSON, timeouts and errors.
• Virtual environments and packages (opens in a new tab) in the official tutorial explains venv and pip in more depth.