Server-Side HTTP Requests

Objectives

By the end of this chapter, you should be able to:

• Use the requests module to make server-side HTTP requests
• Compare client-side and server-side API requests

💡 Why this matters: Scraping HTML gets you data a site never intended to hand you directly. Talking to an API with requests is the cleaner path whenever a service actually offers one.

Client-Side vs. Server-Side Requests

You can make HTTP requests from JavaScript running in a browser (a technique called AJAX), but that approach doesn’t always work: many APIs block browser-based requests for security reasons, a policy called CORS. Making the request from your own Python program instead, server to server, sidesteps that restriction entirely, since CORS is enforced by browsers, not by servers.

The requests Module

Install it with:

pip3 install requests

Here’s a GET request against httpbin.org, a free public test API that simply echoes back whatever you send it (no sign-up required, which makes it a good place to learn the mechanics):

import requests

r = requests.get("https://httpbin.org/get", params={"course": "python-fundamentals"})

print(r.status_code)
print(r.ok)
print(r.json())

That prints something like this (some fields, like headers and origin, are trimmed here since they vary per request):

200
True
{'args': {'course': 'python-fundamentals'}, 'headers': {...}, 'origin': '...', 'url': 'https://httpbin.org/get?course=python-fundamentals'}

requests.get() returns a response object. status_code and ok tell you whether the request succeeded, headers gives you the response’s HTTP headers, and .json() parses a JSON response body into a Python dictionary. The params dictionary got turned into the ?course=python-fundamentals query string automatically, which is exactly what comes back under "args".

Making a POST Request

requests.post() sends data in the body of the request instead of the URL. Use data= for that:

import requests

payload = {"name": "Erin", "role": "student"}
r = requests.post("https://httpbin.org/post", data=payload)

print(r.json()["form"])

{'name': 'Erin', 'role': 'student'}

httpbin.org echoes back whatever you send it under the "form" key, so r.json()["form"] shows exactly what your program sent. r.text is also available on any response if you want the raw body as a string instead of parsed JSON.

Most real APIs work the same way, but require you to sign up for a free API key and send it with every request. The OMDB movie API used in this module’s exercises works this way: you’ll pass your key as an extra entry in the params dictionary, the same technique used above.

Try It

1. Make a GET request to https://httpbin.org/get with a query parameter of your choice, and print r.status_code and r.json().
2. Check r.ok after a request to a URL you know doesn’t exist, and see what status_code comes back.
3. Make a POST request to https://httpbin.org/post with a small dictionary as data, and print r.json()["form"] to confirm it arrived intact.
4. Explain, in your own words, why a server-side request sometimes succeeds where a browser-based request to the same API would be blocked.

Recap

• Server-side requests (from your own Python program) can reach APIs that block direct browser-based requests, because CORS is enforced by browsers, not servers.
• requests.get() and requests.post() return a response object with status_code, ok, headers, .text, and .json().
• Use params= to send data in the URL’s query string, and data= to send it in the request body.
• Many real APIs require a free API key sent along with every request.

Next lesson: regular expressions, for pulling exact patterns out of text.