CodingNic
Sign In Book Session

Web Scraping

Web Scraping with BeautifulSoup

Web Scraping 30 min read

Web Scraping with BeautifulSoup

Objectives

By the end of this chapter, you should be able to:

  • Explain what web scraping is useful for
  • Explain why robots.txt matters when scraping
  • Use BeautifulSoup to scrape web pages

💡 Why this matters: Plenty of useful data on the web has no API behind it. Scraping is what lets you turn an ordinary web page into structured data your program can actually use.

What Is Web Scraping?

Web scraping is downloading and extracting data from a website. It breaks down into three steps:

  1. Downloading the HTML document from a page
  2. Extracting the data you want from that HTML
  3. Doing something with the data, usually saving it somewhere

Ideally you’d get data through a site’s API instead, but plenty of sites don’t expose one. When there’s no programmatic way to get the data you need, scraping is the fallback.

Respect robots.txt

Before scraping any site, check its robots.txt file, a plain text file most sites publish that tells automated programs (like your scraper) what they’re allowed and not allowed to download. It typically lives at the root of a domain, e.g. https://example.com/robots.txt.

Different sites take very different stances here: some are wide open, others heavily restrict what bots may access. Always check a site’s robots.txt before scraping it, and honor what it says. You can read more about the standard at robotstxt.org.

BeautifulSoup

The library we’ll use for scraping is BeautifulSoup:

bash
pip3 install beautifulsoup4

BeautifulSoup handles step two: extracting data from HTML you’ve already downloaded. Without a tool like it, parsing raw HTML by hand is a genuinely hard problem.

Here’s BeautifulSoup pulling text out of a list of <li> elements:

python
import bs4

html = """
<html>
<body>
  <h3>Names</h3>
  <ul>
    <li>Erin</li>
    <li>Jordan</li>
    <li>Maya</li>
    <li>Priya</li>
  </ul>
</body>
</html>
"""

soup = bs4.BeautifulSoup(html, "html.parser")

for li in soup.find_all("li"):
    print(li.text)

BeautifulSoup(html, "html.parser") takes the HTML as its first argument and the parser to use as its second. "html.parser" (built into Python) works fine for most purposes.

You can also find a single element by its id:

python
import bs4

html = """
<html>
<body>
  <div id="interesting-data">Maya's favorite language is Python</div>
</body>
</html>
"""

soup = bs4.BeautifulSoup(html, "html.parser")

div = soup.find(id="interesting-data")
print(div.text)  # "Maya's favorite language is Python"

A few other methods worth knowing:

Method Purpose
.select() Find element(s) using CSS selectors
.children Get all children of an element
.parent Get the parent of an element

Downloading and Scraping a Real Page

Combine urllib.request (to download a page) with bs4 (to parse it):

python
import urllib.request
import bs4

url = "https://news.ycombinator.com/"
data = urllib.request.urlopen(url).read()
soup = bs4.BeautifulSoup(data, "html.parser")

links = soup.select("span.titleline > a")

for link in links:
    print(f"{link['href']} {link.text}")

"span.titleline > a" is a CSS selector, the same kind of syntax used to style web pages. span.titleline means “a <span> with class titleline,” and > a means “an <a> element that is a direct child of it.” .select() always returns a list, even when only one element matches.

A word of caution: the exact selector you need depends entirely on that page’s current HTML structure, which sites change over time. Before scraping any real page, open your browser’s dev tools and inspect the actual markup to find the right selector. Don’t assume an example selector will still match.

Saving Scraped Data

Once you’ve extracted what you want, save it. Here, we’ll use a TSV (tab-separated values) file, since article titles often contain commas:

python
import urllib.request
import bs4
import csv

url = "https://news.ycombinator.com/"
data = urllib.request.urlopen(url).read()
soup = bs4.BeautifulSoup(data, "html.parser")

links = soup.select("span.titleline > a")

with open("articles.tsv", "w") as tsvfile:
    writer = csv.writer(tsvfile, delimiter="\t")
    writer.writerow(("Link", "Title"))
    for link in links:
        writer.writerow((link["href"], link.text))

That’s the full scraping pipeline: download, extract, save.

Try It

  1. Write a small HTML string with a few elements, parse it with BeautifulSoup, and extract text with find_all.
  2. Check the robots.txt file of a real site you’re familiar with, in your browser, at /robots.txt.
  3. Download a real page with urllib.request, inspect its HTML in your browser’s dev tools, and try selecting one element with .select().

Recap

  • Scraping has three steps: download the HTML, extract the data you want, then save it.
  • Always check and honor a site’s robots.txt before scraping it.
  • BeautifulSoup parses downloaded HTML: find, find_all, and .select() are the main ways to locate elements, and .text gets their content.
  • CSS selectors depend on a page’s current markup, which changes over time. Inspect the live page before assuming a selector will work.

Next lesson: making HTTP requests directly to APIs with the requests module.