CodingNic
Sign In Book Session

Web Scraping

Web Scraping Exercises

Web Scraping 45 min read

Web Scraping Exercises

Objectives

This chapter introduces no new concepts. It’s a chance to put web scraping, file I/O, API requests, and regular expressions together into a few complete programs.

Part I: Scrape Headlines

Write a Python program that uses BeautifulSoup to visit a news homepage and print every headline on the page. Then write a function find_headline_by_keyword(keywords) that searches those headlines and returns only the ones matching every keyword you provide.

Note: some news sites render their content with JavaScript after the page loads, which means a simple urllib.request download won’t contain the headlines at all: you’ll only see an empty shell. If the site you pick behaves this way, fall back to a simpler, mostly-static page (the Hacker News homepage from the previous lesson works well) so you can focus on the BeautifulSoup logic itself.

Part II: Scrape a Table to CSV

Wikipedia’s United States presidential election article includes a table of results for every election. Use BeautifulSoup to scrape that table into a CSV file with these columns: order, year, winner, winner electoral votes, runner-up, and runner-up electoral votes. Use commas as the delimiter. The first row of data should look like:

text
1st,1788-1789,George Washington,69,John Adams,34

Setting breakpoints with breakpoint() (or pdb) as you go is a great way to confirm you’re selecting the right elements and pulling the correct text before you commit to writing the whole file.

Part III: Server-Side Requests

Using the requests module and the OMDB API, build a program that asks the user for an actor’s name and a movie title, then reports whether that actor appears in the movie’s listed cast (this will only work reliably for lead actors, since OMDB doesn’t return a full cast list). Bonus: also report the movie’s director and writer.

OMDB requires a free API key: register at omdbapi.com/apikey.aspx, then pass it as an extra apikey entry in the params dictionary you send with each request, the same technique you practiced in the previous lesson.

Part IV: Regular Expressions

Write a function extract_contact_info(text) that takes a block of scraped or pasted text and uses re.findall() to return every email address and every US-style phone number (555-123-4567 or (555) 123-4567) it finds, as two separate lists.

python
sample = "Reach Erin at erin@example.com or 555-123-4567, or Jordan at (555) 987-6543."
extract_contact_info(sample)
# (['erin@example.com'], ['555-123-4567', '(555) 987-6543'])

Bonus: run it against the raw text of a real scraped page and see what it turns up.

Try It

  1. Build the headline scraper and keyword search function from Part I against a real page.
  2. Attempt the Wikipedia table scrape from Part II, using breakpoint() to inspect the HTML structure as you go.
  3. Build the OMDB lookup program from Part III, and try the bonus director/writer feature.
  4. Build extract_contact_info from Part IV, and test it against text containing more than one email and phone number.

Recap

You can now scrape real web pages with BeautifulSoup, respect a site’s robots.txt, save scraped data to a file, make server-side HTTP requests to APIs with the requests module, and extract structured patterns out of raw text with regular expressions. That’s the toolkit this module set out to build.

Next module: the capstone project. It brings everything from this course together into one program.