Challenge 3: Web Data Collector

Objectives

By the end of this chapter, you should be able to:

• Combine web scraping, regular expressions, and a custom context manager in one program
• Process scraped results lazily with a generator instead of building an entire list upfront

💡 Why this matters: Challenges 1 and 2 both worked with data you created yourself. This one pulls in real, live data from the web: the full pipeline from Module 9, plus regex and generators from Module 8, applied together instead of in separate small exercises.

The Project

Build a Web Data Collector: a tool that scrapes headlines (or another list of items) from a real page, filters them lazily by keyword, extracts any patterns you care about with regex, times the whole operation with a context manager, and saves the results to both JSON and CSV.

Core Requirements

Article (or @dataclass)

• title and url, at minimum
• A __str__ (or rely on @dataclass’s generated __repr__) so printing one is readable
• If you use @dataclass (from Module 6), dataclasses.asdict() converts an instance into a plain dictionary, exactly like the to_dict() method you wrote by hand in Challenge 1. That’s what makes it easy to pass to json.dump() or csv.DictWriter.

Scraping

• scrape_articles(url): uses requests and BeautifulSoup to download a page and return a list of Article objects. The Hacker News homepage from Module 9 is a safe, real target: inspect its current markup with your browser’s dev tools before writing your selector, since it changes over time.

Filtering as a Generator

• filter_articles(articles, keyword): a generator function that yields only the articles whose title contains keyword, instead of building a whole filtered list upfront

Pattern Extraction with Regex

• extract_patterns(text, pattern): given a block of text (for instance, every scraped title joined together) and a regex pattern string, returns every match with re.findall(). Use it for something concrete, like pulling out years or dollar amounts from the scraped titles.

Timing with a Context Manager

• Write a timer() context manager (either as a class with __enter__/__exit__, or with @contextmanager from contextlib) that logs how long the scrape took:

with timer():
    articles = scrape_articles(url)

Saving Results

• save_to_json(articles, filename) and save_to_csv(articles, filename): both using what you built in Module 7

Suggested Structure

import re
import csv
import json
import logging
import time
from contextlib import contextmanager
from dataclasses import dataclass, asdict

import requests
import bs4

logging.basicConfig(level=logging.INFO)


@dataclass
class Article:
    title: str
    url: str


@contextmanager
def timer():
    start = time.time()
    yield
    logging.info(f"Took {time.time() - start:.2f} seconds")


def scrape_articles(url):
    data = requests.get(url).text
    soup = bs4.BeautifulSoup(data, "html.parser")
    # inspect the live page first, since this selector depends on current markup
    links = soup.select("span.titleline > a")
    return [Article(title=link.text, url=link.get("href", "")) for link in links]


def filter_articles(articles, keyword):
    for article in articles:
        if keyword.lower() in article.title.lower():
            yield article


def extract_patterns(text, pattern):
    return re.findall(pattern, text)


def save_to_json(articles, filename):
    with open(filename, "w") as f:
        json.dump([asdict(a) for a in articles], f, indent=2)


def save_to_csv(articles, filename):
    with open(filename, "w") as f:
        writer = csv.DictWriter(f, fieldnames=["title", "url"])
        writer.writeheader()
        for a in articles:
            writer.writerow(asdict(a))

Fill in the pieces yourself, and adapt the scraping selector to whatever the target page’s actual markup looks like when you inspect it.

Example Behavior

You can’t predict the exact scraped titles ahead of time (they change every time the page updates), but you can verify each piece works correctly against fixed input first.

extract_patterns on a fixed string:

titles = "OpenAI announces new model in 2024; Rust 2.0 ships in 2025"
extract_patterns(titles, r"\b\d{4}\b")
# ['2024', '2025']

filter_articles on a fixed list:

sample = [Article("Python 3.13 released", "https://a"), Article("New JavaScript framework", "https://b")]
list(filter_articles(sample, "Python"))
# [Article(title='Python 3.13 released', url='https://a')]

Once those work, a full run against a live page looks something like this (the exact count and duration will vary):

INFO:root:Took 0.84 seconds
3 articles mention Python

Bonus Extensions

• Add a @log_duration decorator (similar to the one from Module 8) and apply it to scrape_articles directly, instead of wrapping the call site in with timer():. Compare the two approaches.
• Combine filter_articles and extract_patterns: filter articles by keyword first, then extract a pattern only from the matching subset.
• Add an argparse interface so the keyword, target URL, and output filenames are all provided on the command line rather than hardcoded.

Try It

1. Build scrape_articles against a real page, inspecting its current HTML in your browser first to get the selector right.
2. Build filter_articles as a generator, and loop over its results with a for loop rather than converting it to a list first.
3. Use extract_patterns to pull something concrete out of your scraped titles: years, dollar amounts, or similar.
4. Wrap the scrape in your timer() context manager and confirm it logs a duration.
5. Save your results to both JSON and CSV, and open each to confirm they look right.

Recap

You’ve now built a full pipeline (scrape, filter lazily, extract patterns, time it, save it two ways) using tools from four different modules in a single program. That’s the last of the three capstone challenges, and together with Challenges 1 and 2, they exercise essentially everything this course covered.

Congratulations on completing Python Fundamentals.