Regular Expressions

Objectives

By the end of this chapter, you should be able to:

• Explain what a regular expression is and when to reach for one
• Use the re module to search, match, and extract patterns in text
• Use common pattern elements: character classes, quantifiers, and groups
• Escape special characters and match alternatives with |
• Predict what re.findall() returns when a pattern does, or doesn’t, contain groups

💡 Why this matters: BeautifulSoup gets you the raw text out of HTML, but pulling a phone number, an email address, or a price out of that text is a job for pattern matching, not string slicing. Regular expressions are the standard tool for that, well beyond just scraping.

What Is a Regular Expression?

A regular expression (regex) is a pattern that describes a shape of text: “three digits, then a dash, then four digits,” rather than one exact string. Python’s re module lets you search for, match, and extract text matching a pattern like that.

import re

re.search(r"\d+", "There are 42 apples")  # matches "42"

The r before the string makes it a raw string. It tells Python not to treat backslashes as escape characters, which matters constantly in regex patterns (\d, \w, and so on all rely on the backslash meaning something to re, not to Python itself).

Core Functions

import re

text = "Call me at 555-1234 or 555-5678"

re.search(r"\d{3}-\d{4}", text)       # a match object for "555-1234"
re.findall(r"\d{3}-\d{4}", text)      # ['555-1234', '555-5678']
re.sub(r"\d{3}-\d{4}", "[redacted]", text)
# "Call me at [redacted] or [redacted]"

A re.search() or re.match() result is a match object, not the matched text directly. Call .group() on it to get the actual matched string:

match = re.search(r"\d{3}-\d{4}", text)
match.group()  # "555-1234"

If there’s no match at all, both functions return None, so check for that before calling .group() on the result, or you’ll hit an AttributeError.

Building a Pattern

A few building blocks cover most everyday patterns:

Put together, r"\d{3}-\d{4}" reads as “exactly three digits, a literal dash, then exactly four digits.” That’s exactly what matched the phone numbers above.

A few of these elements are easy to misread, so it’s worth proving each one on its own:

import re

re.findall(r"[abc]", "cab")             # ['c', 'a', 'b'] - each letter matches one of a, b, or c
re.findall(r"colou?r", "color colour")  # ['color', 'colour'] - the "u" is optional (zero or one)
re.findall(r"go*d", "gd god good")      # ['gd', 'god', 'good'] - zero or more "o"s in between
re.search(r"^Erin", "Hi, Erin!")        # None - the string doesn't start with "Erin"
re.search(r"^Erin", "Erin says hi")     # a match - the string does start with "Erin"

Two more elements deserve special attention, because they trip up beginners constantly:

• Characters like (, ), and . mean something special to re. To match one of them literally, escape it with a backslash: \( matches a literal opening parenthesis, instead of starting a group.
• The pipe character | means “or,” letting one pattern match several different shapes of text.

import re

re.findall(r"\(\d{3}\) \d{4}", "Call (555) 1234 now")  # ['(555) 1234'] - \( and \) match literal parentheses
re.findall(r"cat|dog", "I have a cat and a dog")        # ['cat', 'dog'] - either word matches

Capturing Groups

Parentheses mark a group: a piece of the match you want to pull out individually, rather than the whole match:

import re

text = "erin@example.com"
match = re.search(r"(\w+)@(\w+\.\w+)", text)

match.group()   # 'erin@example.com' - the whole match
match.group(1)  # 'erin' - the first group
match.group(2)  # 'example.com' - the second group

This is a common pattern when scraping: extract the whole match to confirm it’s there, then pull out just the piece you actually need.

findall() and Groups: A Common Gotcha

re.findall() behaves differently once a pattern contains groups. Without groups, it returns a list of full matches. With groups, it returns a list of tuples, one tuple per match, with one entry per group, and the full match is not included at all:

import re

text = "erin@example.com"

re.findall(r"(\w+)@(\w+\.\w+)", text)  # [('erin', 'example.com')] - a tuple, one entry per group
re.findall(r"\w+@\w+\.\w+", text)      # ['erin@example.com'] - the full match, since there are no groups

If you want findall() to give you full match strings, drop the parentheses (or don’t add any in the first place). Only reach for groups when you specifically want the pieces split apart.

Try It

1. Use re.findall() to pull every number out of a sentence containing several.
2. Write a pattern that matches a simple email address, and test it against a few strings that should and shouldn’t match.
3. Use re.sub() to replace every occurrence of a word in a sentence with another word.
4. Add a capturing group to one of your patterns, and pull out just that piece with .group(1).
5. Write a pattern using | that matches either of two different words in a sentence, and confirm it with re.findall().
6. Write a pattern that matches a phone number in the form (555) 123-4567, remembering to escape the parentheses.

Recap

• A regular expression describes a shape of text to search for, rather than one exact string.
• re.search() finds the first match anywhere; re.match() only matches at the start; re.findall() returns every match; re.sub() replaces matches.
• \d, \w, \s, and quantifiers like +, *, and {3} cover most everyday patterns.
• Escape special characters like ( and . with a backslash to match them literally, and use | to match one of several alternatives.
• Parentheses create a capturing group, letting you pull out one specific piece of a match with .group(1). Once a pattern has groups, re.findall() returns tuples of the group contents instead of full matches.

Next lesson: put web scraping, requests, and regular expressions into practice with a set of exercises.