Simple Web Scraper

Python Projects (Beginner)
Course 4 · Chapter 7 · Simple Web Scraper

🕸️ Simple Web Scraper

Every project so far has used only the standard library. This one introduces two genuinely popular third-party packages — requests for fetching a web page, and BeautifulSoup for parsing its HTML — putting Course 1, Chapter 9's pip and venv coverage to real use for the first time in this course.

🎮 What We're Building

A scraper for quotes.toscrape.com — a site built specifically as safe, legal scraping practice — that fetches the page, pulls out every quote's text and author, and saves the results to a JSON file.

Step 1: Installing the Libraries

# with a venv active (Course 1, Chapter 9):
pip install requests beautifulsoup4

Two packages installed in one pip install call — requests handles the network request, beautifulsoup4 (imported as bs4) handles parsing the returned HTML into something searchable.

Step 2: Fetching a Page with requests

import requests

response = requests.get("https://quotes.toscrape.com")
print(response.status_code)   # 200 — success
print(response.text[:200])    # the raw HTML, as a string

requests.get(url) sends an HTTP request and returns a Response object. status_code follows the same HTTP status conventions from the site's own API-focused courses (200 = success, 404 = not found); .text is the full HTML the server sent back, as a plain string.

Step 3: Parsing HTML with BeautifulSoup

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, "html.parser")
quote_divs = soup.find_all("div", class_="quote")
print(len(quote_divs))   # 10 — one per quote on the page

BeautifulSoup(html, "html.parser") turns the raw HTML string into a navigable object. .find_all("div", class_="quote") searches for every <div class="quote"> element — the trailing underscore in class_ avoids colliding with Python's own class keyword (Course 2, Chapter 1).

Step 4: Extracting the Data

quotes = []

for div in quote_divs:
    text = div.find("span", class_="text").get_text()
    author = div.find("small", class_="author").get_text()
    quotes.append({"text": text, "author": author})

print(quotes[0])
# {'text': '"The world as we have created it...', 'author': 'Albert Einstein'}

Within each quote's div, .find() (singular — one match) locates the specific <span>/<small> tags holding the text and author; .get_text() extracts just the readable text, stripping the surrounding HTML tags. quotes.append({...}) reuses the exact list-of-dicts pattern from Chapters 4-6's to-do app and contact book.

Step 5: Saving the Results

import json
from pathlib import Path

Path("quotes.json").write_text(json.dumps(quotes, indent=2))

The same pathlib + json save pattern from every earlier project in this course — scraped data is just data, and it persists the same way hand-entered data does.

🏁 The Complete Web Scraper

import json
import requests
from bs4 import BeautifulSoup
from pathlib import Path

response = requests.get("https://quotes.toscrape.com")
soup = BeautifulSoup(response.text, "html.parser")
quote_divs = soup.find_all("div", class_="quote")

quotes = []
for div in quote_divs:
    text = div.find("span", class_="text").get_text()
    author = div.find("small", class_="author").get_text()
    quotes.append({"text": text, "author": author})

Path("quotes.json").write_text(json.dumps(quotes, indent=2))
print(f"Saved {len(quotes)} quotes.")
⚠ Scraping Has Real Legal and Ethical Limits

Not every site allows scraping — many explicitly forbid it in their Terms of Service, and even where it's technically allowed, hammering a server with rapid repeated requests can degrade it for real users or get your IP address blocked. Before scraping any real site: check its /robots.txt file (a standard, machine-readable statement of what's allowed), read its Terms of Service, and add a delay between requests (time.sleep(), Course 3, Chapter 3) if fetching multiple pages. quotes.toscrape.com is used here specifically because it's built and maintained for scraping practice — that permission doesn't transfer to other sites.

requests.get()

Fetches a page — .status_code and .text on the response.

BeautifulSoup

Parses HTML into a searchable structure.

find_all() / find()

Locate matching elements by tag name and class.

Scraping ethics

Check robots.txt and Terms of Service; don't hammer a server.

🚀 Extend This Project

Try these on your own:

  • Follow the "Next →" link on each page to scrape all 10 pages of quotes, not just the first — adding a time.sleep(1) between requests, per this chapter's warn-box.
  • Wrap requests.get() in try/except (Course 2, Chapter 3) to handle a network failure gracefully instead of crashing.
  • Also scrape each quote's tags (class_="tags"), adding a "tags" key to each quote's dict.
  • Count and print how many quotes each unique author has, using a dict as a counter (Course 1, Chapter 7's word-frequency pattern).

🎯 What's Next

Final chapter: Capstone: Expense Tracker — a small CLI app tying together file I/O, functions, and error handling into one complete, slightly larger project.