Web Scraping, Data Collection & Copyright

Chapter 9 · Web Scraping, Data Collection & Copyright

dsproj1-1 built a real scraper and flagged "check robots.txt/ToS" as good practice without ever explaining why. This chapter finally answers that — and, in the spirit of Chapter 8's own three-question breakdown, immediately splits "is scraping legal?" into several genuinely separate legal questions, of which copyright is only one.

"Is Scraping Legal?" Is Not One Question

Consideration	What body of law it actually falls under
Is the scraped content copyrightable?	Copyright law — this course's own primary subject
Does using the site require agreeing to terms prohibiting scraping?	Contract law — genuinely separate from copyright
Does collecting the data require bypassing a technical access barrier?	Computer-fraud-type statutes (in the U.S., the CFAA) — also separate from copyright

A scraping project can be entirely fine on one of these dimensions and genuinely risky on another — treating "scraping" as governed by a single unified law is exactly the kind of oversimplification this course has repeatedly corrected elsewhere.

The Copyright Dimension

The technical act of scraping — copying content from a page into your own systems — does implicate the reproduction right from Chapter 3, if what's being copied is copyrightable expression. But a great deal of what data scientists actually scrape is exactly the kind of material Chapter 1 already excluded from protection entirely.

A direct callback to Chapter 1

Prices, stock levels, sports scores, weather readings, structured product specifications — bare facts, per Chapter 1's own table, "discovered, not created," and not protectable by copyright at all. Scraping this kind of factual data for analysis generally doesn't implicate copyright in any meaningful way, regardless of what a site's own robots.txt or ToS might separately say about it.

Creative/expressive content is a different matter

Scraping actual articles, reviews, or blog posts — genuinely creative written expression, not bare facts — does implicate real copyrightable material. Whatever happens to that scraped text afterward (storing it, redistributing it, using it to train a model per Chapter 8's own Question A) needs its own separate fair use analysis under Chapter 4's four factors, exactly as if the text had been copied any other way.

Correcting a Common Myth: robots.txt Is Not a Law

A voluntary technical signal, not a binding legal barrier

robots.txt is a voluntary technical convention — a file a site publishes to signal which pages it would prefer automated crawlers avoid. It is not, by itself, a legally binding restriction in most circumstances, and ignoring it isn't automatically a copyright violation or any other independent legal claim on its own. That said, ignoring it is against widely accepted web norms, and disregarding it can be cited as evidence of bad faith in a separate legal dispute — it's a real signal worth respecting, just not because the file itself carries independent legal force the way a statute does.

Terms of Service — A Genuinely Different, Binding Mechanism

Unlike robots.txt, a site's Terms of Service can be a real, binding contract — if the scraper actually agreed to it (by creating an account, clicking "I agree," or in some cases simply by using a site whose terms are clearly presented). Violating a ToS provision that prohibits scraping is a potential breach of contract, an entirely different legal claim from copyright infringement, with its own separate remedies (account termination, contract damages) rather than the copyright-specific mechanisms covered in Chapters 3 and 7.

Bypassing Technical Access Controls — A Third, Separate Concern

Actively defeating a login wall, a paywall, or a CAPTCHA to reach data goes beyond ordinary scraping and can raise concerns under computer-fraud-type statutes — in the U.S., most notably the Computer Fraud and Abuse Act (CFAA) — which is a body of law entirely separate from copyright.

A real, significant case: hiQ Labs v. LinkedIn

In this closely watched U.S. case, the Ninth Circuit Court of Appeals found that scraping data that is publicly accessible — not hidden behind a login — generally does not violate the CFAA, since the statute concerns "unauthorized access," and data openly available without authentication isn't accessed without authorization in the relevant sense. The case had a genuinely complex procedural history (including being reconsidered after a separate U.S. Supreme Court decision, Van Buren v. United States, narrowed how courts interpret the CFAA) and eventually settled — a real, useful data point, not a permanently settled rule for every scraping scenario. The general shape it supports, though, is a genuinely useful one: scraping publicly accessible data sits on meaningfully more solid legal ground under CFAA-type analysis than scraping data behind a login or other access barrier.

Putting It Together — Why dsproj1-1's Advice Was Right, For Reasons It Never Explained

Good practice reduces risk across several separate legal dimensions at once

Checking a site's robots.txt and ToS before scraping isn't legally required in every single case, but it's genuinely good practice because it touches all three dimensions covered in this chapter simultaneously: it respects the voluntary technical signal (goodwill, and evidence against a bad-faith argument), it surfaces any binding contractual restriction before it's violated (the genuine ToS/contract-law risk), and checking whether data sits behind a login wall at all is the single clearest signal for whether CFAA-type concerns are even in play. dsproj1-1's own advice was sound — this chapter is simply the legal reasoning underneath it that chapter never had room to cover.

Hands-On Exercises

Exercise 1

A data scientist scrapes only publicly listed product prices from a retail site, with no login required. Explain, using this chapter's own material, why this is very likely fine from a copyright standpoint specifically — while noting what it doesn't tell you about the other two considerations.

📄 View solution

Exercise 2

A colleague says, "our scraper ignores robots.txt, so we're definitely breaking the law." Explain what's wrong with this statement, and what robots.txt actually is instead.

📄 View solution

Exercise 3

Explain what the hiQ Labs v. LinkedIn case suggests about the difference between scraping publicly accessible data and scraping data behind a login wall, and why the case shouldn't be treated as a permanently settled rule.

📄 View solution

Chapter 9 Quick Reference

"Is scraping legal?" splits into three separate questions: copyright, Terms of Service (contract law), and access-control circumvention (CFAA-type law)
Bare factual data (prices, scores, structured specs) generally isn't copyrightable at all, per Chapter 1 — scraping it usually doesn't implicate copyright meaningfully
Creative/expressive scraped text (articles, reviews) does implicate real copyright — any later use needs its own fair use analysis
robots.txt is not legally binding by itself — a voluntary technical convention, not a law, though ignoring it can be cited as evidence of bad faith elsewhere
A site's Terms of Service can be a genuinely binding contract if actually agreed to — violating it is breach of contract, not copyright infringement
Bypassing login walls/CAPTCHAs raises CFAA-type concerns; hiQ Labs v. LinkedIn suggests publicly accessible data scraping sits on firmer ground than login-gated scraping, though the case has real, complex nuance
Next chapter: Capstone — A Real-World Copyright Decision Framework