learning-website-framework1-12  Exercise 3: Watch the Logs
====================================================================

After launch, the server log is your best monitor: it shows which old
addresses people still ask for (404), whether redirects are working (301),
and errors (5xx). Write a summariser for Apache's "combined" log format.

Save as top404.py (with no argument it runs on a small SYNTHETIC sample; give
it a log file path to run on a real log):

import re, sys
from collections import Counter

# Apache "combined" log format:
# host ident user [time] "METHOD /path HTTP/x" status bytes "referer" "agent"
LINE = re.compile(r'^(?P<host>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) [^"]*" (?P<status>\d{3}) (?P<bytes>\S+)')

def summarise(lines):
    status, not_found, redirects = Counter(), Counter(), Counter()
    for line in lines:
        m = LINE.match(line)
        if not m:
            continue
        status[m["status"][0] + "xx"] += 1
        if m["status"] == "404":
            not_found[m["path"]] += 1
        if m["status"] in ("301", "302"):
            redirects[m["path"]] += 1
    return status, not_found, redirects

SAMPLE = """\
203.0.113.5 - - [06/Oct/2026:10:00:01 +0000] "GET /hungary/hungarian-basic-3/hungarian_basic_conversation_3_1/ HTTP/1.1" 200 34567 "-" "Mozilla/5.0"
203.0.113.5 - - [06/Oct/2026:10:00:02 +0000] "GET /japan/hiragana/hiragana-tiles HTTP/1.1" 404 512 "-" "Mozilla/5.0"
198.51.100.7 - - [06/Oct/2026:10:00:05 +0000] "GET /japan/hiragana/hiragana-tiles HTTP/1.1" 404 512 "-" "Mozilla/5.0"
198.51.100.7 - - [06/Oct/2026:10:01:10 +0000] "GET /old-page-from-2019/ HTTP/1.1" 404 512 "-" "Googlebot/2.1"
192.0.2.44 - - [06/Oct/2026:10:02:00 +0000] "GET /linux/shell-and-scripting/vim/ HTTP/1.1" 301 238 "-" "Mozilla/5.0"
192.0.2.44 - - [06/Oct/2026:10:02:01 +0000] "GET /linux/shell-and-scripting/vim/ HTTP/1.1" 200 20111 "-" "Mozilla/5.0"
192.0.2.45 - - [06/Oct/2026:10:03:00 +0000] "GET /robots.txt HTTP/1.1" 404 512 "-" "Googlebot/2.1"
192.0.2.46 - - [06/Oct/2026:10:04:00 +0000] "GET /wp-login.php HTTP/1.1" 404 512 "-" "curl/8.0"
192.0.2.46 - - [06/Oct/2026:10:04:01 +0000] "GET /wp-login.php HTTP/1.1" 404 512 "-" "curl/8.0"
192.0.2.46 - - [06/Oct/2026:10:04:02 +0000] "GET /xmlrpc.php HTTP/1.1" 404 512 "-" "curl/8.0"
203.0.113.9 - - [06/Oct/2026:10:05:00 +0000] "GET /japan/ HTTP/1.1" 200 8000 "-" "Mozilla/5.0"
203.0.113.9 - - [06/Oct/2026:10:05:03 +0000] "GET /missing.pdf HTTP/1.1" 500 100 "-" "Mozilla/5.0"
""".splitlines()

if __name__ == "__main__":
    lines = open(sys.argv[1], encoding="utf-8", errors="ignore") if len(sys.argv) > 1 else SAMPLE
    status, nf, redir = summarise(lines)
    print("requests by status class:", dict(sorted(status.items())))
    print("most requested missing pages (404):")
    for path, n in nf.most_common(5):
        print(f"  {n:>3}  {path}")
    print("redirects served:", sum(redir.values()))

Run it:

  python top404.py                 (synthetic sample)
  python top404.py /var/log/apache2/languages-access.log

Output on the synthetic sample (checked by running it; these twelve lines are
invented, not from your server):

requests by status class: {'2xx': 3, '3xx': 1, '4xx': 7, '5xx': 1}
most requested missing pages (404):
    2  /japan/hiragana/hiragana-tiles
    2  /wp-login.php
    1  /old-page-from-2019/
    1  /robots.txt
    1  /xmlrpc.php
redirects served: 1

Reading it
----------
- A path that appears often under 404 is a missing redirect or a dead link:
  fix the link, or add a redirect.
- robots.txt at 404 means a site is missing its robots.txt (Chapter 7).
- Requests for /wp-login.php and /xmlrpc.php are automated probes for software
  you do not run. They are normal on any public server; do not chase them.
- Any 5xx deserves a look at the error log the same day.

A small monitoring routine
--------------------------
Daily (automatic if possible)
  - disk space: df -h (PDF-heavy releases fill disks)
  - certificate: sudo certbot renew --dry-run  (renewal works before it is needed)
Weekly
  - top404.py on each site's access log; apache error logs
  - the built-site link check (Exercise 1) against what is live
Monthly
  - external link check; Search Console coverage and crawl errors
  - a restore test from a backup (Chapter 8)

WHY THIS WORKS AS AN ANSWER
---------------------------
You cannot fix what you cannot see. A few counts from the log answer the
questions that matter after a migration (what is still being requested, and does
it work), and the routine puts every check on a schedule instead of leaving it
to memory.