learning-website-framework1-12  Exercise 1: Check the Built Site, Not the Source
==============================================================================

Follow every internal link in a BUILT site and confirm each one reaches a file.
This is the test that catches what source checks cannot: broken tiles pages,
missing solution files, and links built by templates.

Rules the checker applies:
- a root-relative or relative link is internal; http(s) links are not checked
  here (external links are checked separately, on a schedule)
- a target exists if it is a file, or a folder containing index.html or
  index.php (the Anime Vault is a PHP application)
- the same broken target on many pages is counted once as a distinct target
  and once per page affected

Save as check_built_links.py:

import os, posixpath, re, sys
from collections import Counter
from urllib.parse import urlsplit, unquote

ATTR = re.compile(r'\b(?:href|src)="([^"]*)"', re.I)

def target_exists(dist, path):
    """A root-relative path exists if it is a file, or a folder with an index page.
    index.php counts too: the Anime Vault is a PHP application, not a static page."""
    p = os.path.join(dist, unquote(path).strip("/"))
    return (os.path.isfile(p)
            or os.path.isfile(os.path.join(p, "index.html"))
            or os.path.isfile(os.path.join(p, "index.php")))

def check(dist):
    pages = links = 0
    broken = Counter()
    for dirpath, dirnames, filenames in os.walk(dist):
        if "index.html" not in filenames:
            continue
        pages += 1
        page_url = "/" + os.path.relpath(dirpath, dist).replace(os.sep, "/") + "/"
        text = open(os.path.join(dirpath, "index.html"), encoding="utf-8", errors="ignore").read()
        for url in set(ATTR.findall(text)):
            u = url.strip()
            if not u or u.startswith(("#", "//")) or u.lower().startswith(("mailto:", "tel:", "javascript:", "data:")):
                continue
            parts = urlsplit(u)
            if parts.scheme or parts.netloc:
                continue                                  # external: not checked here
            path = parts.path
            if not path:
                continue
            if not path.startswith("/"):
                path = posixpath.normpath(posixpath.join(page_url, path))
            links += 1
            if not target_exists(dist, path):
                broken[path] += 1
    return pages, links, broken

if __name__ == "__main__":
    pages, links, broken = check(sys.argv[1])
    print(f"pages checked: {pages}   internal links followed: {links}")
    print(f"distinct broken targets: {len(broken)}   (pages affected: {sum(broken.values())})")
    for path, n in broken.most_common(8):
        print(f"  {n:>5}  {path}")

Run it on the current site's build:

  python check_built_links.py "<debserver>/astro-site/dist"

Output (checked by running it):

pages checked: 5014   internal links followed: 288980
distinct broken targets: 205   (pages affected: 521)
    141  /japan/hiragana/hiragana-tiles
    141  /japan/katakana/katakana-tiles
      6  /about
      2  /mark-used/&lt;id&gt;/
      2  /predictor/
      2  /web-development/scripting-and-backend/javascript/fundamentals/solutions/js1-1_challenge1.txt
      2  /web-development/scripting-and-backend/javascript/fundamentals/solutions/js1-1_challenge2.txt
      2  /web-development/scripting-and-backend/javascript/fundamentals/solutions/js1-1_challenge3.txt

Reading the result
------------------
- 5,014 pages and about 289,000 internal links were followed.
- The two biggest groups are the dead back-links found in Chapter 10: 141
  hiragana pages and 141 katakana pages pointing at tiles pages that do not
  exist.
- The js1-1_challenge links are missing solution files (the Chapter 6 audit
  found 54 of these in JavaScript Fundamentals).
- /about, /predictor and /mark-used/<id>/ are example addresses inside lesson
  text, not real links: a checker has false positives, and each finding needs a
  look. The 205 distinct targets are not all triaged here; only the top of the
  list is.
- A first version of this checker reported 4,994 broken links to /anime/. That
  folder holds a PHP application with index.php, not index.html. The fix was
  to accept index.php too. Takeaway: when one target is "broken" on almost
  every page, suspect the checker before the site.

WHY THIS WORKS AS AN ANSWER
---------------------------
Everything else in this course checks the source or the plan. Following the
links in the finished output tests what a visitor will actually get. Run it
after every build, and fail the build on any new broken target.