learning-website-framework1-12  Exercise 2: Size Budget and Cheaper Releases
================================================================================

PART A: Measure what each future site weighs
---------------------------------------------
Group the files of the current build by the site they would belong to, and show
how much of each is PDFs.

Needs sitemap.py (Chapter 5). Save as budget.py:

import os, sys
from collections import defaultdict
from sitemap import site_for

def site_of(rel):
    path = "/" + rel.replace(os.sep, "/")
    try:
        return site_for(path)
    except (KeyError, IndexError):
        return "(shared / not in a site)"

def measure(dist):
    size = defaultdict(int); files = defaultdict(int); pdf = defaultdict(int)
    for dirpath, dirnames, filenames in os.walk(dist):
        for n in filenames:
            p = os.path.join(dirpath, n)
            s = os.path.getsize(p)
            site = site_of(os.path.relpath(p, dist))
            size[site] += s; files[site] += 1
            if n.lower().endswith(".pdf"):
                pdf[site] += s
    return size, files, pdf

if __name__ == "__main__":
    size, files, pdf = measure(sys.argv[1])
    total = sum(size.values())
    print(f"{'site':<28}{'files':>7}{'MB':>9}{'PDF MB':>9}{'PDF share':>11}")
    for site in sorted(size, key=lambda s: -size[s]):
        mb, pm = size[site] / 1e6, pdf[site] / 1e6
        print(f"{site:<28}{files[site]:>7}{mb:>9.0f}{pm:>9.0f}{(100*pdf[site]/size[site] if size[site] else 0):>10.0f}%")
    print(f"{'TOTAL':<28}{sum(files.values()):>7}{total/1e6:>9.0f}{sum(pdf.values())/1e6:>9.0f}{100*sum(pdf.values())/total:>10.0f}%")

Run it:

  python budget.py "<debserver>/astro-site/dist"

Output (checked by running it):

site                          files       MB   PDF MB  PDF share
programming                    5182      264      205        78%
languages                      1149      210      195        93%
(shared / not in a site)        590      123       90        73%
systems                        2160      118       88        75%
webdevelopment                 1720       95       72        76%
humanities                     1686       57       36        63%
ai                              537       35       28        80%
lifeskills                      698       26       19        73%
creative                        666       16       12        71%
TOTAL                         14388      945      745        79%

Findings: 14,388 files and 945 MB, of which 745 MB (79%) is PDFs. The HTML is
small. In the languages site 93% of the weight is PDFs. The "shared" line
includes the 105 MB Hungarian resources folder (Chapter 10) and the shared
resource folders.

PART B: Do not copy 700 MB of unchanged PDFs on every release
-------------------------------------------------------------
The deploy script from Chapter 8 copies the whole build into every release,
and it keeps several releases. With mostly-unchanged PDFs that wastes a lot of
disk. rsync can HARD-LINK files that did not change to the previous release
(--link-dest), so a release only costs what changed. Test it on a 40 MB file
that never changes, with three releases each way.

deploy_linkdest.sh is deploy.sh (Chapter 8, Exercise 2) with this change in
place of the plain copy:

#!/usr/bin/env bash
# deploy_linkdest.sh <site> <build-dir>   (deploy.sh plus hard links to unchanged files)
# Copies a finished build into a new timestamped release and switches the
# "current" link to it in one atomic step. Keeps the last $KEEP releases.
set -euo pipefail

SITE="${1:?usage: deploy.sh <site> <build-dir>}"
BUILD="${2:?usage: deploy.sh <site> <build-dir>}"
ROOT="${WEB_ROOT:-/var/www}"
KEEP="${KEEP:-5}"

BASE="$ROOT/$SITE"
REL="$BASE/releases/$(date +%Y%m%d-%H%M%S-%N)"

[ -d "$BUILD" ] || { echo "build folder not found: $BUILD" >&2; exit 1; }
mkdir -p "$BASE/releases"
mkdir "$REL"          # no -p: fail if this release name already exists
# Files that did not change are hard-linked to the live release instead of copied,
# so a release costs only what actually changed. --checksum compares file CONTENTS:
# a rebuild gives every file a new modified time, and without it rsync would
# treat every file as changed (or, worse, treat a same-size file written in the
# same second as unchanged).
PREV=""
[ -e "$BASE/current" ] && PREV="$(readlink -f "$BASE/current")"
rsync -a --checksum --delete ${PREV:+--link-dest="$PREV"} "$BUILD"/ "$REL"/

# refuse to publish an empty or broken build
if [ ! -f "$REL/index.html" ]; then
  echo "no index.html in the build; not switching" >&2
  rm -rf "$REL"
  exit 1
fi

# atomic switch: make the new link beside the old one, then rename over it
ln -sfn "$REL" "$BASE/current.new"
mv -Tf "$BASE/current.new" "$BASE/current"
echo "deployed $SITE -> $(basename "$REL")"

# prune old releases: names are timestamps, so sort by NAME (not by modified
# time, which rsync -a copies from the build folder). Never delete the live one.
LIVE="$(readlink "$BASE/current")"
ls -1d "$BASE"/releases/* | sort -r | tail -n +"$((KEEP + 1))"   | { grep -v -x -F "$LIVE" || true; } | xargs -r rm -rf

Save as test_linkdest.sh:

#!/usr/bin/env bash
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
T="$(mktemp -d)"
mkdir -p "$T/build"
head -c 40000000 /dev/urandom > "$T/build/big.pdf"          # stands in for a 40 MB PDF that never changes
echo "page v1" > "$T/build/index.html"

for variant in deploy deploy_linkdest; do
  export WEB_ROOT="$T/www-$variant" KEEP=5
  for v in 1 2 3; do
    echo "page v$v" > "$T/build/index.html"                  # only the page changes
    bash "$HERE/$variant.sh" docs "$T/build" > /dev/null
    sleep 0.2
  done
  echo "$variant: 3 releases of a 40 MB build use $(du -sm "$WEB_ROOT/docs/releases" | cut -f1) MB on disk"
done
echo "live page (linkdest): $(cat "$T/www-deploy_linkdest/docs/current/index.html")"
rm -rf "$T"

Run it (on Linux, under WSL):

  bash test_linkdest.sh

Output (checked by running it):

deploy: 3 releases of a 40 MB build use 115 MB on disk
deploy_linkdest: 3 releases of a 40 MB build use 39 MB on disk
live page (linkdest): page v3

The pitfall the test caught
---------------------------
My first version used --link-dest WITHOUT --checksum. The disk figure looked
right, but the live page still showed version 1 after three deploys. rsync's
quick check treats a file as unchanged when its size and modified time match:
the two versions of index.html had the same size and were written within the
same second, so rsync hard-linked the OLD content. The opposite happens on a
real build: every file is rewritten with a new modified time, so nothing would
be linked at all. --checksum compares contents, which fixes both, at the cost of
reading the files each time (acceptable: about a gigabyte).

An alternative design is to keep the PDFs outside the releases entirely, in a
shared folder that the release links to, and update it separately.

Build time is a separate budget. After the course was written the current site
was built with npm run build: 4,996 pages in about 1 minute 35 seconds in total
(a test build into another folder took 1 minute 21 seconds). Set a limit, and
treat a build that grows past it as a reason to look.

WHY THIS WORKS AS AN ANSWER
---------------------------
The measurement shows where the weight really is (PDFs, not pages). The test
shows the fix (115 MB down to 39 MB) and, because it also checks the live
content, catches a fix that looked right but served stale pages.