learning-website-django1-4  Exercise 3: Import the Real Content and Measure It
============================================================================

Run the importer on the real content folder, measure it, and look at the result.

Point the project at the content (or pass --root):

  set LW_CONTENT_ROOT=<path to>\content          (Linux: export LW_CONTENT_ROOT=...)

Then run, on a fresh database:

  python manage.py migrate
  python manage.py import_content --dry-run
  python manage.py import_content
  python manage.py import_content                  (nothing changed)
  python manage.py import_content --force
  python manage.py import_content --site languages

Output (checked by running it on the real content folder; the times are for a
folder synced by OneDrive on an ordinary PC, so yours will differ):

  migrate (content only)
  Applying content.0001_initial... OK
  Applying content.0002_page_content_hash_page_fragment_page_imported_at... OK

  --dry-run
  DRY RUN (nothing written): created 4403, updated 0, unchanged 0, errors 0  (43.2s)

  first import
  created 4403, updated 0, unchanged 0, errors 0  (19.7s)

  second import, nothing changed
  created 0, updated 0, unchanged 4403, errors 0  (11.5s)

  --force
  created 0, updated 4403, unchanged 0, errors 0  (22.8s)

  --site languages  (everything already imported)
  created 0, updated 0, unchanged 561, errors 0  (1.2s)

  database file size: 99 MB

Save as import_stats.py in the project folder and run it:

"""import_stats.py: look at what the import stored. Run from the project folder."""
import os

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.dev")
import django

django.setup()

from django.db.models import Count, Sum
from django.db.models.functions import Length

from apps.content.models import Course, Page

print(f"pages: {Page.objects.count()}   courses: {Course.objects.count()}")
for row in Page.objects.values("site").annotate(n=Count("id"), size=Sum(Length("fragment"))).order_by("-n"):
    print(f"   {row['site']:<15}{row['n']:>6} pages  {row['size'] / 1e6:>6.1f} MB of HTML")

page = Page.objects.light().get(path__endswith="hiragana_あ.html")
print("a non-ASCII file name:", page.url_path, "|", page.site, "|", page.kind)

course = Course.objects.get(folder="hungary/hungarian-basic-3")
print(course.name, "-> chapters", [p.chapter_no for p in course.pages.light()])

print("pages whose stored body still starts with the banner comment:",
      Page.objects.filter(fragment__startswith="<!-- ====").count())
print("pages with no stored hash:", Page.objects.filter(content_hash="").count())

pages: 4403   courses: 395
   programming      1371 pages    33.4 MB of HTML
   humanities        707 pages     9.5 MB of HTML
   systems           653 pages    18.9 MB of HTML
   languages         561 pages     7.3 MB of HTML
   webdevelopment    514 pages    13.9 MB of HTML
   lifeskills        253 pages     3.2 MB of HTML
   ai                187 pages     3.9 MB of HTML
   creative          157 pages     2.0 MB of HTML
a non-ASCII file name: /japan/japanese-language/reference-materials/hiragana/hiragana_あ/ | languages | full_page
Hungarian Basic Conversation 3 -> chapters [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
pages whose stored body still starts with the banner comment: 0
pages with no stored hash: 0

What this shows
---------------
- All 4,403 pages import with no errors, in about 20 seconds, into 395 courses.
- The database is about 99 MB, which is the size of the HTML itself. The
  largest page is 0.77 MB and none is over 1 MB, so the 2 MB limit never fires.
- The unchanged run takes about 10 seconds. Nearly all of it is reading every
  file to hash it, from a synced folder. Hashing the bytes costs far less than
  parsing, and on the server's own disk it will be faster.
- --site languages first took 16 seconds, because the first version parsed every
  file before it checked the site. Deciding from the path alone, before reading
  the file, brought it to between 1 and 5 seconds, depending on whether the
  files were already in the disk cache. Takeaway: measure, then move the cheap
  test before the expensive work.
- Timings vary a lot with that cache: the dry run took 13 seconds on one run and
  43 on another. Repeat a measurement before drawing a conclusion from it.
- The page whose file name is a Japanese character imports, and its address is
  the same path without .html. 282 of the paths contain non-ASCII characters; none
  differs only by letter case from another, and all are in the NFC form, so the
  database's unique path is safe on both Windows and Linux.
- The programming site holds 33 MB of the HTML, systems 19 MB, web development 14
  MB, and languages 7 MB. The numbers will move as content is added.

WHY THIS WORKS AS AN ANSWER
---------------------------
Numbers from the real tree settle the design questions (is a body in the database
too big? is an import fast enough to run on every deploy?) and showed a slow
path worth fixing. The checks on case and Unicode forms are cheap and prevent
the kind of duplicate-path bug that only appears when the site moves from
Windows to Linux.