learning-website-django1-12 Exercise 3: Apache, Gunicorn, Certificates and a Release With a Way Back ==================================================================================================== Everything in this exercise is configuration for a Debian server. WHAT WAS CHECKED here: the generated Apache text by tests and by reading it, the shell script by "bash -n" (syntax only), the Gunicorn file by parsing it. WHAT WAS NOT: none of it was run, because there is no Apache, Gunicorn or systemd on the machine this was written on. Treat all of it as a first draft to test on the server, step by step. Part A: Apache, generated from the site map so it cannot disagree with it. Save as apps/core/deploy.py: """Apache virtual hosts, generated from the site map so they cannot disagree with it.""" from config.sites_config import DOMAIN, SITES # a folder name may not be . or .., so the pattern cannot be walked out of the site's own folder ASSET_PATTERN = r"^/((?:(?!\.\.?/)[^/]+/)*(?:pdfs|solutions)/[^/]+\.(?:pdf|txt))$" def vhost(host, *, socket, cert_dir, asset_dir=None): """One HTTPS virtual host. asset_dir is the folder of this site's PDFs and solutions (None for the admin host).""" assets = "" if asset_dir: assets = f""" # PDFs and solutions: straight from this site's own folder (Gunicorn never sees them) AliasMatch "{ASSET_PATTERN}" "{asset_dir}/$1" Require all granted Header set X-Content-Type-Options "nosniff" """ proxy_exceptions = f' ProxyPassMatch "{ASSET_PATTERN}" !\n' if asset_dir else "" return f""" ServerName {host} SSLEngine on SSLCertificateFile {cert_dir}/fullchain.pem SSLCertificateKeyFile {cert_dir}/privkey.pem {assets} {proxy_exceptions} ProxyPreserveHost On RequestHeader set X-Forwarded-Proto "https" ProxyPass / unix:{socket}|http://localhost/ ProxyPassReverse / unix:{socket}|http://localhost/ ServerName {host} RedirectMatch 301 ^/(.*)$ https://{host}/$1 """ def render_vhosts(*, socket="/run/lw/gunicorn.sock", cert_dir=None, asset_root="/srv/lw/assets", domain=DOMAIN, admin_host=None): """The configuration for every site and the admin host. One wildcard certificate covers them all by default.""" cert_dir = cert_dir or f"/etc/letsencrypt/live/{domain}" admin_host = admin_host or f"admin.{domain}" parts = [vhost(f"{site}.{domain}", socket=socket, cert_dir=cert_dir, asset_dir=f"{asset_root}/{site}") for site in SITES] parts.append(vhost(admin_host, socket=socket, cert_dir=cert_dir)) return "\n".join(parts) python manage.py apache_vhosts > /etc/apache2/sites-available/lw.conf One HTTPS host per site plus the admin host, each with an HTTP host that redirects to HTTPS, the site's own asset folder served by Apache (never by Gunicorn), and X-Forwarded-Proto set so Django knows the visitor used HTTPS. Needed Apache modules: ssl, proxy, proxy_http, headers, alias. Then: a2enmod ssl proxy proxy_http headers alias; apache2ctl configtest; systemctl reload apache2. The asset pattern is tested against good and bad addresses, including /../pdfs/a.pdf, /x/../solutions/a.txt and /x/./pdfs/a.pdf, which it must not match. Part B: certificates. Eight sites plus the admin host is nine names. Two ways: - One wildcard certificate for osztromok.com and *.osztromok.com. It needs the DNS challenge (certbot with your DNS provider's plugin), and a new site needs no new certificate. - One certificate per name with the HTTP challenge: no DNS access needed, but each new site means a new certificate and a new place for renewal to fail. The generated configuration assumes the wildcard (one folder, /etc/letsencrypt/live/osztromok.com); use --cert-dir to change it. Renewal is automatic with certbot's timer, and a certificate that quietly expires is the most common way these sites fail, so the smoke test (run from outside, by name, over HTTPS) also catches an expired or wrong certificate. Nothing here was run. Part C: Gunicorn under systemd. """Gunicorn settings for the Learning Website (one process group serving every site). Run by systemd (see lw-gunicorn.service): gunicorn -c deploy/gunicorn.conf.py config.wsgi Apache is the only client: it talks to the unix socket, so Gunicorn never listens on the network.""" import multiprocessing bind = "unix:/run/lw/gunicorn.sock" umask = 0o117 # socket readable and writable by the service user and group (Apache's group) only # The work is short database reads and template rendering, so ordinary synchronous workers are enough. # (2 x cores + 1) is Gunicorn's own rule of thumb; a small server does better with fewer: start at 3 and measure. workers = min(3, multiprocessing.cpu_count() * 2 + 1) threads = 2 timeout = 30 # a page that takes longer is a fault, not a slow page graceful_timeout = 30 max_requests = 2000 # restart each worker now and then, so a slow memory leak cannot grow forever max_requests_jitter = 200 # ... and not all of them at once accesslog = "-" # stdout and stderr go to journald errorlog = "-" loglevel = "warning" forwarded_allow_ips = "127.0.0.1" # only believe X-Forwarded-* from the local Apache [Unit] Description=Learning Website (Gunicorn) After=network.target [Service] User=lw Group=www-data RuntimeDirectory=lw WorkingDirectory=/srv/lw/current EnvironmentFile=/etc/lw/environment ExecStart=/srv/lw/venv/bin/gunicorn -c deploy/gunicorn.conf.py config.wsgi ExecReload=/bin/kill -s HUP $MAINPID Restart=on-failure RestartSec=3 # a little hardening: the service needs to read its code and write only its database, backups and asset folders NoNewPrivileges=true PrivateTmp=true ProtectSystem=strict ReadWritePaths=/srv/lw/data /srv/lw/assets /srv/lw/backups [Install] WantedBy=multi-user.target Gunicorn listens on a unix socket that only Apache can reach, so it is never on the network. The environment file /etc/lw/environment holds DJANGO_SETTINGS_MODULE=config.settings.prod, DJANGO_SECRET_KEY, LW_DATABASE, LW_CONTENT_ROOT, LW_ASSET_ROOT, LW_BACKUP_DIR and LW_ALLOW_CRAWLING; it must be readable only by the service user (it contains the secret key). Production settings refuse to start without a secret key. Part D: a release with a way back. #!/usr/bin/env bash # Roll out a new release, and roll back if it is not healthy. # # deploy.sh release build a new release next to the old one and switch to it # deploy.sh rollback go back to the previous release # # Layout: /srv/lw/releases// one folder per release (code only) # /srv/lw/current a symlink to the live release # /srv/lw/data/db.sqlite3 the database (shared by all releases) # /srv/lw/backups/ database backups # /srv/lw/content/ the content folder (a git checkout of the content repository) set -euo pipefail ROOT=/srv/lw PYTHON=$ROOT/venv/bin/python # the same settings the service uses (DJANGO_SETTINGS_MODULE, DJANGO_SECRET_KEY, LW_* paths) set -a # shellcheck disable=SC1091 . /etc/lw/environment set +a health_ok() { # real HTTPS requests to every site by its own name, as a visitor would send them: DNS, certificate, # Apache's virtual host, Gunicorn and Django all have to be right "$PYTHON" "$1/manage.py" smoke_test --tls --per-site 2 } release() { ref=${1:?usage: deploy.sh release } new=$ROOT/releases/$(date +%Y%m%d-%H%M%S) git -C "$ROOT/repo" fetch --quiet git -C "$ROOT/repo" worktree add --detach "$new" "$ref" "$ROOT/venv/bin/pip" install --quiet -r "$new/requirements/prod.txt" cd "$new" # 1. a backup BEFORE anything changes the database: migrations only go forward "$PYTHON" manage.py backup_db "$ROOT/backups" # 2. the new code's own checks, with the real settings "$PYTHON" manage.py check --deploy --fail-level ERROR "$PYTHON" manage.py migrate --noinput "$PYTHON" manage.py collectstatic --noinput "$PYTHON" manage.py import_content --prune "$PYTHON" manage.py collect_assets # 3. switch: a symlink change is atomic, then reload Gunicorn without dropping requests previous=$(readlink -f "$ROOT/current" || true) echo "$previous" > "$ROOT/previous-release" ln -sfn "$new" "$ROOT/current" sudo systemctl reload lw-gunicorn sleep 3 # 4. look at the running site; if it is not healthy, go back at once if ! health_ok "$new"; then echo "The new release failed its smoke test: rolling back." >&2 rollback exit 1 fi echo "Released $new" } rollback() { previous=$(cat "$ROOT/previous-release") [ -d "$previous" ] || { echo "no previous release to go back to" >&2; exit 1; } ln -sfn "$previous" "$ROOT/current" sudo systemctl reload lw-gunicorn echo "Back on $previous." echo "If this release ran a migration, the OLD code may not match the new database." echo "Restore the backup taken at the start of the release if the site misbehaves:" echo " systemctl stop lw-gunicorn; cp /srv/lw/backups/.sqlite3 /srv/lw/data/db.sqlite3; systemctl start lw-gunicorn" } case "${1:-}" in release) shift; release "$@" ;; rollback) rollback ;; *) echo "usage: deploy.sh release | rollback" >&2; exit 2 ;; esac The order matters: backup FIRST (migrations only go forward), then the checks, migrate, collectstatic, import, assets, THEN switch the symlink and reload, then smoke-test, and if that fails go back at once. The switch is a symlink change (atomic) and a reload (Gunicorn finishes current requests). The honest limit of the rollback: it restores the OLD CODE. If the release ran a migration, the old code may not match the new database, so keep migrations backward compatible (add a column in one release, use it in the next), or restore the backup taken at the start of that release, which loses anything written since. import_content --prune has also already changed the pages table, but the pages come back from the files by running it again on the old release. deploy.sh passes "bash -n" (syntax check only) and was never run. It needs a sudoers rule that lets the deploy user run "systemctl reload lw-gunicorn" and nothing else. Part E: the tests for all of the above (the logic of each is tested; the servers are not): Save as tests/test_operations.py: import re import sqlite3 import tempfile from datetime import datetime, timedelta from io import StringIO from pathlib import Path from unittest import mock from django.core.management import call_command from django.core.management.base import CommandError from django.db import DatabaseError from django.test import Client, SimpleTestCase, TestCase from apps.content.models import Page from apps.core import backup, deploy, smoke from config.sites_config import SITES def make(path, site="languages"): return Page.objects.create(path=path, site=site, kind="lesson", title="T", fragment="

x

") class HealthTests(TestCase): def test_an_empty_database_is_not_healthy(self): response = Client(HTTP_HOST="languages.localhost").get("/healthz/") self.assertEqual(response.status_code, 503) self.assertEqual(response.json()["status"], "no pages") def test_a_site_with_pages_is_healthy_and_the_answer_is_never_cached(self): make("hungary/x/a.html") response = Client(HTTP_HOST="languages.localhost").get("/healthz/") self.assertEqual((response.status_code, response.json()), (200, {"status": "ok", "pages": 1})) self.assertEqual(response["Cache-Control"], "no-store") def test_each_site_counts_its_own_pages(self): make("hungary/x/a.html") self.assertEqual(Client(HTTP_HOST="systems.localhost").get("/healthz/").status_code, 503) def test_a_database_error_is_a_503_not_a_crash(self): make("hungary/x/a.html") with mock.patch.object(Page.objects, "filter", side_effect=DatabaseError("disk I/O error")): response = Client(HTTP_HOST="languages.localhost").get("/healthz/") self.assertEqual((response.status_code, response.json()["status"]), (503, "database error")) def test_the_admin_host_answers_too(self): make("hungary/x/a.html") self.assertEqual(Client(HTTP_HOST="admin.localhost").get("/healthz/").status_code, 200) def make_database(folder, *, name="db.sqlite3", pages=2, indexed=True, users=1): path = Path(folder) / name conn = sqlite3.connect(path) conn.execute("CREATE TABLE content_page (id INTEGER PRIMARY KEY, path TEXT)") conn.execute("CREATE TABLE auth_user (id INTEGER PRIMARY KEY)") if indexed: conn.execute("CREATE TABLE page_fts (title TEXT)") conn.executemany("INSERT INTO content_page(path) VALUES (?)", [(f"p{n}",) for n in range(pages)]) conn.executemany("INSERT INTO auth_user DEFAULT VALUES", [()] * users) conn.commit() conn.close() return path class BackupTests(SimpleTestCase): def setUp(self): self.folder = tempfile.TemporaryDirectory() self.addCleanup(self.folder.cleanup) self.db = make_database(self.folder.name) self.backups = Path(self.folder.name) / "backups" def test_a_backup_is_a_checked_copy(self): report = backup.backup_sqlite(self.db, self.backups) self.assertTrue(Path(report["path"]).is_file()) self.assertEqual((report["pages"], report["users"]), (2, 1)) self.assertRegex(Path(report["path"]).name, r"^lw-\d{8}-\d{6}\.sqlite3$") def test_the_original_is_untouched(self): before = self.db.read_bytes() backup.backup_sqlite(self.db, self.backups) self.assertEqual(self.db.read_bytes(), before) def test_only_the_newest_few_are_kept_and_only_our_own_files_are_deleted(self): self.backups.mkdir() (self.backups / "notes.txt").write_text("mine") (self.backups / "lw-backup-of-something-else.sqlite3").write_text("mine") start = datetime(2026, 10, 1, 3, 15) for day in range(5): backup.backup_sqlite(self.db, self.backups, keep=3, now=start + timedelta(days=day)) names = sorted(p.name for p in self.backups.iterdir()) self.assertEqual([n for n in names if n.startswith("lw-2026")], ["lw-20261003-031500.sqlite3", "lw-20261004-031500.sqlite3", "lw-20261005-031500.sqlite3"]) self.assertIn("notes.txt", names) self.assertIn("lw-backup-of-something-else.sqlite3", names) def test_a_backup_that_fails_its_check_is_removed_and_reported(self): db = make_database(self.folder.name, name="noindex.sqlite3", indexed=False) with self.assertRaisesRegex(RuntimeError, "page_fts"): backup.backup_sqlite(db, self.backups) self.assertEqual([p for p in self.backups.iterdir()], []) # the bad copy was deleted def test_an_empty_database_is_not_a_backup_worth_keeping(self): empty = make_database(self.folder.name, name="empty.sqlite3", pages=0) with self.assertRaisesRegex(RuntimeError, "empty"): backup.backup_sqlite(empty, self.backups) def test_a_damaged_file_does_not_verify(self): report = backup.backup_sqlite(self.db, self.backups) path = Path(report["path"]) path.write_bytes(path.read_bytes()[:100] + b"garbage" * 200) result = backup.verify_backup(path) self.assertFalse(result["ok"]) self.assertTrue(result["problem"]) with self.assertRaises(CommandError): call_command("backup_db", verify=str(path), stdout=StringIO()) def test_the_command_reports_and_the_verify_option_checks_a_file(self): out = StringIO() with self.settings(DATABASES={"default": {"ENGINE": "django.db.backends.sqlite3", "NAME": str(self.db)}}): call_command("backup_db", str(self.backups), stdout=out) self.assertIn("2 pages", out.getvalue()) newest = next(self.backups.iterdir()) call_command("backup_db", verify=str(newest), stdout=StringIO()) with self.assertRaises(CommandError): call_command("backup_db") # no folder and no setting @mock.patch("apps.core.smoke.fetch") class SmokeTests(SimpleTestCase): HOSTS = {"languages": "languages.localhost", "systems": "systems.localhost"} def serve(self, fetch, *, broken=None, canonical_ok=True): broken = broken or {} def fake(connect, host, path, **kwargs): if (host, path) in broken: return broken[(host, path)] if path == "/": return 200, {}, b'' if path.endswith(".css"): return 200, {}, b"" if path == "/robots.txt": return 200, {}, b"User-agent: *\n" if path == "/sitemap.xml": urls = "".join(f"http://{host}/p{n}/" for n in range(5)) return 200, {}, f'{urls}'.encode() if path.startswith("/p"): canonical = f"http://{host}{path}" if canonical_ok else "http://wrong.example/" return 200, {}, f''.encode() if path == "/admin/": return 404, {}, b"" return 200, {}, b"ok" fetch.side_effect = fake def run_smoke(self, **kwargs): return smoke.smoke("127.0.0.1:8000", self.HOSTS, per_site=2, **kwargs) def test_a_healthy_pair_of_sites_passes(self, fetch): self.serve(fetch) report = self.run_smoke() self.assertEqual(report.failures, []) self.assertGreater(report.checks, 16) def test_a_failing_page_names_the_site_and_the_address(self, fetch): self.serve(fetch, broken={("systems.localhost", "/healthz/"): (503, {}, b"")}) self.assertEqual(self.run_smoke().failures, ["[systems] /healthz/ -> 503"]) def test_a_canonical_link_that_differs_from_the_sitemap_is_caught(self, fetch): self.serve(fetch, canonical_ok=False) self.assertTrue(any("canonical link missing or different" in f for f in self.run_smoke().failures)) def test_a_missing_stylesheet_link_is_a_failure(self, fetch): self.serve(fetch, broken={("languages.localhost", "/"): (200, {}, b"

no css

")}) self.assertIn("[languages] front page links a stylesheet", self.run_smoke().failures) def test_the_admin_must_ask_for_a_login_and_stay_off_the_content_sites(self, fetch): self.serve(fetch, broken={("admin.localhost", "/"): (302, {"location": "/login/?next=/"}, b"")}) self.assertEqual(self.run_smoke(admin_host="admin.localhost").failures, []) self.serve(fetch, broken={("admin.localhost", "/"): (200, {}, b"")}) self.assertEqual(self.run_smoke(admin_host="admin.localhost").failures, ["[admin] asks for a login -> 200"]) class VirtualHostTests(SimpleTestCase): def setUp(self): self.text = deploy.render_vhosts() def test_there_is_one_https_host_per_site_and_one_for_the_admin(self): names = re.findall(r"\s+ServerName (\S+)", self.text) self.assertEqual(names, [f"{site}.osztromok.com" for site in SITES] + ["admin.osztromok.com"]) def test_every_host_redirects_http_to_https(self): for name in re.findall(r"\s+ServerName (\S+)", self.text): self.assertIn(f"https://{name}/$1", self.text) self.assertEqual(self.text.count(""), len(SITES) + 1) def test_each_site_serves_only_its_own_asset_folder_and_the_admin_serves_none(self): for site in SITES: self.assertIn(f'"/srv/lw/assets/{site}/$1"', self.text) admin_block = self.text.split("ServerName admin.osztromok.com")[1].split("")[0] self.assertNotIn("AliasMatch", admin_block) def test_apache_is_told_to_say_https_to_django(self): self.assertEqual(self.text.count('RequestHeader set X-Forwarded-Proto "https"'), len(SITES) + 1) self.assertIn("ProxyPreserveHost On", self.text) def test_the_asset_pattern_matches_only_pdfs_and_solutions_inside_a_folder(self): pattern = re.compile(deploy.ASSET_PATTERN) for good in ("/hungary/x/pdfs/a.pdf", "/hungary/x/solutions/s1_exercise1.txt", "/a/b/c/pdfs/Course.pdf"): self.assertTrue(pattern.match(good), good) for bad in ("/hungary/x/pdfs/a.html", "/hungary/x/pdfs/", "/hungary/x/other/a.pdf", "/a.pdf", "/../pdfs/a.pdf", "/x/../solutions/a.txt", "/x/./pdfs/a.pdf", "/hungary/x/pdfs/sub/a.pdf.exe"): self.assertFalse(pattern.match(bad), bad) def test_the_certificate_folder_and_socket_can_be_chosen(self): text = deploy.render_vhosts(socket="/tmp/s.sock", cert_dir="/certs/example") self.assertIn("SSLCertificateFile /certs/example/fullchain.pem", text) self.assertIn("unix:/tmp/s.sock|http://localhost/", text) class ProductionSettingsTests(SimpleTestCase): def test_the_logging_setup_is_valid_and_goes_to_stderr(self): import importlib import logging.config import os with mock.patch.dict(os.environ, {"DJANGO_SECRET_KEY": "x" * 60}): prod = importlib.reload(importlib.import_module("config.settings.prod")) logging.config.dictConfig(prod.LOGGING) # raises if the setup is invalid self.assertEqual(prod.LOGGING["handlers"]["console"]["class"], "logging.StreamHandler") self.assertEqual(prod.ADMIN_HOST, "admin.osztromok.com") Test run (whole project): Found 304 test(s). System check identified no issues (0 silenced). Old site: 2 asset, 1 dynamic, 4 page No new site owns 0 page/asset addresses (decide: keep on the old host, or drop). pages owned by a site: 4 2 missing, 1 redirected, 1 same assets owned by a site: 2 2 missing Missing addresses written to C:\Users\emuba\AppData\Local\Temp\tmp9iylp7ym\missing.tsv Creating test database for alias 'default'... ......................................................................................................................................................................................................................................................................C:\lwdj\proj\tests\test_operations.py:118: UserWarning: Overriding setting DATABASES can lead to unexpected behavior. with self.settings(DATABASES={"default": {"ENGINE": "django.db.backends.sqlite3", "NAME": str(self.db)}}): .......................................... ---------------------------------------------------------------------- Ran 304 tests in 80.198s OK Destroying test database for alias 'default'... WHY THIS WORKS AS AN ANSWER --------------------------- It keeps what is testable under test (the generated text, the backup logic, the smoke test's decisions) and says plainly that the rest is untried, instead of presenting an untested server setup as finished.