Deployment and Operations

The sites work on a development machine. This chapter is about the part that decides whether they keep working: running them with the production settings, knowing when something is wrong, backing up what cannot be rebuilt, and releasing a new version in a way that can be undone. It is also where honesty matters most, because the target is a Debian server and this was written on Windows, so a large part of it could not be run. Each section says which part was.

Where Each Thing Runs

Step 1: Check the Production Settings

The test suite runs under the development settings, so it cannot say whether the production ones work. Django's deployment check can: manage.py check --deploy with the production settings reports two warnings, about HSTS for subdomains and HSTS preloading. They are left on deliberately. Including subdomains would make browsers insist on HTTPS for every name under osztromok.com, including any old host without a certificate, and preloading is close to permanent. HSTS is set to 3600 seconds; raise it in steps only when every host is known to work over HTTPS.

Step 2: Know When It Is Broken

Every site (and the admin host) has a /healthz/ address. It answers 200 only when the process can really serve: the database answers and holds pages (for a site, that site's pages). Otherwise it answers 503. It is never cached and says nothing else, so a monitor or the deploy script can call it freely.

The smoke test goes further. It sends real HTTP requests to a running copy and checks, for each site: health, the front page, that the front page links a stylesheet and the stylesheet loads, robots.txt, the sitemap, search, and a few pages from the sitemap whose canonical link must equal the sitemap address. For the admin it checks that it asks for a login and that /admin/ is a 404 on the content sites. Without a connect address it uses each site's real name, so DNS, the certificate and Apache's choice of virtual host are tested too.

The failing runs matter as much as the passing one: a check that has never been seen to fail has not been shown to check anything.

Step 3: Back Up What the Files Cannot Rebuild

The pages are in the content folder and can always be rebuilt. But accounts, reading progress and redirect decisions exist only in the database, and so does the search index (which takes a minute to rebuild). SQLite has an online backup that is safe while the site runs. Copying the file with cp is not: a write in the middle gives a damaged copy.

• The backup command copies with SQLite's own backup, then opens the copy the way a restore would: integrity check, the page and search tables present, pages not zero. A copy that fails is deleted and reported.
• It keeps the newest few (14 in the cron entry) and deletes only files named exactly like its own, so a file you put in the folder is safe.
• A test found a real bug: a damaged file made the check raise instead of reporting “not ok”. It now reports the reason.

So pages and search come back from the files in about a minute; accounts, progress and redirects come back only from a backup. A backup kept on the same disk does not survive the disk, so copy them elsewhere and check the copy.

Step 4: Apache, Certificates and Gunicorn (Not Run)

The Apache configuration is generated from the site map (apache_vhosts), so it cannot disagree with it: one HTTPS host for each of the eight sites and the admin host, an HTTP host for each that redirects to HTTPS, the site's own asset folder served by Apache, and X-Forwarded-Proto set so Django knows the visitor used HTTPS. The asset pattern was tested against good and bad addresses, including /../pdfs/a.pdf and /x/./pdfs/a.pdf, which it must not match.

Certificates

• One wildcard certificate for osztromok.com and *.osztromok.com: needs the DNS challenge, and a new site needs no new certificate. The generated configuration assumes this.
• One certificate per name (nine names now): no DNS access needed, but every new site is a new certificate and a new place for renewal to fail.

A certificate that quietly expires is the most common way sites like this fail. The smoke test, run from outside by name over HTTPS, notices an expired or wrong certificate.

Gunicorn

A small configuration file (3 workers, 2 threads, a 30-second timeout, workers restarted after about 2,000 requests so a slow leak cannot grow forever) and a systemd unit that runs it on a unix socket as an unprivileged user, with a read-only file system apart from the data, asset and backup folders. The secret key lives in an environment file readable only by that user; the production settings refuse to start without it.

Step 5: A Release With a Way Back

Hands-On Exercises

Course Complete

You have built a Django project that serves eight sites from one codebase, one database and one design system, and the tools to move the old site onto it without breaking its addresses. What each chapter left you able to do: