Deployment and Operations

The last chapter is about the day after the code works: how to know a release is healthy before anyone sees it, how to get the accounts back after a mistake, how Apache and systemd run the sites, and how to undo a release. As in the other chapters, what could be run was run, and what could not is said plainly.

One Node Process Per Site, Behind Apache

There were two shapes to choose from: a static export (just files, any web server) or a Node server (next start). The login API, the health address and the per-request checks need a server, so each site runs as its own Node process listening on 127.0.0.1 only, on the port it already has (portfolio 3000, languages 3001, and so on). Apache handles HTTPS and passes each host name to its port. A static export of the pages alone stays possible later if the accounts move elsewhere.

Both the virtual hosts and the systemd units are generated from the site map (node ops.mjs config), so they cannot disagree with it. Only sites that have an app get a virtual host; a site without one keeps answering from the old site until its turn.

Compression, Measured

Next compresses its pages: a lesson page was 59,774 bytes plain and 12,406 bytes with gzip. What a route handler sends is not compressed: asked with Accept-Encoding: gzip, /search-index.json came back as 1,018,766 bytes, exactly the same as without, and so did the sitemap. Compressed with gzip it would be 326,259 bytes. So the virtual host asks Apache to compress JSON, XML and text. Untested: I could not run Apache, so I have not seen this work, nor seen it leave alone a response Next has already compressed (that is its documented behaviour).

Before the Release: Settings

node ops.mjs check stops the release on mistakes that would otherwise show up as a strange site: LW_ENV not prod (every link would point at localhost), no content folder, no data folder or one inside the release folder (the next release would not see the accounts), and the experiment pages switched on. With a good environment it says production settings: ok. With development settings, no data folder and the experiment pages on, it printed three errors and exited with 1.

While It Runs: Health and a Smoke Test

/healthz answers 200 only when the database answers and the site has pages ({"status":"ok","pages":561}), is never cached, and says nothing else. The smoke test then visits the running site as a visitor would, with the Host header Apache will send, and checks 11 things: the health address; the first, the middle and the last page of the site; that a missing address answers 404 and not 200; that a real PDF is served with nosniff; robots, sitemap and search index; and two logins, one from another site's page (must be 403) and one from our own (must get past the lock, 400). Against the running languages app: 11 checks passed. With the app stopped, all 11 failed with ECONNREFUSED instead of hanging.

Backups You Have Restored

The only data that cannot be rebuilt from the content folder is the accounts database. A plain file copy is the wrong tool: the database runs in WAL mode, so recent writes sit in accounts.sqlite-wal, and a copy made mid-write can be damaged. VACUUM INTO asks SQLite itself for a complete, consistent copy while the site runs. The order matters: the copy is made under a temporary name, checked, only then renamed, and only after that are backups beyond the newest 14 removed, so a broken backup can never push out a good one.

A restore checks the backup before touching anything, keeps the database it replaces (accounts.sqlite.before-restore), and moves the old -wal and -shm files away with it, because they belong to the old database. Run for real: two accounts, a backup, a third account, a restore; the third was gone and the three-account database was kept. A file that is not a database was refused: refusing to restore bad.sqlite: file is not a database, and the accounts were unchanged.

A Release With a Way Back

deploy.sh release <ref> builds a new folder next to the old one, then: 1. backs up the database; 2. checks the settings; 3. installs, tests and builds; 4. switches the current symlink (atomic) and restarts the services; 5. runs the smoke test against every running site, and if one check fails, rolls back by itself. A rollback is moving the symlink and restarting. There are no migrations (tables are only created if missing), so old code normally still works with the database; if a release changed a table, restore the backup taken at its start.

A first rollout that cannot hurt

1. Install on the server and run the settings check.
2. Start the languages unit on its port and run the smoke test against it directly (curl -H "Host: ...").
3. Only then add its virtual host (apachectl configtest, then reload) and run the smoke test over HTTPS from outside.
4. Add the RedirectMatch line for its folders to the old site (Chapter 11), and watch the logs for 404s.
5. Keep the old site's lines in place for a few weeks, so going back is removing one line.

Hands-On Exercises

Course Complete

You have built a Next.js workspace that serves several sites from shared packages, with the tools to move the old site across without breaking its addresses. What each chapter left you able to do: