learning-website-nextjs1-10  Exercise 2: The API, and a Real HTTP Test of Every Lock
=================================================================================

Four small routes are the only dynamic part of the site. Each is marked force-dynamic: it runs when asked, never at build time, and never
cached.

Save as apps/languages/lib/accounts.ts:

import { join } from "node:path";

import { isJson, isSameOrigin, openDatabase, type Db } from "@lw/accounts";
import { environment } from "@lw/sites";

export const COOKIE = "lw_session";
const MAX_BODY = 4096;

let db: Db | undefined;

/** The accounts database, opened the first time a request needs it (never while the site is being built). */
export function getDb(): Db {
  if (!db) {
    const folder = process.env["LW_DATA_DIR"] ?? join(process.cwd(), "data");
    db = openDatabase(join(folder, "accounts.sqlite"));
  }
  return db;
}

export function readCookie(request: Request, name: string): string | undefined {
  for (const part of (request.headers.get("cookie") ?? "").split(";")) {
    const [key, ...value] = part.trim().split("=");
    if (key === name) return value.join("=");
  }
  return undefined;
}

/**
 * The login cookie. HttpOnly: no script on the page can read it. SameSite=Lax: another site cannot make a browser send it along with a
 * form post. Secure when the site is on HTTPS (production). No Domain: a browser sends it to this host only, so each site has its own login.
 */
export function sessionCookie(token: string, seconds: number): string {
  return `${COOKIE}=${token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=${seconds}${environment() === "prod" ? "; Secure" : ""}`;
}

export const clearedCookie = () => sessionCookie("", 0);

/** Never cached, and it depends on the cookie: neither a browser nor a shared cache may keep one person's answer for another. */
export const PRIVATE = { "Cache-Control": "no-store", Vary: "Cookie" };

export function json(body: unknown, status = 200, headers: Record<string, string> = {}): Response {
  return Response.json(body, { status, headers: { ...PRIVATE, ...headers } });
}

/** The two locks every POST goes through: it must come from our own pages, and it must say it is JSON. Returns the refusal, or null. */
export function guardPost(request: Request): Response | null {
  if (!isSameOrigin(request.headers)) return json({ error: "Not allowed from here." }, 403);
  if (!isJson(request.headers)) return json({ error: "Send JSON." }, 415);
  return null;
}

/** The body as JSON, or null if it is too big or not JSON. */
export async function readJson(request: Request): Promise<Record<string, unknown> | null> {
  const text = await request.text();
  if (text.length > MAX_BODY) return null;
  try {
    const value: unknown = JSON.parse(text);
    return value !== null && typeof value === "object" && !Array.isArray(value) ? (value as Record<string, unknown>) : null;
  } catch {
    return null;
  }
}

Save as apps/languages/app/api/login/route.ts:

import { SESSION_SECONDS, login } from "@lw/accounts";

import { getDb, guardPost, json, readJson, sessionCookie } from "../../../lib/accounts";

export const dynamic = "force-dynamic";

export async function POST(request: Request) {
  const refused = guardPost(request);
  if (refused) return refused;
  const body = await readJson(request);
  const username = body?.["username"];
  const password = body?.["password"];
  if (typeof username !== "string" || typeof password !== "string" || username.length > 40 || password.length > 200) {
    return json({ error: "Give a user name and a password." }, 400);
  }
  const result = await login(getDb(), username, password);
  if (!result.ok) {
    // the SAME words for a wrong password and a name that does not exist
    return result.reason === "locked"
      ? json({ error: "Too many wrong passwords for that name. Try again in a few minutes." }, 429, { "Retry-After": "900" })
      : json({ error: "Wrong user name or password." }, 401);
  }
  return json({ user: { username: result.user.username } }, 200, { "Set-Cookie": sessionCookie(result.token, SESSION_SECONDS) });
}

Save as apps/languages/app/api/logout/route.ts:

import { logout } from "@lw/accounts";

import { COOKIE, clearedCookie, getDb, guardPost, json, readCookie } from "../../../lib/accounts";

export const dynamic = "force-dynamic";

/** Logging out changes something, so it is a POST with the same two locks: a link or an image on another page cannot do it. */
export async function POST(request: Request) {
  const refused = guardPost(request);
  if (refused) return refused;
  logout(getDb(), readCookie(request, COOKIE));
  return json({ user: null }, 200, { "Set-Cookie": clearedCookie() });
}

Save as apps/languages/app/api/me/route.ts:

import { finishedPaths, userForSession } from "@lw/accounts";

import { COOKIE, getDb, json, readCookie } from "../../../lib/accounts";
import { content } from "../../../lib/site";

export const dynamic = "force-dynamic";

/** Who is logged in (if anyone) and the pages of THIS site they have finished. */
export function GET(request: Request) {
  const db = getDb();
  const user = userForSession(db, readCookie(request, COOKIE));
  if (!user) return json({ user: null });
  const here = new Set(content.all().map((page) => page.path));
  return json({ user: { username: user.username }, done: finishedPaths(db, user.id).filter((path) => here.has(path)) });
}

Save as apps/languages/app/api/progress/route.ts:

import { setDone, userForSession } from "@lw/accounts";

import { COOKIE, getDb, guardPost, json, readCookie, readJson } from "../../../lib/accounts";
import { content } from "../../../lib/site";

export const dynamic = "force-dynamic";

/** Mark a page finished, or not. Only a logged-in visitor, only a real page of THIS site, and nothing but that page's own path is stored. */
export async function POST(request: Request) {
  const refused = guardPost(request);
  if (refused) return refused;
  const db = getDb();
  const user = userForSession(db, readCookie(request, COOKIE));
  if (!user) return json({ error: "Log in first." }, 401);
  const body = await readJson(request);
  const path = body?.["path"];
  const done = body?.["done"];
  if (typeof path !== "string" || typeof done !== "boolean") return json({ error: "Give a path and true or false." }, 400);
  if (!content.all().some((page) => page.path === path)) return json({ error: "No such page on this site." }, 404);
  setDone(db, user.id, path, done);
  return json({ path, done });
}

The locks every POST goes through (guardPost):
1. It must come from OUR OWN pages. A page on another site can make a visitor's browser send a request to ours, with the visitor's cookie. A
   browser always says where such a request came from (Origin), so a request whose Origin is not this host is refused (403). Even another of
   OUR sites is refused: each site has its own login.
2. It must say it is JSON (415 otherwise). A form posted from another site cannot (without the browser asking first).
3. The cookie is SameSite=Lax (another site cannot make the browser send it with a form post), HttpOnly (no script can read it), host-only (no
   Domain: this site's login is not sent to another subdomain), 30 days, and Secure in a production build.
4. The body is read as text first and refused if over 4 KB; types are checked (a user name that is a list, not text, is a 400, not a crash).
5. Marking progress accepts only a path that is a real page of THIS site; anything else (a made-up path, "../../etc/passwd", another site's page,
   empty) is a 404.
6. Answers about a person are "Cache-Control: no-store" and depend on the cookie.

Tested over real HTTP, against the running app (a throwaway database; the passwords are made up for the run and never printed):

Save as accounts-http-test.mjs:

// Test the login and progress API over real HTTP, against a running languages app (started with LW_DATA_DIR pointing at a scratch folder).
//   LW_DATA_DIR=<scratch folder> node accounts-http-test.mjs <url>
// Passwords are made up for the run, kept in memory, and never printed.
import { randomBytes } from "node:crypto";
import { createUser, openDatabase } from "./packages/accounts/src/index.ts";

const base = process.argv[2] ?? "http://localhost:3001";
const origin = base;
const dataDir = process.env.LW_DATA_DIR;
if (!dataDir) throw new Error("Set LW_DATA_DIR to the folder the app was started with.");
const db = openDatabase(`${dataDir}/accounts.sqlite`);
const pw = () => randomBytes(12).toString("hex");
const users = { philip: pw(), guest: pw(), victim: pw() };
for (const [name, password] of Object.entries(users)) {
  try { await createUser(db, name + "-" + process.pid, password); } catch { /* left over from an earlier run */ }
}
const name = (n) => n + "-" + process.pid;

const rows = [];
const check = (ok, what, detail = "") => rows.push([ok ? "PASS" : "FAIL", what, String(detail)]);
const PAGE = "hungary/hungarian-basic-3/hungarian_basic_conversation_3_1.html";

async function call(method, path, { body, cookie, headers = {}, json = true, sendOrigin = true } = {}) {
  const h = { ...headers };
  if (cookie) h.cookie = cookie;
  if (method === "POST") {
    if (sendOrigin) h.origin = origin;
    if (json) h["content-type"] = "application/json";
  }
  const response = await fetch(base + path, { method, headers: h, body: body === undefined ? undefined : typeof body === "string" ? body : JSON.stringify(body), redirect: "manual" });
  const text = await response.text();
  let data = null;
  try { data = JSON.parse(text); } catch { /* not JSON */ }
  return { status: response.status, headers: response.headers, data, text };
}
const login = (user, password, extra = {}) => call("POST", "/api/login", { body: { username: name(user), password }, ...extra });
const cookieOf = (response) => (response.headers.get("set-cookie") ?? "").split(";")[0];

// --- not logged in
let r = await call("GET", "/api/me");
check(r.status === 200 && r.data.user === null, "no cookie: /api/me says nobody is logged in", JSON.stringify(r.data));
check(r.headers.get("cache-control") === "no-store" && /cookie/i.test(r.headers.get("vary") ?? ""), "that answer is never cached and depends on the cookie", `${r.headers.get("cache-control")} | Vary: ${r.headers.get("vary")}`);
r = await call("POST", "/api/progress", { body: { path: PAGE, done: true } });
check(r.status === 401, "marking a page without logging in is refused", r.status);

// --- wrong passwords look the same whether or not the name exists
const wrong = await login("philip", "not-the-password");
const unknown = await call("POST", "/api/login", { body: { username: "nobody-" + process.pid, password: "not-the-password" } });
check(wrong.status === 401 && unknown.status === 401 && JSON.stringify(wrong.data) === JSON.stringify(unknown.data), "wrong password and unknown name: the same status and the same words", `${wrong.status} ${unknown.status} ${wrong.data?.error}`);
check(!wrong.headers.get("set-cookie"), "a failed login sets no cookie");

// --- the locks on a POST
r = await login("philip", users.philip, { sendOrigin: false });
check(r.status === 403, "a login with no Origin header is refused", r.status);
r = await call("POST", "/api/login", { body: { username: name("philip"), password: users.philip }, headers: { origin: "http://evil.example" }, sendOrigin: false });
check(r.status === 403, "a login claiming to come from another site is refused", r.status);
r = await call("POST", "/api/login", { body: { username: name("philip"), password: users.philip }, headers: { origin: "http://systems.localhost:3004" }, sendOrigin: false });
check(r.status === 403, "even from another of OUR sites it is refused (each site has its own login)", r.status);
r = await call("POST", "/api/login", { body: `username=${name("philip")}&password=${users.philip}`, json: false, headers: { "content-type": "application/x-www-form-urlencoded" } });
check(r.status === 415, "a plain form post (not JSON) is refused", r.status);
r = await call("GET", "/api/login");
check(r.status === 405, "GET on the login address is not allowed", r.status);
r = await call("POST", "/api/login", { body: "x".repeat(10000) });
check(r.status === 400, "an oversized body is refused", r.status);
r = await call("POST", "/api/login", { body: { username: ["philip"], password: 1 } });
check(r.status === 400, "wrong types are refused, not crashed on", r.status);

// --- lockout
for (let i = 0; i < 5; i++) await login("victim", "wrong-" + i);
r = await login("victim", users.victim);
check(r.status === 429 && r.headers.get("retry-after") === "900", "after 5 wrong passwords even the right one is refused for a while", `${r.status}, Retry-After ${r.headers.get("retry-after")}`);
r = await login("guest", users.guest);
check(r.status === 200, "another name is not affected", r.status);
const guestCookie = cookieOf(r);

// --- logging in
r = await login("philip", users.philip);
const setCookie = r.headers.get("set-cookie") ?? "";
const cookie = cookieOf(r);
check(r.status === 200 && r.data.user.username === name("philip"), "the right password logs in", r.status);
check(/HttpOnly/i.test(setCookie) && /SameSite=Lax/i.test(setCookie) && /Path=\//.test(setCookie) && /Max-Age=2592000/.test(setCookie) && !/Domain=/i.test(setCookie), "the cookie is HttpOnly, SameSite=Lax, host-only, 30 days", setCookie.replace(/=[^;]{20,};/, "=<token>;"));
check(!/Secure/i.test(setCookie), "no Secure flag in development (plain http), expected", "");
check(!JSON.stringify(r.data).includes(cookie.split("=")[1]), "the token is not in the response body");

r = await call("GET", "/api/me", { cookie });
check(r.data.user?.username === name("philip") && JSON.stringify(r.data.done) === "[]", "/api/me with the cookie knows who it is", JSON.stringify(r.data).slice(0, 80));

// --- progress
r = await call("POST", "/api/progress", { body: { path: PAGE, done: true }, cookie });
check(r.status === 200, "a real page can be marked finished", r.status);
await call("POST", "/api/progress", { body: { path: PAGE, done: true }, cookie });
r = await call("GET", "/api/me", { cookie });
check(JSON.stringify(r.data.done) === JSON.stringify([PAGE]), "marking twice is still one", JSON.stringify(r.data.done));
for (const [path, label] of [["nonsense/page.html", "a path that is no page"], ["../../etc/passwd", "a path that climbs out"], ["linux/system-administration/anything.html", "a page of another site"], ["", "an empty path"]]) {
  r = await call("POST", "/api/progress", { body: { path, done: true }, cookie });
  check(r.status === 404, `${label} is refused`, r.status);
}
r = await call("POST", "/api/progress", { body: { path: PAGE, done: "yes" }, cookie });
check(r.status === 400, "done must be true or false", r.status);
r = await call("GET", "/api/me", { cookie: guestCookie });
check(JSON.stringify(r.data.done) === "[]", "another user does not see it", JSON.stringify(r.data.done));
r = await call("POST", "/api/progress", { body: { path: PAGE, done: false }, cookie: guestCookie });
r = await call("GET", "/api/me", { cookie });
check(r.data.done.length === 1, "and cannot undo it", JSON.stringify(r.data.done).length);
r = await call("POST", "/api/progress", { body: { path: PAGE, done: false }, cookie });
r = await call("GET", "/api/me", { cookie });
check(r.data.done.length === 0, "it can be undone by its owner", JSON.stringify(r.data.done));

// --- the static pages stay the same for everyone
await call("POST", "/api/progress", { body: { path: PAGE, done: true }, cookie });
const page = await fetch(`${base}/${PAGE.replace(/\.html$/, "")}`, { headers: { cookie } });
const html = await page.text();
check(!html.includes(name("philip")) && !html.includes("Mark as finished") && html.includes("Log in"), "a lesson page sent to a logged-in browser is the same logged-out page: nothing about the user", `${html.length} bytes`);
check(/s-maxage|max-age/.test(page.headers.get("cache-control") ?? "") && !/Set-Cookie/i.test([...page.headers.keys()].join()), "the page is cacheable and sets no cookie", page.headers.get("cache-control"));

// --- logging out
r = await call("POST", "/api/logout", { cookie });
check(r.status === 200 && /Max-Age=0/.test(r.headers.get("set-cookie") ?? ""), "logging out clears the cookie", r.headers.get("set-cookie")?.slice(0, 40));
r = await call("GET", "/api/me", { cookie });
check(r.data.user === null, "and the old cookie no longer works (the session is gone from the server)", JSON.stringify(r.data));
r = await call("POST", "/api/logout", { cookie, sendOrigin: false });
check(r.status === 403, "logging out from another site's page is refused", r.status);
r = await call("GET", "/api/me", { cookie: "lw_session=" + "A".repeat(500) });
check(r.status === 200 && r.data.user === null, "a made-up cookie is just 'nobody'", r.status);

for (const [status, what, detail] of rows) console.log(status, what.padEnd(88), detail.slice(0, 90));
console.log(`\n${rows.filter((x) => x[0] === "PASS").length} passed, ${rows.filter((x) => x[0] === "FAIL").length} failed`);

  35 passed, 0 failed
  (not logged in: /api/me says nobody; wrong password and unknown name give the same 401 and the same words, and no cookie; a login with no
  Origin, from another site, or from another of our sites: 403; a form post: 415; GET on login: 405; oversized body: 400; wrong types: 400;
  after 5 wrong passwords the right one gets 429 with Retry-After: 900, while another name is unaffected; the cookie has HttpOnly, SameSite=Lax,
  Path=/, Max-Age=2592000 and no Domain; the token is not in the body; /api/me with the cookie names the user; marking a page twice is one;
  four kinds of bad path are 404; "done" must be true or false; another user cannot see or undo it; the owner can undo it; logging out clears the
  cookie and the OLD cookie stops working at once; a logout from another site's page is refused; a made-up cookie is just "nobody".)

CAN IT FAIL? I took the same-origin check out of guardPost, rebuilt, and ran it again: 31 passed, 4 FAILED (a login with no Origin, from another
site, from another of our sites, and a logout from another site: each answered 200). Put back.

THE STATIC PAGES ARE UNCHANGED: a lesson page requested WITH a valid login cookie is byte for byte the page everyone gets (106,396 bytes, no
user name, no "Mark as finished", it still says "Log in"), is cacheable (s-maxage=31536000) and sets no cookie. This matters: if a page could
differ by user, a shared cache could show one person's page to another. Everything personal arrives through /api/me, which is never cached.

NOT TESTED: HTTPS (so the Secure flag, which is added only by a production build, was not seen on a real response); another process or a second
server sharing the database (SQLite handles several processes on one machine; a network drive is not safe for it); behind Apache. ONE THING TO
KNOW for Apache: the Origin check compares Origin with the Host header, so Apache must pass the original Host on (ProxyPreserveHost On), or every
login will be refused with 403. (Chapter 12.)

WHY THIS WORKS AS AN ANSWER
---------------------------
Every lock is a test that was made to fail, and the claim "the pages stay static" is checked by asking for a page as a logged-in user.