Log Aggregation & Centralized Logging

Chapters 6 and 7 both worked because everything lived on one server, or a small enough handful that checking each one by hand was realistic. This chapter is a conceptual orientation to what happens once that stops being true — not a deep implementation guide, but enough to recognize the pieces and know what they're for.

The Problem: When "SSH Into Each Server" Stops Working

Two things break the manual approach as a system grows:

• Scale. Checking one server's logs by hand is fine. Checking the same thing across dozens or hundreds of servers, one at a time, isn't — the time cost alone makes it impractical during an active incident
• Ephemerality. In containerized and cloud environments, individual containers or instances are routinely destroyed and replaced rather than kept running indefinitely. A log that only exists on a container that no longer exists is a log that's gone permanently — Chapter 1's "silent failure" problem, but caused by the infrastructure itself rather than a missing log statement

The Core Idea: Ship Logs Somewhere Central

Rather than leaving each log where it was generated, a copy is forwarded to a central location as it's written. This solves both problems at once: one place to search instead of many, and logs that outlive the server or container that produced them.

syslog & rsyslog Forwarding

The syslog protocol Chapter 2 already introduced was designed to support forwarding to a remote collector, not just writing to a local file. rsyslog, the syslog daemon most Linux distributions actually run today, can be configured to forward matching log entries to a remote server with a small addition to its own config:

The double @@ specifically means TCP (a single @ means UDP) — a small syntax detail, but one worth getting right, since UDP forwarding can silently drop log messages under load with no error at all.

A Conceptual Tour: The ELK Stack

One of the most common centralized-logging stacks in real production use, built from three (or more) cooperating tools:

A Conceptual Tour: Grafana Loki

A more lightweight alternative, built around a genuinely different tradeoff: rather than indexing the full text of every log line the way Elasticsearch does, Loki indexes only a small set of labels (like which service or server a log came from), storing the actual log text compressed and unindexed. That's a real, deliberate resource-usage tradeoff — cheaper to run at scale, at the cost of full-text search being somewhat less immediate than in an ELK-style setup. Promtail is Loki's typical log-shipping agent, and Grafana — already familiar to many teams from metrics dashboards — is the usual way to query and view Loki's own data.

What Doesn't Change

Hands-On Exercises