Two support engineers get the same ticket: "the site is slow." One opens a browser, clicks around, shrugs, and starts guessing — maybe it's the database, maybe it's the network, maybe it's just Tuesday. The other opens the logs first. This course is entirely about becoming the second engineer — not because guessing never works, but because logs turn "maybe" into evidence, and evidence is what actually gets a ticket closed correctly the first time.

The Guessing Trap

Guessing feels faster than it is. A plausible-sounding theory — "it's probably the database" — sends you off to check database metrics, find nothing obviously wrong, move on to the next guess, and repeat. Each wrong guess costs real time and produces no evidence toward the actual answer. Worse, a confident wrong guess can lead to a wrong fix — restarting a service that was never the problem, "just in case" — which doesn't just fail to help, it can actively muddy the diagnostic trail for whoever looks at this next.

The log-first habit inverts this. Instead of theorizing and then checking, you check first and let what you find narrow the theories that are even worth considering. It's slower to start — reading log output isn't as immediately satisfying as taking an action — but it's dramatically faster to actually finish, because every minute spent reading logs is a minute spent on real evidence, not a guess that might be completely wrong.

What a Log Actually Is

Strip away the specifics of any particular system, and a log is simple: a timestamped record of an event a system was told to record. A web server logs each request it receives. An authentication system logs each login attempt. An application logs errors when something goes wrong internally. Different systems, same underlying idea.

Even without knowing this exact format yet — Chapter 4 covers it in full — a few things are already readable: something happened at a specific moment (14:32:07), it involved a request to /login, and the outcome was a 500, a server-side error. That's already more than a guess would have given you for free.

What Logs Can Actually Tell You

What Logs Can't Tell You — An Honest Limit

Logs are not a complete recording of everything that happened; they're a recording of everything someone chose to record. That distinction matters more than it sounds:

• Silent failures aren't logged at all. If nobody wrote a log line for a particular failure case, it simply won't appear — the absence of an error in the logs is not proof nothing went wrong
• A single log line is often missing context. One line might tell you a request failed, but the actual cause could be recorded in a completely different system's log, at roughly the same timestamp — Chapters 6 and 7 both depend on this kind of cross-referencing
• Logs can be actively misleading if misconfigured. A server with its clock set to the wrong timezone will log technically accurate events at confusingly wrong-looking times; an overly generic error message can point you toward the wrong subsystem entirely

A Concrete Example: One Symptom, Three Different Causes

A user reports "the site gave me an error." That single symptom could mean genuinely different things underneath — a failed database connection, a bug in the application code, or the server running out of memory under load — and each of those has a completely different fix. Without logs, you're choosing between three guesses. With logs, the error message, the timing, and what else was happening at that exact moment usually point clearly at just one of them. Chapters 6 and 7 build this exact skill in full, working through a slow-response scenario and an incorrect-response scenario end to end.

What This Course Covers

Log levels and severity, where to actually find logs on Linux and Windows, reading web server and authentication logs in real depth, two full diagnostic walkthroughs built around genuinely common support scenarios, an orientation to log aggregation at scale, and — closing the loop — how to write to logs yourself, both appending to an existing application's log and building a well-behaved custom one. The capstone ties all of it together against three realistic support tickets.

Hands-On Exercises

Chapter 1 established that a log only records what someone chose to record. Severity levels are the mechanism behind that choice — a way of tagging each log line with how much it actually matters, so both the system generating logs and the person reading them can filter signal from noise.

The Standard Levels, Quietest to Loudest

These five names are the most common convention across application-level logging, but they're a convention, not a universal law — some systems use slightly different names or collapse two of these into one. The relative ordering (quietest/least severe to loudest/most severe) is the part that stays consistent.

Why Levels Exist: Filtering Signal From Noise

Almost every logging system lets you configure a minimum level to actually output or store. Set the minimum to WARN, and every DEBUG and INFO line is silently dropped — only warnings and worse get through. This is deliberate: a healthy, correctly-behaving system genuinely doesn't need to permanently store a line for every single normal event, but it does need a way to see everything in detail the moment something is being actively investigated.

The Real-World Mistake: One Level for Everything

Levels only help if they're used with any discipline. Two genuinely common failure patterns undo the entire benefit:

• Everything logged as INFO (or worse, as ERROR). If routine, expected events are logged at the same level as genuine failures, filtering by level stops working — you're back to reading everything, exactly the noise levels exist to avoid. A classic version of this: logging "user not found" as ERROR during a normal login attempt, when a mistyped password is completely routine, not a system failure.
• Running with DEBUG enabled permanently in production. Debug-level detail is enormously verbose by design — enabling it around the clock produces massive log volume, real disk and performance overhead, and buries the genuinely important lines even deeper than having no levels at all would.

A Practical Example: The Same Situation, Two Different Levels

Same underlying condition — a database connection that isn't responding — logged at two different levels because the outcome was different. The first line describes something that recovered on its own and needed no action; the second describes an operation that genuinely failed. Reading only the level, before even reading the rest of the message, already tells you which of these two deserves attention right now.

Syslog's Own Scheme: Eight Levels, Not Five

Linux's own system-wide logging convention, syslog, predates most application-level logging frameworks and uses a finer-grained, numbered scale — worth knowing before Chapter 3 covers where these logs actually live.

Lower numbers mean higher severity — the opposite direction application-level logging usually reads in, but the same underlying "how much does this matter" idea. You'll see these numeric values directly in real configuration, such as Apache's own LogLevel warn directive or a syslog facility/severity pairing like mail.crit.

Hands-On Exercises

Knowing what a log is and how severity works doesn't help much if you can't actually locate one. This chapter is a practical map: where Linux keeps its logs, the modern systemd alternative to plain log files, how Windows does the equivalent job very differently, and what to do when a log genuinely isn't where the defaults say it should be.

Linux: The /var/log/ Directory

On most Linux systems, /var/log/ is the traditional home for both system-level and application-level log files. A handful of names show up constantly:

The systemd Journal — journalctl

Most modern Linux distributions run systemd as their init system, and systemd includes its own logging component, journald, which stores structured, binary log data rather than plain text files. It's queried with the journalctl command, not a text editor or cat.

Windows: Event Viewer

Windows takes a genuinely different approach from plain text files: the Windows Event Log service stores structured events, each with an Event ID, a source, and a severity level, browsable through the Event Viewer GUI (eventvwr.msc). The three logs worth knowing first:

For a command-line equivalent to browsing Event Viewer's GUI, Get-WinEvent (or the older Get-EventLog) does the same job from PowerShell — see PowerShell Fundamentals and PowerShell Intermediate/Advanced for real depth on scripting against these logs directly.

Application-Specific Log Locations

When the Log Isn't Where You Expect

Defaults are defaults, not guarantees. When a log file genuinely isn't in its usual location, a few things are worth checking before assuming it doesn't exist at all:

• The application's own configuration. Most services let you set a custom log path — the config file is the actual source of truth, not the convention
• The systemd unit file for that service, if it's managed by systemd — StandardOutput/StandardError directives can redirect a service's output somewhere other than the journal entirely
• Log rotation. The file you're looking for might have already been rotated — renamed to access.log.1, or compressed to access.log.2.gz — Chapter 9 covers exactly how and why this happens

Hands-On Exercises

Chapter 3 found the files. This chapter reads what's actually in them — the access log's field-by-field structure, what HTTP status codes really tell you, a genuinely easy-to-miss gap in the default format, and the error log that fills that gap in. Apache and Nginx are covered together throughout, since their conventional log formats are close enough to read with one shared mental model.

The Access Log: Common & Combined Format

Both Apache and Nginx default to a format descended from the same standard, Common Log Format (CLF), almost always extended in practice to Combined Log Format, which adds two more fields. A single line looks like this:

HTTP Status Codes: Reading the Middle Field

That 4xx/5xx split matters for triage: a spike in 404s often means a broken link or a bot scanning for pages that don't exist — rarely a real system problem. A spike in 5xx codes means the server itself is failing to do its job, and is exactly the kind of signal Chapters 6 and 7 build a full diagnostic process around.

The Gap Almost Everyone Assumes Isn't There: Response Time

With %D added, the same request from earlier might look like this — the final field, 4523912, is microseconds, meaning this request took roughly 4.5 seconds:

The Error Log: Where the "Why" Often Lives

The access log tells you what request came in and what status went out. It rarely tells you why a request failed — that's the error log's job.

Both examples describe the same underlying situation — the web server couldn't reach a backend it depends on — from Apache's PHP module and Nginx's own upstream-proxy perspective respectively. Neither the Apache nor the Nginx access log line for this same request would show anything more specific than a 500 or 502; the error log is what actually names the cause.

Reading Both Logs Together

The real technique, used throughout Chapters 6 and 7, is straightforward once both formats are familiar: find the access log line for the problem request, note its exact timestamp, then search the error log for entries at that same moment. The access log confirms that something failed and roughly how; the error log — when the failure produced one — explains why.

Hands-On Exercises

"A user can't log in" has been this course's own running example since Chapter 1. This chapter finally works through it properly — not as one problem, but as four genuinely different layers a request passes through, each with its own log source, each capable of being the actual point of failure.

The Diagnostic Order: Outside In

Working outside-in matters: checking layer 3 first, when the real problem is layer 2, means reading application logs that were never going to show anything, because the request never reached the application to log anything at all.

Layer 1: Never Reaching the Server

If a user reports being unable to log in and nothing in any log on this server — not the access log, not the error log, not the application's own log — shows any trace of a request from them at all, the request most likely never arrived. A DNS problem, a network routing issue, or a client-side connectivity problem on the user's own end can all produce exactly this symptom. This server's own logs, by definition, cannot record a request that never reached it.

Layer 2: Blocked Before the Application

A very common real-world tool here is fail2ban, which watches logs (often auth.log itself) for repeated failed attempts and automatically firewalls off the offending IP for a period of time. Its own actions are logged separately:

Once banned, that IP's later requests are rejected at the firewall — before Apache or Nginx ever sees them, meaning they won't appear in the access log either. A legitimate user who mistyped their password a few times too many can end up banned by exactly the same mechanism meant to stop an attacker; fail2ban-client status <jail> (or the ban log itself) is how you tell the two apart.

A softer version of the same layer: HTTP basic authentication configured directly on the web server (via .htaccess or an equivalent) rejects a request with a 401 before the application underneath ever runs — this does appear in the access log, just without any corresponding application-level log entry.

Layer 3: Authentication Failing in the Application

This is the layer most people assume is the whole problem, and it does cover the most common real causes:

• Wrong credentials — routine, per Chapter 2's own example; logged at INFO, not ERROR, in a well-configured application
• Account locked or disabled — a deliberate security feature after too many failed attempts on one specific account, distinct from fail2ban's IP-level ban
• Account doesn't exist — a typo'd username, or a genuinely deleted/never-created account
• Session or token expired — the user was previously logged in, but their session has since lapsed

Layer 4: Authenticated Successfully, Then Something Breaks

The rarest but most confusing case: the application's own log shows a genuinely successful authentication, but the user still ends up unable to log in — because something after that succeeded step fails. A session store that's unreachable (Redis down, disk full for file-based sessions) or a broken post-login redirect are both real examples. The tell is a successful auth log line immediately followed by an apparently unrelated error somewhere else in the same request's timeframe — exactly the kind of cross-log correlation Chapter 4 introduced.

Hands-On Exercises

Every chapter so far has built one piece: severity levels, log locations, access/error log fields, response-time logging, cross-log correlation. This chapter puts all of it to work on one realistic ticket, start to finish — the "Apache running slow" scenario this course has referenced since Chapter 1.

The Ticket

"Customers are reporting the checkout page is taking 10+ seconds to load. Started sometime this morning. Nothing else seems affected."

Hands-On Exercises

Chapter 6 was one thread of evidence, layered until it pointed at a single cause. This chapter is different on purpose: four genuinely plausible causes, each checked and eliminated with real evidence, until only one survives. "Wrong response" problems are rarely obvious from the symptom alone — this is the second half of the user's own original example, worked through directly.

The Ticket

"The product page shows different prices for different users — some see today's updated prices, some still see yesterday's. Started right after this morning's price update went out."

Four candidate explanations are worth considering before touching anything: a bug in the update itself, a caching layer serving stale content, a misconfigured virtual host serving from the wrong source, or a backend that didn't receive the update. The temptation is to jump straight to the one that sounds most familiar — here, that's caching, since "different users see different things" is the classic caching fingerprint. This walkthrough deliberately doesn't skip the other three just because one guess feels obviously right.

Hands-On Exercises

Chapters 6 and 7 both worked because everything lived on one server, or a small enough handful that checking each one by hand was realistic. This chapter is a conceptual orientation to what happens once that stops being true — not a deep implementation guide, but enough to recognize the pieces and know what they're for.

The Problem: When "SSH Into Each Server" Stops Working

Two things break the manual approach as a system grows:

• Scale. Checking one server's logs by hand is fine. Checking the same thing across dozens or hundreds of servers, one at a time, isn't — the time cost alone makes it impractical during an active incident
• Ephemerality. In containerized and cloud environments, individual containers or instances are routinely destroyed and replaced rather than kept running indefinitely. A log that only exists on a container that no longer exists is a log that's gone permanently — Chapter 1's "silent failure" problem, but caused by the infrastructure itself rather than a missing log statement

The Core Idea: Ship Logs Somewhere Central

Rather than leaving each log where it was generated, a copy is forwarded to a central location as it's written. This solves both problems at once: one place to search instead of many, and logs that outlive the server or container that produced them.

syslog & rsyslog Forwarding

The syslog protocol Chapter 2 already introduced was designed to support forwarding to a remote collector, not just writing to a local file. rsyslog, the syslog daemon most Linux distributions actually run today, can be configured to forward matching log entries to a remote server with a small addition to its own config:

The double @@ specifically means TCP (a single @ means UDP) — a small syntax detail, but one worth getting right, since UDP forwarding can silently drop log messages under load with no error at all.

A Conceptual Tour: The ELK Stack

One of the most common centralized-logging stacks in real production use, built from three (or more) cooperating tools:

A Conceptual Tour: Grafana Loki

A more lightweight alternative, built around a genuinely different tradeoff: rather than indexing the full text of every log line the way Elasticsearch does, Loki indexes only a small set of labels (like which service or server a log came from), storing the actual log text compressed and unindexed. That's a real, deliberate resource-usage tradeoff — cheaper to run at scale, at the cost of full-text search being somewhat less immediate than in an ELK-style setup. Promtail is Loki's typical log-shipping agent, and Grafana — already familiar to many teams from metrics dashboards — is the usual way to query and view Loki's own data.

What Doesn't Change

Hands-On Exercises

Every chapter so far has been about reading logs someone else's system already produced. This one flips the direction: writing them yourself — either adding to a log that already exists, or building a well-behaved new one from scratch.

Two Different Situations

Appending to an Existing Log Correctly

An application's log file is often already being read by other tools — dashboards, alerting rules, or simply a colleague's own habitual grep commands. Writing a new line into it that doesn't match the existing format (different timestamp style, missing the usual level tag, fields in a different order) can silently break every one of those downstream consumers, even though the new line is perfectly readable to a human.

• Use the application's own logging mechanism where one exists, rather than writing raw text to the file directly — it already knows the correct format, and often already handles rotation and concurrent-write safety for you
• Match level conventions from Chapter 2 — don't introduce a new ad-hoc severity scheme into a file that already has an established one

Creating a Custom Log File

For a standalone script or tool with no existing log to append to, a simple, self-written logging function is often enough:

Two things this small function already gets right, on purpose: a consistent, sortable timestamp format, and a bracketed level tag matching Chapter 2's own convention — the same discipline that matters for an application's existing log applies just as much to a brand-new one.

Location matters too — a system-wide tool conventionally logs under /var/log/, in its own subdirectory, with permissions that match who's actually allowed to read or write it, rather than being placed wherever happens to be convenient at the time.

The Danger of Unbounded Growth

Chapter 1 warned against deleting a log file mid-incident to reclaim disk space. This is the other half of that same problem, addressed in advance: a custom log file with no rotation in place will simply grow forever, until it eventually does fill the disk on its own — turning "we might need to free some space one day" into a real, urgent incident.

logrotate: Automating the Solution

logrotate is the standard Linux tool for automatically rotating, compressing, and eventually discarding old log files on a schedule — it's what actually produces the access.log.1 and access.log.2.gz files Chapter 3 mentioned in passing.

Hands-On Exercises

Nine chapters built the pieces — levels, locations, formats, correlation, layered diagnosis, aggregation, writing your own. This capstone applies all of it to three fresh tickets, worked more briskly than Chapters 6 and 7's own dedicated walkthroughs, since the underlying discipline should already feel familiar by now.

Ticket 1: "I can't log into the admin panel"

An internal support-team member reports being unable to log into the admin panel since this morning, insisting their password is correct.

Ticket 2: "The reporting dashboard is really slow"

Export generation on the reporting dashboard has been taking 20+ seconds since yesterday afternoon.

Ticket 3: "EU customers are seeing US content"

Customers configured for the EU region occasionally see US-region homepage content instead.

Chapter Attribution

Honest Scope Note

Hands-On Exercises