Writing to Logs: Existing Files vs. Custom Log Files

Logging & Log Analysis

Chapter 9 · Writing to Logs: Existing Files vs. Custom Log Files

Every chapter so far has been about reading logs someone else's system already produced. This one flips the direction: writing them yourself — either adding to a log that already exists, or building a well-behaved new one from scratch.

Two Different Situations

Appending to an existing logCreating a custom log file
WhenAdding to an application's own established log streamA standalone script or tool with nothing existing to append to
Main riskBreaking a format other tools already depend onChoosing a bad location, or forgetting rotation entirely
Key disciplineMatch the existing conventions exactlyEstablish good conventions from the start

Appending to an Existing Log Correctly

An application's log file is often already being read by other tools — dashboards, alerting rules, or simply a colleague's own habitual grep commands. Writing a new line into it that doesn't match the existing format (different timestamp style, missing the usual level tag, fields in a different order) can silently break every one of those downstream consumers, even though the new line is perfectly readable to a human.

  • Use the application's own logging mechanism where one exists, rather than writing raw text to the file directly — it already knows the correct format, and often already handles rotation and concurrent-write safety for you
  • Match level conventions from Chapter 2 — don't introduce a new ad-hoc severity scheme into a file that already has an established one
Concurrent writes without locking can corrupt log lines
If two processes write to the same file at the same moment without any coordination, their output can interleave mid-line — producing a genuinely corrupted, unparseable entry that's neither one process's line nor the other's. This is exactly why using an established logging mechanism (which typically handles this safely) is preferable to raw file writes from multiple sources.

Creating a Custom Log File

For a standalone script or tool with no existing log to append to, a simple, self-written logging function is often enough:

log() { local level="$1"; shift echo "$(date '+%Y-%m-%d %H:%M:%S') [$level] $*" >> /var/log/myapp/custom.log } log INFO "Backup completed successfully"

Two things this small function already gets right, on purpose: a consistent, sortable timestamp format, and a bracketed level tag matching Chapter 2's own convention — the same discipline that matters for an application's existing log applies just as much to a brand-new one.

Location matters too — a system-wide tool conventionally logs under /var/log/, in its own subdirectory, with permissions that match who's actually allowed to read or write it, rather than being placed wherever happens to be convenient at the time.

The Danger of Unbounded Growth

Chapter 1 warned against deleting a log file mid-incident to reclaim disk space. This is the other half of that same problem, addressed in advance: a custom log file with no rotation in place will simply grow forever, until it eventually does fill the disk on its own — turning "we might need to free some space one day" into a real, urgent incident.

logrotate: Automating the Solution

logrotate is the standard Linux tool for automatically rotating, compressing, and eventually discarding old log files on a schedule — it's what actually produces the access.log.1 and access.log.2.gz files Chapter 3 mentioned in passing.

/* /etc/logrotate.d/myapp */ /var/log/myapp/custom.log { daily rotate 14 compress missingok notifempty create 0640 myapp myapp }
DirectiveWhat it does
dailyRotate once a day (also commonly weekly or size-based)
rotate 14Keep 14 old rotated copies before deleting the oldest
compressGzip rotated files once they're no longer the active log
missingokDon't error out if the log file happens to be missing
notifemptySkip rotating a file that's currently empty
create 0640 myapp myappRecreate the active log file with these permissions and ownership immediately after rotating
Test a logrotate config before trusting it live
logrotate -d /etc/logrotate.d/myapp runs in debug mode — it shows exactly what logrotate would do without actually touching any files, letting you confirm the configuration behaves as intended before it runs for real on a schedule.

Hands-On Exercises

Exercise 1

A developer adds a new line format to an existing application log, using their own preferred timestamp style instead of the file's established one. Explain what this chapter says could go wrong, even though the new line is still readable to a human.

📄 View solution
Exercise 2

Explain how this chapter's warning about unbounded log growth relates to Chapter 1's own warning about deleting logs mid-incident — how are they two sides of the same underlying problem?

📄 View solution
Exercise 3

Explain what rotate 14 and compress each do in a logrotate config, and why testing with logrotate -d before relying on a new config is good practice.

📄 View solution

Chapter 9 Quick Reference

  • Appending to an existing log: match its format exactly, use the application's own logging mechanism where possible, respect Chapter 2's level conventions
  • Concurrent, uncoordinated writes to one file can interleave and corrupt lines — a real reason to prefer an established logging mechanism over raw writes
  • A custom log file needs a consistent format, a sensible location (typically under /var/log/), and correct permissions from the start
  • Unbounded log growth is the same underlying risk as Chapter 1's "don't delete logs mid-incident" warning, addressed in advance rather than reacted to
  • logrotate automates rotation, compression, and eventual deletion — daily/rotate N/compress/create are the core directives
  • Test any new logrotate config with logrotate -d before trusting it to run unattended
  • Next chapter: Capstone — Triaging Three Real Support Tickets