Monitoring & Maintenance

Chapter 8 — Monitoring & Maintenance

Security is not a state you reach — it's a process you maintain. Chapters 1 through 7 built the defences: encrypted connections, firewall rules, hardened SSH, security headers, malware scanning. This chapter closes the loop: making sure those defences stay current, that you're notified when something unusual happens, and that routine maintenance doesn't get missed. A server you're not watching is a server you don't control.

What this chapter covers: Why continuous monitoring matters after hardening. logwatch for daily human-readable log digests. unattended-upgrades for automatic security patches. Setting up msmtp as an email relay so the server can send alerts. Apache log analysis — top IPs, scanning patterns, suspicious requests. Auth log monitoring — successful logins and sudo usage. Disk and certificate expiry alerts. Cloudflare analytics as a complement to server logs. Regular maintenance checklist (weekly → monthly → quarterly). Full course security stack summary.

The Monitoring Gap

What you've built across 8 chapters — without monitoring: [Attacker] ──▶ UFW: blocked ← Ch.3 — working [Attacker] ──▶ SSH brute force: fail2ban ban ← Ch.4 — working [Attacker] ──▶ SSH key auth: rejected ← Ch.5 — working [Attacker] ──▶ Uploads web shell ← ClamAV should catch it [ClamAV] ──▶ finds shell.php at 3 AM ← Ch.7 scan found it [No alert] ──▶ nobody sees it for 3 days ← THE MONITORING GAP With monitoring: [ClamAV] ──▶ finds shell.php at 3 AM ← Ch.7 scan [Script] ──▶ emails emubantam@gmail.com ← you know within hours [logwatch] ──▶ morning email: "847 POST requests to /uploads at 2:59 AM" ← context [You] ──▶ investigate → find entry point → patch → done

Every tool in this course generates logs. The question is whether anyone reads them. Manually checking logs daily is unrealistic. The solution is to let the tools summarise the logs and surface anomalies — only alerting you when something needs attention.

Email Alerts — Setting Up msmtp

logwatch, unattended-upgrades, ClamAV scan scripts, and certificate renewal all want to send emails. A home server typically doesn't have a full mail server — msmtp relays outgoing mail through an existing account (Gmail works well).

Install msmtp and mailutils.

$ sudo apt install msmtp msmtp-mta mailutils -y # msmtp — the SMTP relay # msmtp-mta — makes msmtp the system sendmail (used by cron, logwatch, etc.) # mailutils — provides the "mail" command for testing

Create a Gmail App Password. Gmail requires an App Password (not your regular password) for SMTP access from non-Google apps.

Go to: myaccount.google.com → Security → 2-Step Verification → App passwords. Create a new app password for "Mail / Linux computer". Copy the 16-character code — you'll only see it once.

Create the msmtp config file.

$ sudo nano /etc/msmtprc

# /etc/msmtprc — system-wide msmtp config defaults auth on tls on tls_trust_file /etc/ssl/certs/ca-certificates.crt logfile /var/log/msmtp.log account gmail host smtp.gmail.com port 587 from emubantam@gmail.com user emubantam@gmail.com password abcd efgh ijkl mnop ← your 16-char App Password (spaces OK) account default : gmail

# Secure the config — it contains your App Password $ sudo chmod 600 /etc/msmtprc $ sudo chown root:root /etc/msmtprc

Test email sending.

$ echo "Test from webserver" | mail -s "msmtp test" emubantam@gmail.com # Check your inbox — it should arrive within a minute. # If it doesn't, check the log: $ sudo tail /var/log/msmtp.log host=smtp.gmail.com tls=on auth=on from=emubantam@gmail.com recipients=emubantam@gmail.com mailsize=... smtpstatus=250 smtpmsg='...' exitcode=EX_OK

msmtp-mta creates a symlink so that any program calling /usr/sbin/sendmail (what logwatch, cron, and unattended-upgrades all use) goes through msmtp. Once msmtp is configured, all the other tools work without any further changes.

logwatch — Daily Log Digest

logwatch parses your system logs overnight and emails you a human-readable summary — SSH activity, Apache traffic, package updates, disk events, fail2ban bans, and more. It turns a stack of machine-readable log files into a five-minute morning read.

$ sudo apt install logwatch -y # Run logwatch immediately to see what it produces (uses yesterday's logs by default) $ sudo logwatch --detail Med --range Yesterday --format text --output stdout | head -80 ################### Logwatch 7.8 (01/21/24) #################### Processing Initiated: Mon Jun 15 08:00:00 2026 Date Range Processed: yesterday ( 2026-Jun-14 ) Period is day. ################################################################## --------------------- Connections (secure-log) Begin ------------------------ Unmatched Entries sshd[1234]: Accepted publickey for philip from 192.168.1.5 port 54321 : 1 Time(s) ---------------------- Connections (secure-log) End ------------------------- --------------------- SSHD Begin ------------------------ Users logging in through sshd: philip: 192.168.1.5 (home-laptop): 3 times ...

# Configure logwatch to email you daily $ sudo nano /etc/logwatch/conf/logwatch.conf

# /etc/logwatch/conf/logwatch.conf # Only include settings you want to override from the defaults MailTo = emubantam@gmail.com MailFrom = webserver-logwatch@osztromok.com Detail = Med # Low / Med / High — Med is a good balance Range = Yesterday # report on yesterday's logs Format = text # text or html # logwatch runs nightly via cron (/etc/cron.daily/00logwatch) # No further setup needed — it runs automatically at ~6 AM

# Send a test report to your email (today's logs) $ sudo logwatch --detail Med --range Today --mailto emubantam@gmail.com # Check your inbox — arrives within a minute via msmtp

What to scan for in logwatch reports: SSH logins from unexpected IPs (anything not 192.168.1.x is worth noting). Large numbers of failed auth attempts that fail2ban didn't catch. Apache: unusually high 404 counts. Package manager: packages that failed to update (can indicate a signing key problem). Disk: filesystem nearing capacity.

Automatic Security Updates

Most compromises exploit known vulnerabilities — ones with published patches. Keeping security updates current is one of the highest-value maintenance tasks. unattended-upgrades applies security patches automatically, without waiting for you to log in and run apt upgrade.

$ sudo apt install unattended-upgrades -y # Enable automatic updates (creates the trigger files) $ sudo dpkg-reconfigure --priority=low unattended-upgrades # Select "Yes" when prompted # The two config files that control this: # /etc/apt/apt.conf.d/20auto-upgrades — controls WHEN to run # /etc/apt/apt.conf.d/50unattended-upgrades — controls WHAT to install

# Verify 20auto-upgrades has the right settings $ cat /etc/apt/apt.conf.d/20auto-upgrades APT::Periodic::Update-Package-Lists "1"; ← run apt update daily APT::Periodic::Unattended-Upgrade "1"; ← apply upgrades daily

# Edit 50unattended-upgrades for email and cleanup settings $ sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

# Key settings to configure in 50unattended-upgrades: // Email report when updates are applied (or fail) Unattended-Upgrade::Mail "emubantam@gmail.com"; Unattended-Upgrade::MailReport "on-change"; // only email if something changed // Remove packages that are no longer needed after upgrades Unattended-Upgrade::Remove-Unused-Dependencies "true"; // Automatic reboot after security kernel updates — set to false for a home server // (you want to control when the server reboots) Unattended-Upgrade::Automatic-Reboot "false"; // Log to syslog (logwatch will include this in its daily report) Unattended-Upgrade::SyslogEnable "true";

# Test the configuration — dry run (shows what would be upgraded, installs nothing) $ sudo unattended-upgrade --dry-run --debug Initial blacklisted packages: Initial whitelisted packages: Starting unattended upgrades script Allowed origins are: ... No packages found that can be upgraded unattended and no pending auto-removals # Check what the last run actually did $ sudo cat /var/log/unattended-upgrades/unattended-upgrades.log | tail -20

Automatic reboots: Some kernel security updates require a reboot to take effect. With Automatic-Reboot "false", the new kernel is installed but the old one keeps running. You'll see a "System restart required" message when you SSH in. Plan a manual reboot during off-peak hours when you see this message — sudo shutdown -r +5 "Rebooting for kernel update".

Apache Log Analysis

Apache's access and error logs contain a continuous record of everything hitting your server. Knowing what's normal helps you spot what isn't.

Access log patterns to watch for

# ── Top requesting IPs (spot scanners and heavy users) ──────────── $ sudo awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -rn | head -15 1847 162.158.32.1 ← Cloudflare IP — expected (your tunnel) 924 162.158.32.2 ← Another Cloudflare IP — expected 83 192.168.1.5 ← your laptop — expected 47 45.33.32.156 ← external IP with 47 requests — worth investigating # ── Most requested 404 paths (reveals what scanners are looking for) ── $ sudo grep " 404 " /var/log/apache2/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -15 43 /wp-admin/setup-config.php ← WordPress scanner (you're not running WP) 31 /phpMyAdmin/index.php ← phpMyAdmin scanner 28 /.env ← looking for exposed .env files 19 /admin/login ← generic admin panel scanner 3 /favicon.ico ← normal — browsers auto-request this # ── Suspicious POST requests ────────────────────────────────────── $ sudo grep '"POST' /var/log/apache2/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -10 847 /uploads/image.php ← POST to a PHP file in uploads — VERY suspicious 4 /contact.php ← form submissions — expected # ── Status code distribution (spot unusual error spikes) ────────── $ sudo awk '{print $9}' /var/log/apache2/access.log | sort | uniq -c | sort -rn 12847 200 ← normal responses 431 404 ← not found — expected some 124 301 ← redirects — expected 8 500 ← server errors — investigate if high

NORMAL Cloudflare IPs in access log

All public web traffic comes through Cloudflare's IPs — expected. With mod_remoteip (Chapter 4), real visitor IPs are logged in the %h field. Without it, you'd only see Cloudflare IPs.

INVESTIGATE Dozens of 404s for /wp-admin, /phpMyAdmin, /.env from one IP

Automated vulnerability scanner. Annoying but mostly harmless — your site doesn't have these paths. Consider adding the IP to fail2ban: sudo fail2ban-client set apache-noscript banip IP. fail2ban's apache-badbots jail should catch persistent scanners automatically.

ALERT POST requests to files in /uploads directory

Attempted web shell execution. Investigate immediately — find the request in detail, check if the uploaded file exists, run ClamAV scan on the uploads directory. The Apache config (Chapter 7) should return 403 for these — if they're getting through, the upload protection isn't working.

ALERT Spike in 500 errors coinciding with unusual requests

Could indicate a successful injection causing PHP/Apache errors. Check the Apache error log: sudo grep " 500 " /var/log/apache2/error.log | tail -20. Look for PHP errors, path traversal attempts, or unusual file paths in the error messages.

INVESTIGATE Sudden large traffic spike from a region you don't expect

Could be: legitimate traffic (your site was shared somewhere), DDoS attempt (Cloudflare handles this), or a scraped content farm. Check Cloudflare Analytics for the source breakdown. If DDoS, enable Cloudflare "Under Attack" mode temporarily.

Auth log monitoring

# All successful SSH logins (who, from where, when) $ sudo grep "Accepted" /var/log/auth.log Jun 15 09:14:22 webserver sshd[2341]: Accepted publickey for philip from 192.168.1.5 port 51234 ssh2 Jun 15 22:03:11 webserver sshd[3891]: Accepted publickey for philip from 203.0.113.45 port 49123 ssh2 # ↑ Second login is from an external IP — not your home network. # Is this expected (you SSHing in from work/phone)? # If not, treat as a security incident. # All sudo commands run (who ran what) $ sudo grep "COMMAND" /var/log/auth.log | tail -20 Jun 15 10:22:01 webserver sudo: philip : TTY=pts/0 ; PWD=/var/www ; USER=root ; COMMAND=/bin/nano config.php # New user accounts created (should be rare) $ sudo grep "new user" /var/log/auth.log # Any output here is worth investigating # Failed login attempts (fail2ban handles these, but useful for the count) $ sudo grep "Failed password" /var/log/auth.log | wc -l 847 ← 847 failed attempts since the log was last rotated — normal background noise

Disk Space and Certificate Expiry Alerts

Disk space alert script

# A web root disk filling up means Apache logs stop being written # (a security blind spot). Alert yourself before it happens. $ sudo nano /usr/local/bin/disk-alert.sh

#!/bin/bash # /usr/local/bin/disk-alert.sh # Email alert if any filesystem exceeds 80% usage THRESHOLD=80 EMAIL="emubantam@gmail.com" df -H | grep -vE '^Filesystem|tmpfs|cdrom|udev' | awk '{print $5 " " $1 " " $6}' | while read usage fs mount; do pct="${usage%%%}" if [ "$pct" -ge "$THRESHOLD" ]; then echo "Disk alert on $(hostname): $fs ($mount) is at $usage" | \ mail -s "DISK SPACE ALERT: $mount at $usage" "$EMAIL" fi done

$ sudo chmod +x /usr/local/bin/disk-alert.sh # Add to root's crontab — check daily at 8 AM $ sudo crontab -e

0 8 * * * /usr/local/bin/disk-alert.sh

Certificate expiry check

# Check current certificate expiry dates $ sudo certbot certificates Found the following certs: Certificate Name: osztromok.com Domains: osztromok.com www.osztromok.com Expiry Date: 2026-09-12 09:00:00+00:00 (VALID: 89 days) Certificate Path: /etc/letsencrypt/live/osztromok.com/fullchain.pem # The certbot renewal timer handles auto-renewal (Chapter 1) # But verify the timer is running: $ sudo systemctl status certbot.timer Active: active (waiting) $ sudo systemctl list-timers certbot.timer NEXT ACTIVATES Mon 2026-06-16 09:00:00 certbot.service # Check certificate from outside (verifies the live cert, not just what certbot says) $ echo | openssl s_client -connect osztromok.com:443 -servername osztromok.com 2>/dev/null | openssl x509 -noout -dates notBefore=Jun 14 09:00:00 2026 GMT notAfter=Sep 12 09:00:00 2026 GMT ← 89 days from now — healthy # Monthly cron to warn you if the cert renews less than 30 days before expiry # (catches scenarios where auto-renewal silently fails) $ sudo crontab -e

0 9 1 * * certbot certificates 2>&1 | grep -i "VALID:" | grep -v "VALID: [3-9][0-9] days\|VALID: [0-9][0-9][0-9]" && echo "Certificate expiring soon on $(hostname) — check certbot renewal" | mail -s "CERT EXPIRY WARNING" emubantam@gmail.com

Cloudflare Analytics

Cloudflare sees all traffic before it reaches your server — its analytics give you a perspective that Apache logs can't: traffic that was blocked before it even hit the tunnel, geographic breakdowns, and threat intelligence.

Analytics → Traffic: total requests, cached vs uncached, unique visitors, top countries. Baseline this — a sudden spike from an unexpected country may indicate a targeted scan.
Security → Events: every request Cloudflare blocked — WAF rules triggered, bot fight mode blocks, rate limiting hits. These are attacks that never reached Apache.
DNS → Analytics: DNS query volume — useful for detecting DNS-based probing or amplification if you've had a public IP leak.
Speed → Performance: cache hit rate — higher is better; frequently requested static assets (CSS, images) should cache at Cloudflare and never reach Apache.

Cross-reference Cloudflare Security Events with your Apache logs. If Cloudflare shows 500 blocked requests from a specific IP and your Apache logs show the same IP made 3 requests that got through — those 3 succeeded despite Cloudflare's block, which means they came via a direct path (not the tunnel). Investigate how they bypassed Cloudflare.

Regular Maintenance Checklist

Daily Handled automatically — just read the emails

logwatch report — scan the morning email. Look for: unexpected SSH logins, POST requests to unusual paths, high 404 counts, packages that failed to update.

ClamAV scan result — the scan script emails you only on detection. No email = nothing found.

unattended-upgrades — emails you when security packages are updated. If you see a kernel update, plan a reboot.

Weekly 5–10 minute check

fail2ban stats: sudo fail2ban-client status sshd — are bans happening? Any single IP banned repeatedly (persistent attacker)?

Disk space: df -h — confirm nothing is nearing 80%. Web root, /var/log, and /var/lib/mysql are the common culprits.

Running services: sudo systemctl status apache2 clamav-freshclam clamav-daemon fail2ban cloudflared — all should be active (running).

New files in web root: sudo find /var/www/ -newer /var/www/osztromok.com/index.html -type f | head -10 — any unexpected new files?

Monthly 20–30 minute review

Full manual apt upgrade: sudo apt update && sudo apt upgrade — applies non-security updates that unattended-upgrades skips. Review packages before confirming.

Certificate expiry: sudo certbot certificates — should show 60+ days remaining. If under 30, investigate why auto-renewal hasn't run.

rkhunter scan: sudo rkhunter --check --skip-keypress — check for rootkits and backdoors. Update first: sudo rkhunter --update.

lynis audit: sudo lynis audit system — check for configuration drift. Score should stay stable or improve over time.

Review authorized_keys: cat ~/.ssh/authorized_keys — do you recognise every key? Remove any from machines you no longer use.

Cloudflare Security Events: review blocked threats in Cloudflare dashboard — anything targeting specific paths warrants adding an Apache deny rule.

After updates / changes Verify nothing broke

After Apache config changes: sudo apache2ctl configtest then sudo systemctl reload apache2, then curl your site.

After kernel update reboot: check all services are running — Apache, fail2ban, clamav-daemon, cloudflared.

After adding new PHP application: run ClamAV scan on the new files, check Apache logs for unusual requests within the first 24 hours.

After rkhunter propupd: only run rkhunter --propupd after intentional system changes (package upgrades) — it updates the baseline, so running it after a compromise hides the evidence.

Troubleshooting

logwatch is not sending email / emails go to spam

Test msmtp first: echo "test" | mail -s "test" emubantam@gmail.com — if this also fails, the problem is msmtp, not logwatch. Check sudo tail /var/log/msmtp.log for SMTP errors. If emails arrive but go to spam: the From: address doesn't match the sending domain — set MailFrom in logwatch.conf to your Gmail address, not a custom domain. Gmail's spam filter is more lenient with mail from the same account.

unattended-upgrades is not running — no log entries

Check the trigger: cat /etc/apt/apt.conf.d/20auto-upgrades — both Update-Package-Lists and Unattended-Upgrade should be "1". Check the service: sudo systemctl status apt-daily-upgrade.timer and sudo systemctl status apt-daily.timer — both should be active. Check the log: sudo cat /var/log/unattended-upgrades/unattended-upgrades.log — even "no upgrades available" is a valid log entry that confirms it ran.

logwatch report shows SSH logins from an IP I don't recognise

Don't panic — first rule out legitimate sources: is it a VPN exit node you use? A mobile data IP? Use whois IPADDRESS or an IP lookup tool to identify the organisation. Check when the login occurred and what was done after: sudo grep "IPADDRESS" /var/log/auth.log for the full authentication record, then sudo grep "COMMAND" /var/log/auth.log for any sudo usage around that time. If it's genuinely unexpected and key-based auth succeeded (not just an attempt), change your authorized_keys immediately and investigate which machine's private key may be compromised.

Disk alert fires but df shows plenty of space

The disk-alert script checks all mounted filesystems. The alert may be for a small filesystem you're not thinking of — check: df -H — look at /boot (old kernels accumulate here), /var/log (log rotation may be misconfigured), or Docker's overlay filesystem if Docker is installed. Clean old kernels: sudo apt autoremove. Set up log rotation properly: sudo logrotate --force /etc/logrotate.d/apache2.

Course Summary — Your Complete Security Stack

Here is everything you've built across all eight chapters — a layered security posture that turns a default Apache install into a hardened, monitored web server.

Ch. 1 HTTPS / Let's Encrypt DNS-01 challenge via Cloudflare plugin bypasses Virgin Media's port block. 90-day certificates with automatic renewal via systemd timer. Full (Strict) SSL mode in Cloudflare for end-to-end encryption through the tunnel.

Ch. 2 Enforcing HTTPS HTTP redirects to HTTPS in Apache. HSTS header with escalating max-age. mod_rewrite with X-Forwarded-Proto header for Cloudflare compatibility — avoiding the redirect loop trap.

Ch. 3 UFW Firewall Default-deny inbound. Only SSH open to the network — web traffic arrives via the Cloudflare Tunnel (outbound, not subject to inbound rules). ufw limit ssh for rate limiting. UFW logging enabled.

Ch. 4 fail2ban SSH jail with 24h bans, Apache jails with mod_remoteip for real visitor IPs, escalating bans via bantime.increment. Home subnet whitelisted in ignoreip. Covers the gap between UFW's rate limit and persistent attackers.

Ch. 5 SSH Hardening Ed25519 key pair per device. PasswordAuthentication no — brute-force is mathematically infeasible. PermitRootLogin no. AllowUsers whitelist. Drop-in file in sshd_config.d/ for clean, upgrade-safe configuration.

Ch. 6 Security Headers ServerTokens Prod hides version. X-Frame-Options against clickjacking. X-Content-Type-Options against MIME confusion. Referrer-Policy for URL privacy. Content Security Policy in Report-Only mode, ready to enforce.

Ch. 7 ClamAV & Malware Daily scheduled scan of the web root with quarantine for finds. Apache upload directories: PHP execution blocked, AllowOverride None prevents .htaccess bypass. Supplemented by rkhunter (rootkits) and AIDE (file integrity baseline).

Ch. 8 Monitoring logwatch daily digest by email. unattended-upgrades for automatic security patches. msmtp relay for all server alerts. Apache and auth log analysis patterns. Disk and certificate expiry alerts. Regular maintenance schedule.

Securing Your Web Server — Complete

osztromok.com now runs behind eight layers of active defence. Traffic is encrypted end-to-end. The network firewall admits nothing it shouldn't. Brute-force attacks hit a wall at fail2ban. SSH accepts no passwords at all. The browser is told exactly what the server permits. Malware is scanned nightly. And every anomaly generates an alert before it becomes a problem.

Security is a practice, not a destination. The maintenance checklist in this chapter is the ongoing contract: read the daily emails, apply the monthly checks, and the stack stays sharp.

Quick Reference — Chapter 8

Command	Purpose
sudo logwatch --detail Med --range Today --mailto EMAIL	Run logwatch immediately and email the report
sudo unattended-upgrade --dry-run --debug	Test unattended-upgrades without installing anything
cat /var/log/unattended-upgrades/unattended-upgrades.log	Review what was automatically updated and when
echo "test" \| mail -s "test" EMAIL	Test msmtp email relay is working
sudo awk '{print $1}' /var/log/apache2/access.log \| sort \| uniq -c \| sort -rn \| head -15	Top requesting IPs — spot scanners and unusual sources
sudo grep " 404 " /var/log/apache2/access.log \| awk '{print $7}' \| sort \| uniq -c \| sort -rn \| head -15	Most common 404 paths — reveals what scanners are probing for
sudo grep "Accepted" /var/log/auth.log	All successful SSH logins — verify all are expected
sudo grep "COMMAND" /var/log/auth.log	All sudo commands run — audit admin activity
sudo certbot certificates	Check certificate expiry dates and renewal status
df -h	Disk space — watch /var/log and /var/www for growth
sudo systemctl status apache2 fail2ban clamav-daemon cloudflared	Quick service health check — all should show active (running)