Capstone — Building a Real Storage Stack

Linux Filesystems

Chapter 11 · Capstone — Building a Real Storage Stack

Ten chapters of concepts and commands, now combined into one real build: a redundant RAID 10 array, LVM on top of it, ext4 formatted and mounted, a live snapshot taken, and then a deliberate, simulated disk failure and replacement — proving the theory from every prior chapter for real, in order.

The Plan

Four disks (/dev/sdb, /dev/sdc, /dev/sdd, /dev/sde), assembled into RAID 10 — chosen per fs1-7/fs1-10's own recommendation for fast, low-risk rebuilds and predictable performance. LVM sits on top for live resize flexibility (fs1-5/fs1-9's own traditional-stack pattern), formatted with ext4 (fs1-2, validated by fs1-10's decision framework for this kind of general-purpose workload). /boot itself would live on its own small, plain partition outside this array entirely, per fs1-5's own GRUB-compatibility guidance — not built here in detail, since partitioning and GRUB itself were already covered in full in grub1.

Step 1 — Building the RAID 10 Array

mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 mdadm --detail --scan >> /etc/mdadm/mdadm.conf update-initramfs -u cat /proc/mdstat # confirm [UUUU] — all four members healthy

Step 2 — LVM on Top of the Array

pvcreate /dev/md0 vgcreate datavg /dev/md0 lvcreate -L 50G -n datalv datavg mkfs.ext4 /dev/datavg/datalv mount /dev/datavg/datalv /mnt/data

Step 3 — Taking a Live Snapshot

lvcreate -L 5G -s -n datalv-snap /dev/datavg/datalv lvs # confirm the snapshot exists, check its usage %

Directly reusing fs1-6's own real-world pattern — a live snapshot taken immediately before doing something deliberately risky, protecting the current, known-good state before the next step intentionally breaks something.

Step 4 — Simulating a Disk Failure

mdadm /dev/md0 --fail /dev/sdc1 cat /proc/mdstat # shows a degraded array

Because this is RAID 10, data stays fully available right through the failure — the mirrored partner of the failed disk is still up. This is fs1-7's own RAID 10 claim ("survives multiple disk failures, as long as no single mirrored pair loses both of its own disks") demonstrated concretely rather than left theoretical.

mdadm /dev/md0 --remove /dev/sdc1 # -- physically replace the disk -- mdadm /dev/md0 --add /dev/sdc1 cat /proc/mdstat # shows recovering, a percentage climbing quickly

The rebuild here is fast and comparatively low-risk specifically because RAID 10 recovery means re-mirroring from the surviving partner disk — not RAID 5/6's own much slower whole-array parity reconstruction. This is fs1-7's own rebuild-risk comparison, made concrete rather than abstract.

Step 5 — Verifying Recovery

mdadm --detail /dev/md0 # confirm [UUUU] again, array fully healthy lvs # confirm the snapshot is untouched, still holds the pre-failure state

Worth being explicit about a real limitation of this exact stack: nothing in it — not ext4, not LVM, not mdadm — ever actually checksums the data content itself, per fs1-9's own honest layering discussion. This build trusts that the RAID rebuild reconstructed the data correctly; it has no independent way to verify that at the content level, the way a Btrfs- or ZFS-based capstone (per fs1-3/fs1-4) would have gotten for free. That's a deliberate, informed tradeoff here, following fs1-10's own framework for this kind of workload — not an oversight.

Chapter Attribution Table

Piece of the buildChapter
RAID 10 concepts (why this level)fs1-7
mdadm array creation, monitoring, failure/rebuildfs1-8
LVM layer (PV/VG/LV)fs1-5
ext4 formatting choicefs1-2, fs1-10
Live snapshot before a risky stepfs1-6
Traditional vs. integrated layering decisionfs1-9
Overall stack choice justificationfs1-10
/boot placement notefs1-5, grub1-8
Still out of scope
No checksumming or self-healing exists anywhere in this stack — a deliberate choice per fs1-10's framework for this workload, but a real limitation if bit-rot protection ever became an actual requirement, at which point Btrfs or ZFS (fs1-3/fs1-4) would be the correct answer instead. The snapshot taken in Step 3 is not a backup, per fs1-3's and fs1-6's own explicit warnings — it lives on the same array it's protecting. No /boot partition or GRUB configuration was actually built here (that's grub1's own dedicated territory). And this was a single-node build — network storage, iSCSI, and clustering are not covered anywhere in this course.
Check status after every step, not at the end
Run each command's own status check (cat /proc/mdstat, lvs) immediately after the step that could have gone wrong, not after batching several steps together. Catching a problem right after the step that caused it is dramatically easier than untangling it after several more steps have piled on top.

Hands-On Exercises

Exercise 1

Rebuild this capstone's own plan, but for a scenario where bit-rot protection genuinely is a hard requirement. Identify exactly which two chapters' worth of changes would be needed, and what would be dropped from this chapter's own stack as a result.

📄 View solution
Exercise 2

Explain why the RAID 10 rebuild in Step 4 was low-risk enough to not require the LVM snapshot from Step 3 for protection, while a hypothetical RAID 5 rebuild in the same scenario would make that snapshot's timing genuinely important.

📄 View solution
Exercise 3

A reader finishes this capstone and concludes "great, my data is now fully protected." Using this chapter's own warn-box, identify what's still missing from that conclusion.

📄 View solution

Chapter 11 Quick Reference — The Full Build

  • 1. mdadm --create — RAID 10 across four disks (fs1-7, fs1-8)
  • 2. pvcreate / vgcreate / lvcreate / mkfs.ext4 — LVM + ext4 on top (fs1-5, fs1-2)
  • 3. lvcreate -s — a live snapshot before the risky step (fs1-6)
  • 4. mdadm --fail / --remove / --add — simulate and recover from a disk failure (fs1-8)
  • 5. mdadm --detail / lvs — verify full recovery
  • This stack has no content checksumming anywhere — a deliberate tradeoff, not an oversight
  • The snapshot is not a backup — same array, same physical risk