Large Repos

Course 3 · Ch 7

Large Repos

Git LFS for big files, shallow clones and sparse checkout for big histories and big trees, and a look at monorepo strategy

Everything in this series so far assumes a repository of a size where git clone just works in a few seconds and nobody thinks about it. Past a certain size — large binary assets, years of deep history, or a single repo containing many independent projects — that assumption breaks down, and git offers specific tools for each distinct problem.

Identifying Which Large-Repo Problem You Actually Have

📦 Large binary files

Videos, design files, datasets, compiled assets — git's diffing and storage model handles text well, but stores every full version of a binary separately, bloating repo size fast.

Fix: Git LFS

📜 Deep, long history

Years of commits — cloning the entire history just to start contributing today is unnecessary overhead for most contributors.

Fix: shallow clone

🗂️ Huge file tree

A monorepo with dozens of independent projects — most contributors only ever touch one or two of them, but a normal clone downloads everything.

Fix: sparse checkout

Git LFS — Large File Storage

Git LFS replaces large files in your repo with small text pointers, while the actual file content lives in separate LFS storage — git's normal history-tracking machinery (Chapter 1's object model) only ever sees the lightweight pointer, not the heavy binary itself.

Git's history tracks a tiny pointer; LFS storage holds the actual large file, fetched on demand

$ # One-time setup per machine
$ git lfs install

$ # Tell LFS which file types to track (writes to .gitattributes)
$ git lfs track "*.mp4"
$ git lfs track "*.psd"

$ # From here, normal git commands just work
$ git add .gitattributes video.mp4
$ git commit -m "Add product demo video"
$ git push

LFS must be set up BEFORE large files are committed, ideally

Converting a file already tracked normally into an LFS-tracked one after the fact requires rewriting history (a job for git lfs migrate) — every previous commit's full-size version is still sitting in the regular object database otherwise. Setting up .gitattributes at the start of a project, before the first large file is ever committed, avoids this entirely.

Shallow Clones — Downloading Only Recent History

$ # Only the most recent commit, no deep history at all
$ git clone --depth 1 https://github.com/big-org/huge-project.git

$ # Or the last 50 commits, if you need a bit more context
$ git clone --depth 50 https://github.com/big-org/huge-project.git

A shallow clone is dramatically faster for repos with years of history — most CI systems (Course 2/3's Actions chapters) use shallow clones by default, since a workflow run usually only needs the current state, not the full history.

Some operations need full history and won't work on a shallow clone

Bisecting (next chapter), inspecting old commits beyond the depth limit, or rebasing against history outside the shallow window all require more history than a shallow clone has. git fetch --unshallow converts it to a full clone if you later find you need that history after all.

Sparse Checkout — Downloading Only Part of the Tree

For a monorepo where the full history is needed but only a fraction of the file tree is actually relevant to you, sparse checkout downloads everything's history but only checks out the specific folders you choose into your working directory.

$ git clone --filter=blob:none --sparse https://github.com/big-org/monorepo.git
$ cd monorepo
$ git sparse-checkout set frontend/web-app shared/ui-components
/* only these two folders now appear in your working directory */

The --filter=blob:none flag avoids downloading file contents you haven't checked out yet, fetching them lazily only when actually needed — a meaningful speed difference on a genuinely large monorepo.

Monorepo Strategy — A Brief Orientation

A monorepo holds multiple, often-independent projects in a single repository, rather than splitting each into its own repo (the more traditional approach, and what every previous chapter implicitly assumes).

Advantages: atomic cross-project changes in a single commit/PR, shared tooling and CI config, easier code sharing between projects without the submodule/subtree complexity from Chapter 4.
Trade-offs: repo size and clone time grow with every project added (exactly what this chapter's tools mitigate), and CI needs to be smart enough to only test/build what actually changed rather than the entire repo on every push.
Tooling matters more at scale. Beyond a certain size, dedicated monorepo tools (Nx, Turborepo, Bazel) handle the "only build what changed" problem far better than hand-rolled CI logic — worth researching directly if a monorepo grows large enough that this chapter's git-level tools alone aren't enough.

Most projects never need any of this chapter

Git LFS, shallow clones, and sparse checkout exist specifically for scale problems — a typical project, even a fairly large one, works fine with ordinary clones and normal git usage from every previous chapter. Reach for these tools when an actual, measured problem (slow clones, repo size complaints, genuinely large binary assets) appears, not preemptively.

Command Reference

Command	What it does
git lfs install	One-time setup enabling LFS on a machine
git lfs track "*.ext"	Marks a file pattern to be stored via LFS, written to .gitattributes
git clone --depth N	Shallow clone — only the last N commits of history
git fetch --unshallow	Converts a shallow clone into a full one
git clone --filter=blob:none --sparse	Clones with full history but no file content yet, ready for sparse checkout
git sparse-checkout set <paths>	Chooses which folders actually appear in the working directory

Chapter 7 Quick Reference

Git LFS — for large binary files; stores a lightweight pointer in git, real content in separate LFS storage
Set up LFS tracking before committing large files where possible — converting after the fact needs a history rewrite
Shallow clone (--depth N) — for deep history you don't need; common default for CI runners
Some operations (bisect, old-commit inspection) need full history — git fetch --unshallow if needed later
Sparse checkout — for a huge file tree (monorepo); full history, only checked-out folders you actually need
Monorepo trade-off: easier cross-project changes and sharing, vs. growing repo size and CI complexity
Most projects never need this chapter's tools — reach for them when a real, measured scale problem appears
Next chapter: git bisect — finding exactly which commit introduced a regression