Searching Folders

Chapter 1 searched named files. Real work is usually “somewhere in this project”: dozens of folders, files you did not write, build output you do not want, hidden folders, and a few files that are not text at all. This chapter is about telling grep where to look and what to skip.

A Practice Project

Make an empty folder, change into it and paste this. It builds a small project folder with source files, a build folder, a hidden .git folder, a hidden .env file, a file name with spaces, and one binary file with the word TODO hidden inside it. You finish inside project.

grep Does Not Enter Folders by Itself

Name a folder and plain grep refuses. The option that makes it go in is -r (recursive): search every file in the folder, every folder inside that, and so on.

Searching a Whole Tree: -r

That is every file that contains TODO, with the path in front of each line. Notice what was searched without being asked: the hidden .git folder, the generated build folder, and data/image.bin, which is not text. Most of this chapter is about getting rid of that noise. (The binary file's line, Binary file … matches, is grep 3.0's wording; grep 3.11 prints grep: ./data/image.bin: binary file matches instead, and prints it on standard error. More on that below.)

With no folder at all, GNU grep searches the current folder; the only difference is that the paths lose their leading ./:

The star missed .env, which is exactly the sort of file that holds a password. If a search seems to be missing things, check whether you used *.

Just the File Names: -l and -L

Often you do not want the lines, only which files. -l lists the files that contain a match, each once:

-L is the opposite: the files that do not contain it.

Do not use if grep -L ... as a test for “some file lacks the word”; look at its output instead.

Choosing Files: --include and --exclude

Search only files whose name matches a pattern with --include. Quote the pattern so the shell does not expand it:

Repeat the option to allow more than one kind:

--exclude does the opposite: skip files whose name matches.

The name is matched against the file name only, not the folder it is in. So build/bundle.js and src/main.js are both removed by --exclude='*.js'.

If you give both, the last option that matches a file decides. I tried both orders on the same pattern; the first keeps the JavaScript files, the second finds nothing at all:

Skipping Folders: --exclude-dir

--exclude-dir takes a folder name and skips that folder wherever it appears in the tree. The two you will use most are .git and a build or dependency folder such as node_modules or build. Repeat the option for each:

In bash, braces do the repeating for you (this is bash, not plain sh), with the same result:

Binary Files

A file that contains a zero byte is treated as binary. If it matches, grep does not print the line (which would fill your screen with junk); it only says so:

That is grep 3.0. In grep 3.11 the same search prints grep: data/image.bin: binary file matches, and it goes to standard error, not standard output. So a script that filters output with | grep -v '^Binary' works on one and not the other, and 2>/dev/null hides the message on the newer one. Three ways to deal with binary files:

-I printed nothing and gave status 1: “no match”, as far as it is concerned. With -a you get the text back (I used -o, which prints only the matching part, so the zero byte before it does not reach your screen):

-a is for the occasional file you know is mostly text. Do not use it on a whole tree: it will dump raw binary into your terminal. For recursive searches of source code, -I is nearly always right:

Note that -l without -I did list data/image.bin earlier: a binary file that matches is still a match.

Symbolic Links: -r and -R

A symbolic link is a name that points at another file or folder. -r does not follow links it finds inside the tree, so it does not search the same folder twice (or go round in a circle); -R follows them all. A link you name yourself on the command line is followed by both. This needs a real Linux file system, so I ran it in WSL's home folder (grep 3.11 only; Git Bash cannot make real links without special settings and would copy the folder instead, which proves nothing):

Use -R only when you know the tree is meant to be walked through its links, and be aware that a link pointing back up the tree can make it loop.

File Names with Spaces: -Z and xargs -0

Our tree has my notes/meeting notes.txt. A common pattern is to feed grep's list of files to another command with xargs. Here it just prints one argument per line, so you can see how it split the names:

xargs splits on spaces, so one file became three arguments, and any real command would complain that ./my does not exist. The cure is to separate the names with a zero byte, which cannot appear in a file name. -Z makes grep print them that way, and xargs -0 reads them that way:

Make -lZ / xargs -0 a habit whenever the next command does something, and always whenever it changes or deletes files.

grep or find?

grep -r chooses files by name (--include, --exclude) and folder name. find can choose by far more: type, size, age, depth, the whole path, permissions. When the file selection is the hard part, let find pick the files and hand them to grep with -exec ... {} +. Add -H so a single file still shows its name:

The second one used -path, which matches the whole path, something --include cannot do. For everyday work, grep -r with --include and --exclude-dir is shorter. (Chapter 8 looks at tools such as ripgrep that make the common case shorter still.)

A Good Default for Searching Code

Recursive, skip binary files, show line numbers, skip the .git folder. Add more --exclude-dir options for the folders your own projects generate. If the search finds nothing and you expected something, quiet failures are the likely cause: -s hides “no such file” errors, and the exit status is 2 either way:

Hands-On Exercises