15. Searching Text

It's quite common to search through text for a given sequence of characters (such as a word or phrase), called a string, or even for a pattern describing a set of such strings; this chapter contains recipes for doing these kind of things.

15.1 Searching for a Word or Phrase		Finding a word or phrase.
15.2 Regular Expressions--Matching Text Patterns		How to specify and find patterns.
15.3 Searching More than Plain Text Files		Searching in other than text files.
15.4 Outputting the Context of a Search		Searching in a certain context.
15.5 Searching and Replacing Text		Searching and replacing text.
15.6 Searching Text in Emacs		Searching in Emacs.
15.7 Searching Text in Less		Searching in less.

15.1 Searching for a Word or Phrase

The primary command used for searching through text is the rather froglike-sounding tool called grep (the origin of its name is explained in Regular Expressions--Matching Text Patterns, where its advanced usage is discussed). It outputs lines of its input that contain a given string or pattern.

To search for a word, give that word as the first argument. By default, grep searches standard input; give the name of a file to search as the second argument.

• To output lines in the file `catalog' containing the word `CD', type:

  $ grep CD catalog RET

To search for a phrase, specify it in quotes.

• To output lines in the file `catalog' containing the word `Compact Disc', type:

  $ grep 'Compact Disc' catalog RET

The preceding example outputs all lines in the file `catalog' that contain the exact string `Compact Disc'; it will not match, however, lines containing `compact disc' or any other variation on the case of letters in the search pattern. Use the `-i' option to specify that matches are to be made regardless of case.

• To output lines in the file `catalog' containing the string `compact disc' regardless of the case of its letters, type:

  $ grep -i 'compact disc' catalog RET

This command outputs lines in the file `catalog' containing any variation of the pattern `compact disc', including `Compact Disc', `COMPACT DISC', and `comPact dIsC'.

One thing to keep in mind is that grep only matches patterns that appear on a single line, so in the preceding example, if one line in `catalog' ends with the word `compact' and the next begins with `disc', grep will not match either line. There is a way around this with grep (see section Finding Phrases Regardless of Spacing), or you can search the text in Emacs (see section Searching for a Phrase in Emacs).

You can specify more than one file to search. When you specify multiple files, each match that grep outputs is preceded by the name of the file it's in (and you can suppress this with the `-h' option.)

• To output lines in all of the files in the current directory containing the word `CD', type:

  $ grep CD * RET

• To output lines in all of the `.txt' files in the `~/doc' directory containing the word `CD', suppressing the listing of file names in the output, type:

  $ grep -h CD ~/doc/*.txt RET

Use the `-r' option to search a given directory recursively, searching all subdirectories it contains.

• To output lines containing the word `CD' in all of the `.txt' files in the `~/doc' directory and in all of its subdirectories, type:

  $ grep -r CD ~/doc/*.txt RET

NOTE: There are more complex things you can search for than simple strings, as will be explained in the next section.

15.2 Regular Expressions--Matching Text Patterns

In addition to word and phrase searches, you can use grep to search for complex text patterns called regular expressions. A regular expression--or "regexp"---is a text string of special characters that specifies a set of patterns to match.

Technically speaking, the word or phrase patterns described in the previous section are regular expressions--just very simple ones. In a regular expression, most characters--including letters and numbers--represent themselves. For example, the regexp pattern 1 matches the string `1', and the pattern bee matches the string `bee'.

There are a number of reserved characters called metacharacters that don't represent themselves in a regular expression, but have a special meaning that is used to build complex patterns. These metacharacters are as follows: ., *, [, ], ^, $, and \.

To specify one of these literal characters in a regular expression, precede the character with a `\'.

• To output lines in the file `catalog' that contain a `$' character, type:

  $ grep '\$' catalog RET

• To output lines in the file `catalog' that contain the string `$1.99', type:

  $ grep '\$1\.99' catalog RET

• To output lines in the file `catalog' that contain a `\' character, type:

  $ grep '\\' catalog RET

The following table describes the special meanings of the metacharacters and gives examples of their usage.

NOTE: The name `grep' derives from a command in the now-obsolete Unix ed line editor tool--the ed command for searching globally through a file for a regular expression and then printing those lines was g/re/p, where re was the regular expression you'd use. Eventually, the grep command was written to do this search on a file when not using ed.(22)

The following sections describe some regexp recipes for commonly searched-for patterns.

15.2.1 Matching Lines Beginning with Certain Text		Matching text at the beginning of a line.
15.2.2 Matching Lines Ending with Certain Text		Matching text at the end of a line.
15.2.3 Matching Lines of a Certain Length		Matching a line as a pattern.
15.2.4 Matching Lines That Contain Any of Some Regexps		Matching lines containing any of some regexps.
15.2.5 Matching Lines That Contain All of Some Regexps		Matching lines containing all of some regexps.
15.2.6 Matching Lines That Don't Contain a Regexp		Finding lines that don't match.
15.2.7 Matching Lines That Only Contain Certain Characters		Matching lines of a certain length.
15.2.8 Finding Phrases Regardless of Spacing		Matching phrases regardless of spacing.
15.2.9 Finding Patterns in Certain Contexts		Matching patterns within a context.
15.2.10 Using a List of Regexps to Match From		Matching a list of patterns.
15.2.11 Regexps for Common Situations		Table of sample regular expressions.

15.2.1 Matching Lines Beginning with Certain Text

Use `^' in a regexp to denote the beginning of a line.

• To output all lines in `/usr/dict/words' beginning with `pre', type:

  $ grep '^pre' /usr/dict/words RET

• To output all lines in the file `book' that begin with the text `in the beginning', regardless of case, type:

  $ grep -i '^in the beginning' book RET

NOTE: These regexps were quoted with ' characters; this is because some shells otherwise treat the `^' character as a special "metacharacter" (see section Passing Special Characters to Commands).(23)

15.2.2 Matching Lines Ending with Certain Text

Use `$' as the last character of quoted text to match that text only at the end of a line.

• To output lines in the file `sayings' ending with an exclamation point, type:

  $ grep '!$' sayings RET

NOTE: To use `$' in a regexp to find words that rhyme with a given word, see Listing Words that Match a Pattern.

15.2.3 Matching Lines of a Certain Length

To match lines of a particular length, use that number of `.' characters between `^' and `$'---for example, to match all lines that are two characters (or columns) wide, use `^..$' as the regexp to search for.

• To output all lines in `/usr/dict/words' that are exactly two characters wide, type:

  $ grep '^..$' /usr/dict/words RET

For longer lines, it is more useful to use a different construct: `^.\{number\}$', where number is the number of lines to match. Use `,' to specify a range of numbers.

• To output all lines in `/usr/dict/words' that are exactly seventeen characters wide, type:

  $ grep '^.\{17\}$' /usr/dict/words RET

• To output all lines in `/usr/dict/words' that are twenty-five or more characters wide, type:

  $ grep '^.\{25,\}$' /usr/dict/words RET

15.2.4 Matching Lines That Contain Any of Some Regexps

To match lines that contain any of a number of regexps, specify each of the regexps to search for between alternation operators (`\|') as the regexp to search for. Lines containing any of the given regexps will be output.

• To output all lines in `playlist' that contain either the patterns `the sea' or `cake', type:

  $ grep 'the sea\|cake' playlist RET

This command outputs any lines in `playlist' that match the patterns `the sea' or `cake', including lines matching both patterns.

15.2.5 Matching Lines That Contain All of Some Regexps

To output lines that match all of a number of regexps, use grep to output lines containing the first regexp you want to match, and pipe the output to a grep with the second regexp as an argument. Continue adding pipes to grep searches for all the regexps you want to search for.

• To output all lines in `playlist' that contain both patterns `the sea' and `cake', regardless of case, type:

  $ grep -i 'the sea' playlist | grep -i cake RET

NOTE: To match lines containing some regexps in a particular order, see Regexps for Common Situations.

15.2.6 Matching Lines That Don't Contain a Regexp

To output all lines in a text that don't contain a given pattern, use grep with the `-v' option--this option reverts the sense of matching, selecting all non-matching lines.

• To output all lines in `/usr/dict/words' that are not three characters wide, type:

  $ grep -v '^...$' RET

• To output all lines in `access_log' that do not contain the string `http', type:

  $ grep -v http access_log RET

15.2.7 Matching Lines That Only Contain Certain Characters

To match lines that only contain certain characters, use the regexp `^[characters]*$', where characters are the ones to match.

• To output lines in `/usr/dict/words' that only contain vowels, type:

  $ grep -i '^[aeiou]*$' /usr/dict/words RET

The `-i' option matches characters regardless of case; so, in this example, all vowel characters are matched regardless of case.

15.2.8 Finding Phrases Regardless of Spacing

One way to search for a phrase that might occur with extra spaces between words, or across a line or page break, is to remove all linefeeds and extra spaces from the input, and then grep that.

To do this, pipe the input(24) to tr with `'\r\n:\>\|-'' as an argument to the `-d' option (removing all linebreaks from the input); pipe that to the fmt filter with the `-u' option (outputting the text with uniform spacing); and pipe that to grep with the pattern to search for.

• To search across line breaks for the string `at the same time as' in the file `notes', type:

  $ cat notes | tr -d '\r\n:\>\|-' | fmt -u | grep 'at the same time as' RET

NOTE: The Emacs editor has its own special search for doing this--see Searching for a Phrase in Emacs.

15.2.9 Finding Patterns in Certain Contexts

To search for a pattern that only occurs in a particular context, grep for the context in which it should occur, and pipe the output to another grep to search for the actual pattern.

For example, this can be useful to search for a given pattern only when it is quoted with an `>' character in an email message.

• To list lines from the file `email-archive' that contain the word `narrative' only when it is quoted, type:

  $ grep '^>' email-archive | grep narrative RET

You can also reverse the order and use the `-v' option to output all lines containing a given pattern that are not in a given context.

• To list lines from the file `email-archive' that contain the word `narrative', but not when it is quoted, type:

  $ grep narrative email-archive | grep -v '^>' RET

15.2.10 Using a List of Regexps to Match From

You can keep a list of regexps in a file, and use grep to search text for any of the patterns in the file. To do this, specify the name of the file containing the regexps to search for as an argument to the `-f' option.

This can be useful, for example, if you need to search a given text for a number of words--keep each word on its own line in the regexp file.

• To output all lines in `/usr/dict/words' containing any of the words listed in the file `forbidden-words', type:

  $ grep -f forbidden-words /usr/dict/words RET

• To output all lines in `/usr/dict/words' that do not contain any of the words listed in `forbidden-words', regardless of case, type:

  $ grep -v -i -f forbidden-words /usr/dict/words RET

15.2.11 Regexps for Common Situations

The following table lists sample regexps and describes what they match. You can use these regexps as boilerplate when building your own regular expressions for searching text. Remember to enclose regexps in quotes.

15.3 Searching More than Plain Text Files

The following recipes are for searching data other than in plain text files.

15.3.1 Matching Lines in Compressed Files		Matching lines in compressed files.
15.3.2 Matching Lines in Web Pages		Matching lines in Web pages.

15.3.1 Matching Lines in Compressed Files

Use zgrep to search through text in files that are compressed. These files usually have a `.gz' file name extension, and can't be searched or otherwise read by other tools without uncompressing the file first (for more about compressed files, see section Compressed Files).

The zgrep tool works just like grep, except it searches through the text of compressed files. It outputs matches to the given pattern as if you'd searched through normal, uncompressed files. It leaves the files compressed when it exits.

• To search through the compressed file `README.gz' for the text `Linux', type:

  $ zgrep Linux README.gz RET

15.3.2 Matching Lines in Web Pages

You can grep a Web page or other URL by giving the URL to lynx with the `-dump' option, and piping the output to grep.

• To search the contents of the URL http://example.com/ for lines containing the text `gonzo' or `hunter', type:

$ lynx -dump http://example.com/ | grep 'gonzo\|hunter' RET

15.4 Outputting the Context of a Search

It is sometimes useful to see a matched line in its context in the file--that is, to see some of the lines that surround it.

Use the `-C' option with grep to output results in context---it outputs matched lines with two lines of "context" both before and after each match. To specify the number of context lines output both before and after matched lines, use that number as an option instead of `-C'.

• To search `/usr/dict/words' for lines matching `tsch' and output two lines of context before and after each line of output, type:

  $ grep -C tsch /usr/dict/words RET

• To search `/usr/dict/words' for lines matching `tsch' and output six lines of context before and after each line of output, type:

  $ grep -6 tsch /usr/dict/words RET

To output matches and the lines before them, use `-B'; to output matches and the lines after them, use `-A'. Give a numeric option with either of these options to specify that number of context lines.

• To search `/usr/dict/words' for lines matching `tsch' and output two lines of context before each line of output, type:

  $ grep -B tsch /usr/dict/words RET

• To search `/usr/dict/words' for lines matching `tsch' and output six lines of context after each line of output, type:

  $ grep -A6 tsch /usr/dict/words RET

• To search `/usr/dict/words' for lines matching `tsch' and output ten lines of context before and three lines of context after each line of output, type:

  $ grep -B10 -A3 tsch /usr/dict/words RET

15.5 Searching and Replacing Text

A quick way to search and replace some text in a file is to use the following one-line perl command:

$ perl -pi -e "s/oldstring/newstring/g;" filespec RET

In this example, oldstring is the string to search, newstring is the string to replace it with, and filespec is the name of the file or files to work on. You can use this for more than one file.

• To replace the string `helpless' with the string `helpful' in all files in the current directory, type:

  $ perl -pi -e "s/helpless/helpful/g;" * RET

You can also search and replace text in an Emacs buffer; to do this, use the replace-regexp function and give both the expression to search for and the expression to replace it with.

• To replace the text `helpless' with the text `helpful' in the current buffer, type:

  M-x replace-regexp RET helpless RET helpful RET

NOTE: You can also search and replace text in most text editors, including Emacs; see Searching and Replacing in Emacs.

15.6 Searching Text in Emacs

The following sections show ways of searching for text in Emacs--incrementally, for a word or phrase, or for a pattern--and for searching and then replacing text.

15.6.1 Searching Incrementally in Emacs		Searching incrementally.
15.6.2 Searching for a Phrase in Emacs		Searching for a word or phrase.
15.6.3 Searching for a Regexp in Emacs		Searching for a pattern.
15.6.4 Searching and Replacing in Emacs		Replacing text in Emacs.

15.6.1 Searching Incrementally in Emacs

Type C-s to use the Emacs incremental search function. It takes text as input in the minibuffer and it searches for that text from point toward the end of the current buffer. Type C-s again to search for the next occurrence of the text you're searching for; this works until no more matches occur. Then, Emacs reports `Failing I-search' in the minibuffer; type C-s again to wrap to the beginning of the buffer and continue the search from there.

It gets its name "incremental" because it begins searching immediately when you start to type text, and so it builds a search string in increments---for example, if you want to search for the word `sunflower' in the current buffer, you start to type

C-s s

At that point Emacs searches forward through the buffer to the first `s' character, and highlights it. Then, as you type u, it searches forward to the first `su' in the buffer and highlights that (if a `u' appears immediately after the `s' it first stopped at, it stays where it's at, and highlights the `s' and the `u'). It continues to do this as long as you type and as long as there is a match in the current buffer. As soon as what you type does not appear in the buffer, Emacs beeps and a message appears in the minibuffer stating that the search has failed.

To search for the next instance of the last string you gave, type C-s again; if you keep CTRL held down, every time you press the S key, Emacs will advance to the next match in the buffer.

This is generally the fastest and most common type of search you will use in Emacs.

You can do an incremental search through the buffer in reverse---that is, from point to the beginning of the buffer--with the isearch-backward function, C-r.

• To search for the text `moon' in the current buffer from point in reverse to the beginning of the buffer, type:

  C-r moon

15.6.2 Searching for a Phrase in Emacs

Like grep, the Emacs incremental search only works on lines of text, so it only finds phrases on a single line. If you search for `hello, world' with the incremental search and the text `hello,' appears at the end of a line and the text `world' appears at the beginning of the next line, it won't find it.

To find a multi-word phrase across line breaks, use the word-search-forward function. It searches for a phrase or words regardless of punctuation or spacing.

• To search forward through the current buffer for the phrase `join me', type:

  M-x word-search-forward RET join me RET

NOTE: The word-search-backward function does the same as word-search-forward, except it searches backward through the buffer, from point to the beginning of the buffer.

15.6.3 Searching for a Regexp in Emacs

Use the search-forward-regexp function to search for a regular expression from point to the end of the current buffer.

• To search forward through the current buffer for the regexp `@.*\.org', type:

  M-x search-forward-regexp RET @.*\.org RET

The keyboard accelerator for this command is M-C-s---on most keyboards, you press and release ESC and then hold down CTRL while you type s. To repeat the last regexp search you made, type M-C-s C-s; then, as long as you have CTRL held down, you can keep typing s to advance to the next match, just as you would with an incremental search.

NOTE: There is a search-backward-regexp function that is identical but searches backward, from point to the top of the buffer.

15.6.4 Searching and Replacing in Emacs

To search for and replace text in Emacs, use the replace-regexp function. When you run this function, Emacs will ask for both the text or regexp to search for and the text to replace it with.

• To replace the text `day' with the text `night' in the current buffer, type:

  M-x replace-regexp RET day RET night RET

This function is especially useful for replacing control characters with text, or for replacing text with control characters, which you can specify with C-q, the quoted-insert function (see section Inserting Special Characters in Emacs).

• To replace all the `^M' characters in the current buffer with regular linefeeds, type:

  M-x replace-regexp RET C-q C-m RET C-q 012 RET RET

15.7 Searching Text in Less

There are two useful commands in less for searching through text: / and ?. To search forward through the text, type / followed by a regexp to search for; to search backward through the text, use ?.

When you do a search, the word or other regexp you search for appears highlighted throughout the text.

• To search forward through the text you are perusing for the word `cat', type:

  /cat RET

  To search backward through the text you are perusing for the regexp `[ch]at', type:

  ?[ch]at RET