13. Analyzing Text

There are many ways to use command-line tools to analyze text in various ways, such as counting the number of words in a text, creating a concordance, and comparing texts to see if (and where) they differ. There are also other tricks you can do with text that count as analysis, such as finding anagrams and palindromes, or cutting up text to generate unexpected combinations of words. This chapter covers all these topics.

13.1 Counting Text		Counting words, lines and characters.
13.2 Making a Concordance of a Text		Making a text concordance.
13.3 Text Relevance		Finding similar or relevant text.
13.4 Finding Anagrams in Text		Finding anagrams.
13.5 Finding Palindromes in Text		Finding palindromes.
13.6 Text Cut-Ups		Cutting up text.

13.1 Counting Text

Use the "word count" tool, wc, to count characters, words, and lines in text.

Give the name of a file as an argument; if none is given, wc works on standard input. By default, wc outputs three columns, displaying the counts for lines, words, and characters in the text.

• To output the number of lines, words, and characters in file `outline', type:

  $ wc outline RET

The following subsections describe how to specify just one kind of count with wc, and how to count text in Emacs.

NOTE: You can get a count of how many different words are in a text, too--see Making a Concordance of a Text. To count the average length of words, sentences, and paragraphs, use style (see section Checking Text for Readability).

13.1.1 Counting the Characters in a Text		Counting characters.
13.1.2 Counting the Words in a Text		Counting words.
13.1.3 Counting the Lines in a Text		Counting lines.
13.1.4 Counting the Occurrences of Something		Counting the number of some text string.
13.1.5 Counting Lines per Page in Emacs		Counting pages in Emacs.

13.1.1 Counting the Characters in a Text

Use wc with the `-c' option to specify that just the number of characters be counted and output.

• To output the number of characters in file `classified.ad', type:

  $ wc -c classified.ad RET

13.1.2 Counting the Words in a Text

Use wc with the `-w' option to specify that just the number of words be counted and output.

• To output the number of words in the file `story', type:

  $ wc -w story RET

To output counts for several files, first concatenate the files with cat, and then pipe the output to wc.

• To output the combined number of words for all the files with a `.txt' file name extension in the current directory, type:

  $ cat *.txt | wc -w RET

NOTE: To read more about concatenation with cat, see Concatenating Text.

13.1.3 Counting the Lines in a Text

Use wc with the `-l' option to specify that just the number of lines be counted and output.

• To output the number of lines in the file `outline', type:

  $ wc -l outline RET

13.1.4 Counting the Occurrences of Something

To find the number of occurrences of some text string or pattern in a file or files, use grep to search the file(s) for the text string, and pipe the output to wc with the `-l' option.

• To find the number of lines in the file `outline' that contain the string `chapter', type:

  $ grep chapter outline | wc -l RET

NOTE: For more recipes for searching text, and more about grep, see Searching Text.

13.1.5 Counting Lines per Page in Emacs

The count-lines-page function in Emacs outputs in the minibuffer the number of lines on the current page (as delimited by pagebreak characters, if any---see section Paginating Text), followed by the number of lines in the buffer before the line that point is on, and the number of lines in the buffer after point.

• To count the number of lines per page in the current buffer in Emacs, type:

  C-x l

Emacs outputs the number of lines per page of the current buffer in the echo area.

For example, if the output in the minibuffer is

Page has 351 lines (69 + 283)

13.2 Making a Concordance of a Text

A concordance is an index of all the words in a text, along with their contexts. A concordance-like functionality--an alphabetical listing of all words in a text and their frequency--can be made fairly easily with some basic shell tools: tr, sort, and uniq.

• To output a word-frequency list of the text file `naked_lunch', type:

  $ tr ' ' ' RET > ' < naked_lunch | sort | uniq -c RET

These commands translate all space characters to newline characters, outputting the text with each word on its own line; this is then sorted alphabetically, and that output is passed to uniq, which outputs only the unique lines--that is, all non-duplicate lines--while the `-c' option precedes each line with its count (the number of times it occurs in the text).

To get a word frequency count--that is, the total number of different words in a text--just pipe the output of the frequency list to wc with the `-l' option. This counts all the lines of its input, which in this case will be the list of unique words, one per line.

• To output a count of the number of unique words in the text file `naked_lunch', type:

  $ tr ' ' ' RET > ' < naked_lunch | sort | uniq -c | wc -l RET

13.3 Text Relevance

The following recipes show how to analyze a given text for its relevancy to other text, either to keywords or to whole files of text.

You can also use the diff family of tools to analyze differences in text; those tools are especially good for comparing different revisions of the same file (see section Comparing Files).

13.3.1 Sorting Text in Order of Relevance		Sorting text by relevance.
13.3.2 Listing Relevant Files in Emacs		An Emacs tool for finding relevant text.

13.3.1 Sorting Text in Order of Relevance

@sf{Debian}: `rel'
@sf{WWW}: http://www.johncon.com/

rel takes as an option the keyword to search for in quotes; you can build a boolean expression by grouping multiple keywords in parentheses and using any of the following operators between them:

Give as arguments the names of the files to rank.

• To rank the files `report.a', `report.b', and `report.c' in order of relevance to the keywords `saving' and `profit', type:

  $ rel "(saving & profit)" report.a report.b report.c RET

Give the name of a directory tree to analyze all files in the directory tree.

• To output a list of any files containing either `invitation' or `request' in the `~/mail' directory, ranked in order of relevancy, type:

  $ rel "(invitation | request)" ~/mail RET

• To output a list of any files containing `invitation' and not `wedding' in the `~/mail' directory, ranked in order of relevancy, type:

  $ rel "(invitation ! wedding)" ~/mail RET

• To output a list of any files containing `invitation' and `party' in the `~/mail' directory, ranked in order of relevancy, type:

  $ rel "(invitation & party)" ~/mail RET

13.3.2 Listing Relevant Files in Emacs

@sf{Debian}: `remembrance-agent'
@sf{WWW}: http://www.media.mit.edu/~rhodes/RA/

When installing the Remembrance Agent, you create three databases of the files to use when making relevance suggestions; when remembrance-agent is running, it searches these three databases in parallel, looking for relevant text. You could create, for example, one database of saved email, one of your own writings, and one of saved documents.

• To toggle the Remembrance Agent in the current buffer, type:

  C-c r t

When remembrance-agent is running, suggested buffers will be displayed in the small `*Remembrance*' buffer at the bottom of the screen. To open a suggestion in a new buffer, type C-c r number, where number is the number of the suggestion.

• To open the second suggested file in a new buffer, type:

  C-c r 2

13.4 Finding Anagrams in Text

@sf{Debian}: `an'

Use an to find and output anagrams. Give as an argument the word or quoted phrase to use; an writes its results to the standard output.

• To output all anagrams of the word `lake', type:

  $ an lake RET

• To output all anagrams of the phrase `lakes and oceans', type:

  $ an 'lakes and oceans' RET

To limit the anagrams output to those containing a given string, specify that string with the `-c' option.

• To output only anagrams of the phrase `lakes and oceans' which contain the string `seas', type:

  $ an -c seas 'lakes and oceans' RET

To print all of the words that some or all letters in a given word or phrase can make, use the `-w' option. This outputs words that are not anagrams, since anagrams must contain all of the letters of the other word or phrase.

• To output all of the words that can be made from the letters of the word `seas', type:

  $ an -w seas RET

This command outputs all of the words that can be formed from all or some of the characters in `seas', including `see' and `as'.

13.5 Finding Palindromes in Text

A palindrome is a word that reads the same both forwards and backwards; for example, "Mom," "madam," and "nun" are all palindromes.

To find palindromes in a file, use this simple Perl "one-liner," and substitute file for the name of the file to check:

perl -lne 'print if $_ eq reverse' file

To check for palindromes in the standard input, specify `-' as the file name to check. This is useful for putting at the end of a pipeline.

• To output all of the palindromes in the system dictionary, type:

  $ perl -lne 'print if $_ eq reverse' /usr/dict/words RET

13.6 Text Cut-Ups

A cut-up is a random rearrangement of a physical layout of text, made with the intention of finding unique or interesting phrases in the rearrangement. Software for rearranging text in random ways has existed since the earliest text-processing tools; the popularity of these tools will never die.

The cut-up technique in literature was discovered by painter Brion Gysin and American writer William S. Burroughs in 1959; they believed it brought the montage technique of painting to the written word.

"All writing is in fact cut-ups," Burroughs wrote.(21) "A collage of words read heard overheard ... [u]se of scissors renders the process explicit and subject to extension and variation."

These recipes describe a few of the common ways to make text cut-ups; more free software tools for making cut-ups are listed at http://dsl.org/comp/cutups.shtml.

13.6.1 Making Simple Text Cut-Ups		Simple text cutups.
13.6.2 Making Random Word Cut-Ups		Advanced text cutups.
13.6.3 Making Cut-Ups in Emacs		Cutups in Emacs.

13.6.1 Making Simple Text Cut-Ups

@sf{WWW}: http://dsl.org/comp/tinyutils/

• To make a cut-up from a file called `nova', type:

  $ cutup nova RET

13.6.2 Making Random Word Cut-Ups

@sf{Debian}: `dadadodo'
@sf{WWW}: http://www.jwz.org/dadadodo/

Give as an argument the name of the text file to be used; by default, dadadodo outputs text to standard output until you interrupt it by typing C-c.

• To output random text based on the text in the file `nova', type:

  $ dadadodo nova RET

This command will output passages of random text based on the text in the file `nova' until it is interrupted by the user.

You can analyze a text and save the analysis to a file of compiled data; this analysis can then be used to generate random text when the original input text is not present. The following table describes this and other dadadodo options.

13.6.3 Making Cut-Ups in Emacs

The dissociated-press function in Emacs makes random cut-ups of the current buffer in a new buffer called `*Dissociation*'; the original buffer is not modified. The text in the new buffer is generated by combining random portions of the buffer by overlapping characters or words, thus (usually) creating plausible-sounding sentences. It pauses occasionally and asks whether or not you want to continue the dissociation.

• To generate a Dissociated Press cut-up from the current buffer, type:

  M-x dissociated-press RET

Give a positive argument to the dissociated-press function to specify the number of characters to use for overlap; give a negative argument to specify the number of words for overlap.

• To generate a Dissociated Press cut-up from the current buffer, always overlapping by three characters, type:

  C-u 3 M-x dissociated-press RET

• To generate a Dissociated Press cut-up from the current buffer, always overlapping by one word, type:

  C-u -1 M-x dissociated-press RET