The most simple text filter is the cat, it does nothing else than sending the data from stdin to stdout:

$ echo "hello world" | cat
hello world
      

Another useful feature is that you can let it send the contents of a file to the standard output:

$ cat file.txt
Hello, this is the content of file.txt
      

cat really lives up to its name when multiple files are added as arguments. This will concatenate the files, in the sense that it will send the contents of all files to the standard output, in the same order as they were specified as an argument. The following screen snippet demonstrates this:

$ cat file.txt file1.txt file2.txt
Hello, this is the content of file.txt
Hello, this is the content of file1.txt
Hello, this is the content of file2.txt
      

The wc command provides statistics about a text file or text stream. Without any parameters, it will print the number of lines, the number of words, and the number of bytes respectively. A word is delimited by one white space character, or a sequence of whitespace characters.

The following example shows the number of lines, words, and bytes in the canonical “Hello world!” example:

$ echo "Hello world!" | wc 
       1       2      13
      

If you would like to print just one of these components, you can use one of the -l (lines), -w (words), or -c (bytes) parameters. For instance, adding just the -l parameter will show the number of lines in a file:

$ wc -l /usr/share/dict/words 
  235882 /usr/share/dict/words
      

Or, you can print additional fields by adding a parameter:

$ wc -lc /usr/share/dict/words
 235882 2493082 /usr/share/dict/words
      

Please note that, no matter the order in which the options were specified, the output order will always be the same (lines, words, bytes).

Since -c prints the number bytes, this parameter may not represent the number of characters that a text holds, because the character set in use maybe be wider than one byte. To this end, the -m parameter has been added which prints the number of characters in a text, independent of the character set. -c and -m are substitutes, and can never be used at the same time.

The statistics that wc provides are more useful than they may seem on the surface. For example, the -l parameter is often used as a counter for the output of a command. This is convenient, because many commands seperate logical units by a newline. Suppose that you would like to count the number of files in your home directory having a filename ending with .txt. You could do this by combining find to find the relevant files and wc to count the number of occurences:

$ find ~ -name '*.txt' -type f | wc -l
      

The tr command can be used to do common character operations, like swapping characters, deleting characters, and squeezing character sequences. Depending on the operation, one or two sets of characters should be specified. Besides normal characters, there are some special character sequences that can be used:

$ echo 'Hello world!' | tr 'eo' 'ia'
Hilla warld!
	

$ echo 'Hello world!' | tr 'eaiou' '[@*]'
H@ll@ w@rld!
	

$ echo 'abcdef' | tr 'abcdef' '@[-*4]!'
@----!
	

$ echo "Let's squeeze this." | tr -s 'e'
Let's squeze this.
	

$ echo "eenie meenie minie moe" | tr 'aeiou' '[@*]' | tr -s '@'
@n@ m@n@ m@n@ m@
	

$ echo 'Hello world!' | tr -d 'lr'
Heo wod!
	

The cut command is provided by UNIX systems to “cut” one or more columns from a file or stream, printing it to the standard output. It is often useful to selectively pick some information from a text. cut provides three approaches to cutting information from files:

In all three approaches, you can specify the element to choose by its number starting at 1. You can specify a range by using a dash (-). So, M-N means the Mth to the Nth element. Leaving M out (-N) selects all elements from the first element to the Nth element. Leaving N out (M-) selects the Mth element to the last element. Multiple elements or ranges can be combined by separating them by commas (,). So, for instance, 1,3- selects the first element and the third to the last element.

Data can be cut by field with the -f fields parameter. By default, the horizontal tab is used a separator. Let's have a look at cut in action with a tiny Dutch to English dictionary:

$ cat dictionary
appel   apple
banaan  banana
peer    pear
      

We can get all English words by selecting the first field:

$ cut -f 2 dictionary
apple
banana
pear
      

That was quite easy. Now let's do the same thing with a file that has a colon as the field separator. We can easily try this by converting the dictionary with the tr command that we have seen earlier, replacing all tabs with colons:

$ tr '\t' ':' < dictionary > dictionary-new
$ cat dictionary-new
appel:apple
banaan:banana
peer:pear
      

If we use the same command as in the previous example, we do not get the correct output:

$ cut -f 2 dictionary-new
appel:apple
banaan:banana
peer:pear
      

What happens here is that the delimiter could not be found. If a line does not contain the delimiter that is being used, the default behavior of cut is to print the complete line. You can prevent this with the -s parameter.

To use a different delimiter than the horizontal tab, add the -d delimter_char parameter to set the delimiting character. So, in this case of our dictionary-new file, we will ask cut to use the colon as a delimiter:

$ cut -d ':' -f 2 dictionary-new
apple
banana
pear
      

If a field that was specified does not exist in a line, that particular field is not printed.

The -b bytes and -c characters respectively select bytes and characters from the text. On older systems a character used to be a byte wide. But newer systems can provide character sets that are wider than one byte. So, if you want to be sure to grab complete characters, use the -c parameter. An entertaining example of seeing the -c parameter in action is to find the ten most common sets of the first three characters of a word. Most UNIX systems provide a list of words that are separated by a new line. We can use cut to get the first three characters of the words in the word list, add uniq to count identical three character sequences, and use sort to sort them reverse-numerically (sort is described in Section 9.1.5, “Sorting text”). Finally, we will use head to get the ten most frequent sequences:

$ cut -c 1-4 /usr/share/dict/words | uniq -c | sort -nr | head
    254 inte
    206 comp
    169 cons
    161 cont
    150 over
    125 tran
    111 comm
    100 disc
     99 conf
     96 reco
      

Having concluded with that nice piece of UNIX commands in action, we will move on to the paste command, which combines files in columns in a single text stream.

Usage of paste is very simple, it will combine all files given as an argument, separated by a tab. With the list of English and Dutch words, we can generate a tiny dictionary:

$ paste dictionary-en dictionary-nl
apple   appel
banana  banaan
pear    peer
      

You can also combine more than two files:

$ paste dictionary-en dictionary-nl dictionary-de 
apple   appel   Apfel
banana  banaan  Banane
pear    peer    Birne
      

If one of the files is longer, the column order is maintained, and empty entries are used to fill up the entries of the shorter files.

You can use another delimiter by adding the -d delimiter parameter. For example, we can make a colon-separated dictionary:

$ paste -d ':' dictionary-en dictionary-nl
apple:appel
banana:banaan
pear:peer
      

Normally, paste combines files as different columns. You can also let paste use the lines of each file as columns, and put the columns of each file on a separate line. This is done with the -s parameter:

$ paste -s dictionary-en dictionary-nl dictionary-de
apple   banana  pear
appel   banaan  peer
Apfel   Banane  Birne
      

UNIX offers the sort command to sort text. sort can also check whether a file is in sorted order, and merge two sorted files. sort can sort in dictionary and numerical orders. The default sort order is the dictionary order. This means that text lines are compared character by character, sorted as specified in the current collating sequence (which is specified through the LC_COLLATE environment variable). This has a catch when you are sorting numbers, for instance, if you have the numbers 1 to 10 on different lines, the sequence will be 1, 10, 2, 3, etc. This is caused by the per-character interpretation of the dictionary sort. If you want to sort lines by number, use the numerical sort.

If no additional parameters are specified, sort sorts the input lines in dictionary order. For instance:

$ cat << EOF | sort
orange
apple
banana
EOF
apple
banana
orange
      

As you can see, the input is correctly ordered. Sometimes there are two identical lines. You can merge identical lines by adding the -u parameter. The two samples listed below illustrate this.

$ cat << EOF | sort
orange
apple
banana
banana
EOF
apple
banana
banana
orange
$ cat << EOF | sort -u
orange
apple
banana
banana
EOF
apple
banana
orange
      

There are some additional parameters that can be helpful to modify the results a bit:

You can sort files numerically by adding the -n parameter. This parameter stops reading the input line when a non-numeric character was found. The leading minus sign, decimal point, thousands separator, radix character (that separates an exponential from a normal number), and blanks can be used as a part of a number. These characters are interpreted where applicable.

The following example shows numerical sort in action, by piping the output of du to sort. This works because du specifies the size of each file as the first field.

$ du -a /bin | sort -n
0       /bin/kernelversion
0       /bin/ksh
0       /bin/lsmod.modutils
0       /bin/lspci
0       /bin/mt
0       /bin/netcat
[...]
      

In this case, the output is probably not useful if you want to read the output in a paginator, because the smallest files are listed first. This is where the -r parameter becomes handy. This reverses the sort order.

$ du -a /bin | sort -nr
4692    /bin
1036    /bin/ksh93
668     /bin/bash
416     /bin/busybox
236     /bin/tar
156     /bin/ip
[...]
      

The -r parameter also works with dictionary sorts.

Quite often, files use a layout with multiple columns, and you may want to sort a file by a different column than the first column. For instance, consider the following score file named score.txt:

John:US:4
Herman:NL:3
Klaus:DE:5
Heinz:DE:3
      

Suppose that we would like to sort the entries in this file by the two-letter country name. sort allows us to sort a file by a column with the -k col1[,col2] parameter. Where col1 up to col2 are used as fields for sorting the input. If col2 is not specified, all fields up till the end of the line are used. So, if you want to use just one column, use -k col1,col1. You can also specify the the starting character within a column by adding a period (.) and a character index. For instance, -k 2.3,4.2 means that the second column starting from the third character, the third column, and the fourth column up to (and including) the second character.

There is yet another particularity when it comes to sorting by columns: by default, sort uses a blank as the column separator. If you use a different separator character, you will have to use the -t char parameter, that is used to specify the field separator.

With the -t and -k parameters combined, we can sort the scores file by country code:

$ sort -t ':' -k 2,2 scores.txt
Heinz:DE:3
Klaus:DE:5
Herman:NL:3
John:US:4
    

So, how can we sort the file by the score? Obviously, we have to ask sort to use the third column. But sort uses a dictionary sort by default^[6]. You could use the -n, but sort also allows a more sophisticated approach. You can append the one or more of the n, r>, f, d, i, or b to the column specifier. These letters represent the sort parameters with the same name. If you add just the starting column, append it to that column, otherwise, add it to the ending column.

The following command sorts the file by score:

$ sort -t ':' -k 3n /home/daniel/scores.txt
Heinz:DE:3
Herman:NL:3
John:US:4
Klaus:DE:5
    

It is good to follow this approach, rather than using the parameter variants, because sort allows you to use more than one -k parameter. And, adding these flags to the column specification, will allow you to sort by different columns in different ways. For example using sort with the -k 3,3n -k 2,2 parameters will sort all lines numerically by the third column. If some lines have identical numbers in the third column, these lines can be sorted further with a dictionary sort of the second column.

If you want to check whether a file is already sorted, you can use the -c parameter. If the file was in a sorted order, sort will return the value 0, otherwise 1. We can check this by echoing the value of the ? variable, which holds the return value of the last executed command.

$ sort -c scores.txt ; echo $?
1
$ sort scores.txt | sort -c ; echo $?
0
    

The second command shows that this actually works, by piping the output of the sort of scores.txt to sort.

Finally, you can merge two sorted files with the -m parameter, keeping the correct sort order. This is faster than concatenating both files, and resorting them.

# sort -m scores-sorted.txt scores-sorted2.txt
    

Since text streams, and text files are very important in UNIX, it is often useful to show the differences between two text files. The main utilities for working with file differences are diff and patch. diff shows the differences between files. The output of diff can be processed by patch to apply the changes between two files to a file. “diffs” are also form the base of version/source management systems. The following sections describe diff and patch. To have some material to work with, the following two C source files are used to demonstrate these commands. These files are named hello.c and hello2.c respectively.

#include <stdio.h>

void usage(char *programName);

int main(int argc, char *argv[]) {
  if (argc == 1) {
    usage(argv[0]);
    return 1;
  }

  printf("Hello %s!\n", argv[1]);

  return 0;
}

void usage(char *programName) {
  printf("Usage: %s name\n", programName);
}

      

#include <stdio.h>
#include <time.h>

void usage(char *programName);

int main(int argc, char *argv[]) {
  if (argc == 1) {
    usage(argv[0]);
    return 1;
  }

  printf("Hello %s!\n", argv[1]);

  time_t curTime = time(NULL);
  printf("The date is %s\n", asctime(localtime(&curTime)));


  return 0;
}

void usage(char *programName) {
  printf("Usage: %s name\n", programName);
}
      

$ diff hello.c hello2.c
1a2 
> #include <time.h> 
12a14,17
>   time_t curTime = time(NULL);
>   printf("The date is %s\n", asctime(localtime(&curTime)));
>
	

$ diff hello2.c hello.c
2d1 
< #include <time.h> 
14,16d12
<   time_t curTime = time(NULL);
<   printf("The date is %s\n", asctime(localtime(&curTime)));
<
	

$ diff -u hello.c hello2.c
--- hello.c     2006-11-26 20:28:55.000000000 +0100 
+++ hello2.c    2006-11-26 21:27:52.000000000 +0100 
@@ -1,4 +1,5 @@ 
 #include <stdio.h> 
+#include <time.h> 

 void usage(char *programName);

@@ -10,6 +11,9 @@

   printf("Hello %s!\n", argv[1]);

+  time_t curTime = time(NULL);
+  printf("The date is %s\n", asctime(localtime(&curTime)));
+
   return 0;
 }

	

$ diff -u hello2.c hello.c

--- hello2.c    2006-11-26 21:27:52.000000000 +0100
+++ hello.c     2006-11-26 20:28:55.000000000 +0100
@@ -1,5 +1,4 @@
 #include <stdio.h>
-#include <time.h>

 void usage(char *programName);

@@ -11,9 +10,6 @@

   printf("Hello %s!\n", argv[1]);

-  time_t curTime = time(NULL);
-  printf("The date is %s\n", asctime(localtime(&curTime)));
-
   return 0;
 }


	

$ diff -ru hello.orig hello
diff -ru hello.orig/hello.c hello/hello.c

--- hello.orig/hello.c  2006-12-04 17:37:14.000000000 +0100
+++ hello/hello.c       2006-12-04 17:37:48.000000000 +0100
@@ -1,4 +1,5 @@
 #include <stdio.h>
+#include <time.h>

 void usage(char *programName);

@@ -10,6 +11,9 @@

   printf("Hello %s!\n", argv[1]);

+  time_t curTime = time(NULL);
+  printf("The date is %s\n", asctime(localtime(&curTime)));
+
   return 0;
 }


	

$ diff -ruN hello.orig hello

diff -ruN hello.orig/hello.c hello/hello.c
--- hello.orig/hello.c  2006-12-04 17:37:14.000000000 +0100
+++ hello/hello.c       2006-12-04 17:37:48.000000000 +0100
@@ -1,4 +1,5 @@
 #include <stdio.h>
+#include <time.h>

 void usage(char *programName);

@@ -10,6 +11,9 @@

   printf("Hello %s!\n", argv[1]);

+  time_t curTime = time(NULL);
+  printf("The date is %s\n", asctime(localtime(&curTime)));
+
   return 0;
 }

diff -ruN hello.orig/Makefile hello/Makefile
--- hello.orig/Makefile 1970-01-01 01:00:00.000000000 +0100
+++ hello/Makefile      2006-12-04 17:39:44.000000000 +0100
@@ -0,0 +1,2 @@
+hello: hello.c
+       gcc -Wall -o $@ $<

	

$ diff -u hello.c hello2.c > hello_add_date.diff
	

$ cat diff1 diff2 diff3 > combined_diff
	

$ patch < hello_add_date.diff
patching file hello.c
	

$ diff -u hello.c hello2.c
	

$ patch -R < hello_add_date.diff
	

$ patch helloworld.c < hello_add_date.diff
patching file helloworld.c
	

$ cd hello.orig
$ patch -p 1 < ../hello.diff
	

$ patch -p 1 -d hello.orig < hello.diff
patching file hello.c
patching file Makefile
	

$ patch -b < hello_add_date.diff
$ ls -l hello.c*
-rw-r--r-- 1 daniel daniel 382 2006-12-04 21:41 hello.c
-rw-r--r-- 1 daniel daniel 272 2006-12-04 21:12 hello.c.orig
	

In daily life, you will often want to some text that matches to a certain pattern, rather than a literal string. Many UNIX utilities implement a language for matching text patterns, regular expressions (regexps). Over time the regular expression language has grown, there are now basically three regular expression syntaxes:

POSIX regexps are mostly a superset of traditional UNIX regexps, and PCREs a superset of POSIX regexps. The syntax that an application supports differs per application, but almost all applications support at least POSIX regexps.

Each syntactical unit in a regexp expresses one of the following things:

This section describes traditional UNIX regexps. Because of a lack of standardisation, the exact syntax may differ a bit per utility. Usually, the manual page of a command provides more detailed information about the supported basic or traditional regular expressions. It is a good idea to learn traditional regexps, but to use POSIX regexps for your own scripts.

POSIX regular expressions build upon traditional regular expressions, adding some other useful primitives. Another comforting difference is that grouping parenthesises, quantification accolades, and the alternation sign (|) are not backslash-escaped. If they are escaped, they will match the literal characters instead, thus resulting in the opposite behavior of traditional regular expressions. Most people find POSIX extended regular expressions much more comfortable, making them more widely used.

We have now arrived at one of the most important utilties of the UNIX System, and the first occasion to try and use regular expressions. The grep command is used to search a text stream or a file for a pattern. This pattern is a regular expression, and can either be a basic regular expression or a POSIX extended regular expression (when the -E parameter is used). By default, grep will write the lines that were matched to the standard output. In the most basic syntax, you can specify a regular expression as an argument, and grep will search matches in the text from the standard input. This is a nice manner to practice a bit with regular expressions.

$ grep '^\(ab\)\{2,3\}$'
ab
abab
abab
ababab
ababab
abababab
      

The example listed above shows a basic regular expression in action, that matches a line solely consisting of two or three times the ab string. You can do the same thing with POSIX extended regular expressions, by adding the -E (for extended) parameter:

$ grep -E '^(ab){2,3}$'
ab
abab
abab
ababab
ababab
abababab
      

Since the default behavior of grep is to read from the standard input, you can add it to a pipeline to get the interesting parts of the output of the preceding commands in the pipeline. For instance, if you would like to search for the string 2006 in the third column in a file, you could combine the cut and grep command:

$ cut -f 3 | grep '2006'
      

Naturally, grep can also directly read a file, rather than the standard input. As usual, this is done by adding the files to be read as the last arguments. The following example will print all lines from the /etc/passwd file that start with the string daniel:.

$ grep "^daniel" /etc/passwd
daniel:*:1001:1001:Daniel de Kok:/home/daniel:/bin/sh
      

With the -r option, grep will recursively traverse a directory structure, trying to find matches in each file that was encountered during the traversal. Though, it is better to combine grep with find and the -exec operand in scripts that have to be portable.

$ grep -r 'somepattern' somedir
      

is the non-portable functional equivalent of

$ find /somedir -type f -exec grep 'somepattern' {} \; -print
      

grep can also print all lines that do not match the pattern that was used. This is done by adding the -v parameter:

$ grep -Ev '^(ab){2,3}$'
ab
ab
abab
ababab
abababab
abababab
      

If you want to use the pattern in a case-insensitive manner, you can add the -i parameter. For example:

$ grep -i "a"
a
a
A
A
      

You can also match a string literally with the -F parameter:

$ grep -F 'aa*'
a
aa*
aa*
      

As we have seen, you can use the alternation character (|) to match either of two or more subpatterns. If two patterns that you would like to match differ a lot, it is often more comfortable to make two separate patterns. grep allows you to use more than one pattern by separating patterns with a newline character. So, for example, if you would like to print lines that match either the a or b pattern, this can be done easily by starting a new line:

$ grep 'a
b'
a
a
b
b
c
      

This works, because quotes are used, and the shell passes quoted parameters literally. Though, it must be admitted that this is not quite pretty. grep accepts one or more -e pattern parameters, giving the opportunity to specify more than one parameter on one line. The grep invocation in the previous example could be rewritten as:

$ grep -e 'a' -e 'b'