uniq

remove duplicate lines from a sorted file

Liunx version BDS Mac OS X (darwin)
uniq [option]... [input [output]]

Ignore all but one of successive identical lines from input (or standard input), writing to output (or standard output).

-u
--unique
output only unique lines, (supressing lines with multiple occurances)default
-d
--repeated
output duplicate lines, i.e. not lines with only 1 occurance
-c
--count
prefix output lines by the number of occurrences
-D
--all-repeated[=dm]
output all duplicate lines
delimit-method : dm={none(default),prepend,separate}
Delimiting is done with blank lines.
-f
--skip-fields=n
A field is delimited by whitespace
Fields are skipped before chars.
-s
--skip-chars=n
for compare
-w
--check-chars=n
compare no more than n characters in lines
-i
--ignore-case
--help
--version

cat 00
11
11
2
3
4
44
44
5 
 
  uniq 00
11
2
3
4
44
5 
 uniq -c 00
      2 11
      1 2
      1 3
      1 4
      2 44
      1 5 
cat 00
11
11
2
3
4
44
44
5 
 uniq -u 00
2
3
4
5 
cat 00
11
11
2
3
4
44
44
5 
  uniq -D 00
11
11
44
44 

tersified by Dennis German


BSD Mac OS X (darwin)

uniq [-c | -d | -u] [-i] [-f n] [-s n] [input [output]]

Reads input comparing adjacent lines and writes each unique line to the output.

If input is a single dash (-) or absent, the stdout is read.
Standard output is default used for output.

Only duplicate adjacent lines are recognized, so it may be necessary to sort the file(s) first.

-u Unique lines only are output, i.e. lines with more than one adjacent occurance are NOT output (default).
-d Duplicates Only are output.
-c count is prefixed to each output line Skip the first n characters in each input line when doing comparisons.
Counting begins with 1, i.e., the first field is 1. with -f, the first n characters after the first n fields will be skipped.
Counting begins with 1, i.e., the first character is 1.

-f nIgnore the first n fields in each input line when doing comparisons.
A field is delimited by whitespace
-s n When skipping Characters or Fields, outputting Duplicates only the first line with duplicate fields is output. Beware of trailing blankes on SOME lines.

Example:

input    uniq -f1   output     
    xxx 21            xxx 21
    yyy 21            zzz 23
    zzz 23    
-i Ignore case
There seems to be no way to only compare first n characters, see
sort.

Examples

> cat ffff

11
11
2
3
4
44
44
5 
 > uniq ffff

11
2
3
4
44
5 
 
> uniq -d ffff
11
44

> uniq -u ffff

2
3
4
5 
 
> uniq -c ffff
   1 
   2 11
   1 2
   1 3
   1 4
   2 44
   1 5 
   1  

ENVIRONMENT

The LANG, LC_ALL, LC_COLLATE and LC_CTYPE environment variables affect the execution as described in environ.

EXIT STATUS

exits 0 on success, and >0 if an error occurs.