Tuesday, April 15, 2008

Remove repeated lines

Unix command:
$ uniq input.txt > result.txt
or
(sorted output)
$ sort -u input.txt > result.txt

Perl:
open (FH,$ARGV[0]) or die ("No input file\n");
read (FH,$file,-s $ARGV[0]);
$file =~ s/^(.*[\n\r]+)\1+/$1/mg;
print $file;


or
cat a.txt | perl -ne 'print unless($s{$_}++)'