«

Hi,

I have a file with the following sample data delimited by "|" with
duplicate records:

20100430|20100429|John Smith|-0.07|-0.08|
20100430|20100429|John Smith|-0.07|-0.08|
20100430|20100429|Ashley Cole|1.09|1.08|
20100430|20100429|Bill Thompson|0.76|0.78|
20100429|20100428|Time Apache|2.10|2.24|

The first three fields "date_1", "date_2" and "name" are unique
identifiers of a record.

Is there a simple way, like a one liner to remove the duplicates such
as with "John Smith"?

Thanks in advance.

Nick Li

On Fri, 30 Apr 2010 08:55:12 -0700 (PDT), Ninja Li <> wrote:

>Hi,
>
>I have a file with the following sample data delimited by "|" with
>duplicate records:
>
>20100430|20100429|John Smith|-0.07|-0.08|
>20100430|20100429|John Smith|-0.07|-0.08|
>20100430|20100429|Ashley Cole|1.09|1.08|
>20100430|20100429|Bill Thompson|0.76|0.78|
>20100429|20100428|Time Apache|2.10|2.24|
>
>The first three fields "date_1", "date_2" and "name" are unique
>identifiers of a record.
>
>Is there a simple way, like a one liner to remove the duplicates such
>as with "John Smith"?
>
>Thanks in advance.
>
>Nick Li

I could think of a way, but it takes 2 lines, sorry.
-sln

Ninja Li <> writes:

> Hi,
>
> I have a file with the following sample data delimited by "|" with
> duplicate records:
>
> 20100430|20100429|John Smith|-0.07|-0.08|
> 20100430|20100429|John Smith|-0.07|-0.08|
> 20100430|20100429|Ashley Cole|1.09|1.08|
> 20100430|20100429|Bill Thompson|0.76|0.78|
> 20100429|20100428|Time Apache|2.10|2.24|
>
> The first three fields "date_1", "date_2" and "name" are unique
> identifiers of a record.
>
> Is there a simple way, like a one liner to remove the duplicates such
> as with "John Smith"?

Yes.

But have you tried to write a multi-line Perl program first? Moving from
a working Perl program to a one-liner might be easier than starting
straight with the one-liner.

Also read up on what the various options of perl do.

--
John Bokma j3b

Hacking & Hiking in Mexico - http://johnbokma.com/
http://castleamber.com/ - Perl & Python Development

On Fri, 30 Apr 2010 09:06:52 -0700, wrote:

>On Fri, 30 Apr 2010 08:55:12 -0700 (PDT), Ninja Li <> wrote:
>
>>Hi,
>>
>>I have a file with the following sample data delimited by "|" with
>>duplicate records:
>>
>>20100430|20100429|John Smith|-0.07|-0.08|
>>20100430|20100429|John Smith|-0.07|-0.08|
>>20100430|20100429|Ashley Cole|1.09|1.08|
>>20100430|20100429|Bill Thompson|0.76|0.78|
>>20100429|20100428|Time Apache|2.10|2.24|
>>
>>The first three fields "date_1", "date_2" and "name" are unique
>>identifiers of a record.
>>
>>Is there a simple way, like a one liner to remove the duplicates such
>>as with "John Smith"?
>>
>>Thanks in advance.
>>
>>Nick Li
>
>I could think of a way, but it takes 2 lines, sorry.

Wait, this might work.

c:\temp>perl -a -F"\|" -n -e "/^$/ and next or !exists $hash{$key = join '',@F[0
...2]} and ++$hash{$key} and print" file.txt
20100430|20100429|John Smith|-0.07|-0.08|
20100430|20100429|Ashley Cole|1.09|1.08|
20100430|20100429|Bill Thompson|0.76|0.78|
20100429|20100428|Time Apache|2.10|2.24|

c:\temp>

-sln

Ninja Li wrote:

> I have a file with the following sample data delimited by "|" with
> duplicate records:
>
> 20100430|20100429|John Smith|-0.07|-0.08|
> 20100430|20100429|John Smith|-0.07|-0.08|
> 20100430|20100429|Ashley Cole|1.09|1.08|
> 20100430|20100429|Bill Thompson|0.76|0.78|
> 20100429|20100428|Time Apache|2.10|2.24|
>
> The first three fields "date_1", "date_2" and "name" are unique
> identifiers of a record.
>
> Is there a simple way, like a one liner to remove the duplicates such
> as with "John Smith"?

If the data is as strict as presented, you can use

sort -u <input

sort <input |uniq

or simply use the whole line as a hash key:

perl -wne'$_{$_}++ or print' <input

(the first underscore is not really necessary)

--
Ruud

Ninja Li <> wrote:
>I have a file with the following sample data delimited by "|" with
>duplicate records:
>
>20100430|20100429|John Smith|-0.07|-0.08|
>20100430|20100429|John Smith|-0.07|-0.08|
>20100430|20100429|Ashley Cole|1.09|1.08|
>20100430|20100429|Bill Thompson|0.76|0.78|
>20100429|20100428|Time Apache|2.10|2.24|
>
>The first three fields "date_1", "date_2" and "name" are unique
>identifiers of a record.
>
>Is there a simple way, like a one liner to remove the duplicates such
>as with "John Smith"?

Your data is sorted already, so a simple call to 'uniq' will do the job:
http://en.wikipedia.org/wiki/Uniq

jue

On Fri, 30 Apr 2010 08:55:12 -0700 (PDT), Ninja Li <> wrote:

>Hi,
>
>I have a file with the following sample data delimited by "|" with
>duplicate records:
>
>20100430|20100429|John Smith|-0.07|-0.08|
>20100430|20100429|John Smith|-0.07|-0.08|
>20100430|20100429|Ashley Cole|1.09|1.08|
>20100430|20100429|Bill Thompson|0.76|0.78|
>20100429|20100428|Time Apache|2.10|2.24|
>
>The first three fields "date_1", "date_2" and "name" are unique
>identifiers of a record.
>
>Is there a simple way, like a one liner to remove the duplicates such
>as with "John Smith"?
>
>Thanks in advance.
>
>Nick Li

Another way:

perl -anF"\|" -e "tr/|// > 1 and ++$seen{qq<@F[0..2]>} > 1 and next or print" file.txt

-sln