RE: UNIX: remove duplicate lines

From: Mohan, Ross <MohanR_at_STARS-SMI.com>
Date: Wed, 3 Jan 2001 17:18:44 -0500
Message-Id: <10730.125645@fatcity.com>

This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible.

------_=_NextPart_001_01C075D3.22934540
Content-Type: text/plain;

charset="iso-8859-1"

"HP" -

A PERL script would do it in about 3 lines ( excluding wrappers, etc. ) Of course, TIMTOWTDI. (ask Jared!)

Hell, if it's under several hundred thousand lines, load it via sqlldr into a table, and declare the text as a primary key, enable the key, stash the violations by ROWID into another table, delete as necessary, and spool it back out to a file.

Rube Goldberg

-----Original Message-----
From: hp [mailto:hp_at_lechateau.ca]
Sent: Wednesday, January 03, 2001 4:56 PM To: Multiple recipients of list ORACLE-L Subject: UNIX: remove duplicate lines

does anyone know of a way i can remove duplicate lines from a file.

I can use the sort -u command but is there any other way to do it without sorting the file?
--

Please see the official ORACLE-L FAQ: http://www.orafaq.com
--

Author: hp
INET: hp_at_lechateau.ca

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists
--------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).

------_=_NextPart_001_01C075D3.22934540
Content-Type: text/html;

charset="iso-8859-1"

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2653.12">
<TITLE>RE: UNIX: remove duplicate lines</TITLE>
</HEAD>
<BODY>

<P><FONT SIZE=2>&quot;HP&quot; - </FONT>

<P><FONT SIZE=2>A PERL script would do it in about 3 lines ( excluding wrappers, etc. )</FONT>
<BR><FONT SIZE=2>Of course, TIMTOWTDI. (ask Jared!) </FONT>
</P>

<P><FONT SIZE=2>Hell, if it's under several hundred thousand lines, load it via </FONT>
<BR><FONT SIZE=2>sqlldr into a table, and declare the text as a primary key, enable</FONT>
<BR><FONT SIZE=2>the key, stash the violations by ROWID into another table, delete </FONT>

as necessary, and spool it back out to a file.

- Rube Goldberg

<P><FONT SIZE=2>-----Original Message-----</FONT>
<BR><FONT SIZE=2>From: hp [<A HREF="mailto:hp_at_lechateau.ca">mailto:hp_at_lechateau.ca</A>]</FONT>
<BR><FONT SIZE=2>Sent: Wednesday, January 03, 2001 4:56 PM</FONT>
<BR><FONT SIZE=2>To: Multiple recipients of list ORACLE-L</FONT>
<BR><FONT SIZE=2>Subject: UNIX: remove duplicate lines</FONT>
</P>
<BR>

does anyone know of a way i can remove duplicate lines from a file.

<P><FONT SIZE=2>I can use the sort -u command but is there any other way to do it without</FONT>
<BR><FONT SIZE=2>sorting the file?</FONT>
<BR><FONT SIZE=2>-- </FONT>
<BR><FONT SIZE=2>Please see the official ORACLE-L FAQ: http://www.orafaq.com</FONT>
<BR><FONT SIZE=2>-- </FONT>
<BR><FONT SIZE=2>Author: hp</FONT>
<BR><FONT SIZE=2>&nbsp; INET: hp_at_lechateau.ca</FONT>
</P>

<P><FONT SIZE=2>Fat City Network Services&nbsp;&nbsp;&nbsp; -- (858) 538-5051&nbsp; FAX: (858) 538-5051</FONT>
<BR><FONT SIZE=2>San Diego, California&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; -- Public Internet access / Mailing Lists</FONT>
<BR><FONT SIZE=2>--------------------------------------------------------------------</FONT>
<BR><FONT SIZE=2>To REMOVE yourself from this mailing list, send an E-Mail message</FONT>
<BR><FONT SIZE=2>to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in</FONT>
<BR><FONT SIZE=2>the message BODY, include a line containing: UNSUB ORACLE-L</FONT>

(or the name of mailing list you want to be removed from).  You may also send the HELP command for other information (like subscribing). Received on Wed Jan 03 2001 - 16:18:44 CST