RE: SyncSort - Unix - Datawarehousing.

From: <Witold.Iwaniec_at_atl.bluecross.ca>
Date: Fri, 15 Jun 2001 08:31:08 -0700
Message-ID: <F001.0032B74B.20010615083858@fatcity.com>

Rajaram

Look at inline comments

Rajaram <rajaram_k_at_netzero.net> on 06/15/2001 12:16:08

Please respond to ORACLE-L_at_fatcity.com

To: Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com> cc: (bcc: Witold Iwaniec/ATL_BLUECROSS_CA)

Hi,
Thanks Witold for the info. I have some more Q's.

For the Conversion part - I believe you must have used some other tools like - FilePort ( from the SyncSort company). Is it right?

No and yes...
No - I did not use any other third party tools Yes - as I said I wrote some C++ programs according to our needs. The conversion process was a mixture of our programs and syncsort. It was a project in which we merged assessment, registry of deeds, and land information centre systems and it included very serious remodelling. We used C++ programs to generate unique IDs, find matching data in different files, create records in different formats, extract/generate data that we needed, etc. And we used Syncsort to sort files, eliminate duplicate reocrds, merge files in the same format. Syncsort ability to work with tab-delimited files saved us a lot of space (and processing time). If you have a few records it's not a big deal but if you have hundreds of millions of records, a few spaces in each record make big difference...

2. Also, the conversion is from Mainframe flat file to Unix Flat file - but not directly in Oracle DB . Right?

That's correct. We received the data on tapes in EBCDIC format (some also as dfb files or flat files - dump from Oracle databases) and ended up with tab-delimited files that were loaded with SQL*Loader.

3. Last Q, Would I be right in saying that SyncSort replaces/enhances the capability provided by awk in Unix?

My exposure to awk was very limited - few excersises at school so can not answer this question.

HTH Witold

TIA,
Rajaram

-----Original Message-----

Sent: Friday, June 15, 2001 9:51 AM
To: Multiple recipients of list ORACLE-L

I used Syncsort on UNIX few years ago working on a large data conversion project - converting millions of records from mainframe to Oracle database. Syncsort was a great tool and it has number of extra options. We used it to sort files, eliminate duplicate records, merge files, extract certain columns only, etc.
And many of the files were over 500 MB, maybe even close to 1GB in size.

In my case I mixed Syncsort with C++ programs.

Should I ever work on a similar project, syncsort would definitely be part of it - well, I would like it to be part of it...

HTH Witold

Kevin Lange <kgel_at_ppoone.com> on 06/15/2001 09:51:52

Please respond to ORACLE-L_at_fatcity.com

To: Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com> cc: (bcc: Witold Iwaniec/ATL_BLUECROSS_CA)

-----Original Message-----

Hi,

I want some informarmation about SyncSort for Unix. If anyone is using sincsort under unix - I need the following details: 1. What is the basic purpose of syncsort? ( I have gone thru www.syncsort.com - So, I dont want this answer). 2. How is sync sort used in data loads or data warehousing? Is there any documentation online? Any whitepapers?

TIA,
Rajaram

--

Please see the official ORACLE-L FAQ: http://www.orafaq.com
--

Author:
INET: Witold.Iwaniec_at_atl.bluecross.ca

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists


--------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message

to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).

NetZero Platinum
No Banner Ads and Unlimited Access
Sign Up Today - Only $9.95 per month!
http://www.netzero.net
--

Please see the official ORACLE-L FAQ: http://www.orafaq.com
--

Author: Rajaram
INET: rajaram_k_at_netzero.net

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists


--------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message

--

Please see the official ORACLE-L FAQ: http://www.orafaq.com
--

Author:
INET: Witold.Iwaniec_at_atl.bluecross.ca

Fat City Network Services    -- (858) 538-5051  FAX: (858) 538-5051
San Diego, California        -- Public Internet access / Mailing Lists


--------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message