Re: Flat file generation integrity ideas...

Unfortunately it is a trust issue...

Trust me when I say a file can get scrambled. I have seen it happen. In our wierdest scenario two received files appeared to be merged into a single file - on the source system they had two intact files, on our system 1.5 files merged into a single file and .5 of a file missing.

We could never replicate it, we had extensive testing on ftp processes, etc, all we know is that it happened and our validation techniques saw it and saved us a lot of greif.

                                                                                                                   
                    "Yechiel Adar"                                                                                 
                    <adar76_at_inter.       To:     Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com>       
                    net.il>              cc:                                                                       
                    Sent by:             Subject:     Re: Flat file generation integrity ideas...                  
                    root_at_fatcity.c                                                                                 
                    om                                                                                             
                                                                                                                   
                                                                                                                   
                    25/10/2002                                                                                     
                    06:14                                                                                          
                    Please respond                                                                                 
                    to ORACLE-L                                                                                    
                                                                                                                   
                                                                                                                   

I do not see how the file can get "scrambled". You write it out ok.
The ftp is guaranteed.
So what is the problem.

I will go along with the suggestion to zip it. It saves on the ftp time and also gives you some protection.

Yechiel Adar
Mehish

• Original Message ----- From: Grabowy, Chris To: Multiple recipients of list ORACLE-L Sent: Thursday, October 24, 2002 7:34 PM Subject: Flat file generation integrity ideas...

 I have to create packages that will generate several flat files of data  from tables that will be sent to other systems to be processed.

 I am looking for ideas on how to ensure data integrity in the flat files.

 For example, the expected record count is stored on the first line of the  file to ensure that the correct amount of records was received.

 The systems group is chartered to ensure the flat files are correctly  FTPed between systems, so that's covered.

 I just worry that if "somehow" a flat file is scrambled then the scrambled  data is loaded into the database, therefore corrupting it.

 At this phase, XML is not an option

 I keep thinking that some sort of CRC should be stored with each line in  the flat file. And then before the line is loaded into the database, the  CRC is compared against the generated CRC of the just read line. Has  anyone done anything like this? Any examples out there?

 Many TIA!!

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

   Privileged/Confidential information may be contained in this message.

          If you are not the addressee indicated in this message
       (or responsible for delivery of the message to such person),
            you may not copy or deliver this message to anyone.
In such case, you should destroy this message and kindly notify the sender
           by reply e-mail or by telephone on (61 3) 9612-6999.
   Please advise immediately if you or your employer does not consent to
                Internet e-mail for messages of this kind.
        Opinions, conclusions and other information in this message
              that do not relate to the official business of
                         Transurban City Link Ltd
         shall be understood as neither given nor endorsed by it.
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.com
-- 
Author: Mark Richard
  INET: mrichard_at_transurban.com.au

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services
---------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).