| Oracle FAQ | Your Portal to the Oracle Knowledge Grid | |
 |
 | |||
| Oracle FAQ | Your Portal to the Oracle Knowledge Grid | |
|
| |||
Home -> Community -> Mailing Lists -> Oracle-L -> Re: Flat file generation integrity ideas...
When you get files from an external source no amount of checking is enough.
Yechiel Adar
Mehish
----- Original Message -----
To: Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com>
Sent: Tuesday, October 29, 2002 6:19 PM
Actually, I am sending and receiving files.
They will be handling the FTP of the files, and making sure it has FTPed correctly.
I just have to have a sanity check of the file. Basically, I decided to prefix each data line with 'DAT', and the "CRC line" with 'CRC'. The flat file is "read into the database" via an external table. I query the CRC record, get the expected record count and then count how many rows were actually sent. And then if there is a number column, then I sum that up and check it against the "CRC" expected sum.
Perhaps all this is overkill, but I know that the odds of data corruption are slim to none. I don't like making assumptions, and I can't assume the file is ok.
Many thanks to everyone that responded!!!
-----Original Message-----
Sent: Monday, October 28, 2002 11:54 AM
To: Multiple recipients of list ORACLE-L
But he was talking about sending, not receiving. and he says that the ftp is assured to work ok.
Yechiel Adar
Mehish
----- Original Message -----
To: Multiple recipients of list ORACLE-L <ORACLE-L_at_fatcity.com>
Sent: Friday, October 25, 2002 2:48 AM
> Unfortunately it is a trust issue...
>
> Trust me when I say a file can get scrambled. I have seen it happen.
> In our wierdest scenario two received files appeared to be merged into
> a single file - on the source system they had two intact files, on our
system
> 1.5 files merged into a single file and .5 of a file missing.
>
> We could never replicate it, we had extensive testing on ftp
> processes, etc, all we know is that it happened and our validation
> techniques saw it and saved us a lot of greif.
>
>
>
>
>
> "Yechiel Adar"
> <adar76_at_inter. To: Multiple recipients
of
list ORACLE-L <ORACLE-L_at_fatcity.com>
> net.il> cc:
> Sent by: Subject: Re: Flat file
generation integrity ideas...
> root_at_fatcity.c
> om
>
>
> 25/10/2002
> 06:14
> Please respond
> to ORACLE-L
>
>
>
>
>
>
> I do not see how the file can get "scrambled".
> You write it out ok.
> The ftp is guaranteed.
> So what is the problem.
>
> I will go along with the suggestion to zip it. It saves on the ftp
> time
and
> also gives you some protection.
>
> Yechiel Adar
> Mehish
> ----- Original Message -----
> From: Grabowy, Chris
> To: Multiple recipients of list ORACLE-L
> Sent: Thursday, October 24, 2002 7:34 PM
> Subject: Flat file generation integrity ideas...
>
>
>
> I have to create packages that will generate several flat files of
> data from tables that will be sent to other systems to be processed.
>
>
> I am looking for ideas on how to ensure data integrity in the flat
> files.
>
>
> For example, the expected record count is stored on the first line of
> the file to ensure that the correct amount of records was received.
>
>
> The systems group is chartered to ensure the flat files are correctly
> FTPed between systems, so that's covered.
>
>
> I just worry that if "somehow" a flat file is scrambled then the
scrambled
> data is loaded into the database, therefore corrupting it.
>
>
> At this phase, XML is not an option
>
>
> I keep thinking that some sort of CRC should be stored with each line
> in the flat file. And then before the line is loaded into the
> database, the CRC is compared against the generated CRC of the just
> read line. Has anyone done anything like this? Any examples out
> there?
>
>
> Many TIA!!
>
>
>
>
>
>
>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<---->>>>>>>>>>>>>>>>>>>>>>>>>>>>>
-- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Yechiel Adar INET: adar76_at_inter.net.il Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Grabowy, Chris INET: cgrabowy_at_fcg.com Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing). -- Please see the official ORACLE-L FAQ: http://www.orafaq.com -- Author: Yechiel Adar INET: adar76_at_inter.net.il Fat City Network Services -- 858-538-5051 http://www.fatcity.com San Diego, California -- Mailing list and web hosting services --------------------------------------------------------------------- To REMOVE yourself from this mailing list, send an E-Mail message to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).Received on Wed Oct 30 2002 - 09:54:10 CST
 |
 |