Re: Migrate 200 Million of rows

vslabs_at_onwe.co.za (Billy Verreynne) wrote in message news:<1a75df45.0307272215.6c3be3db_at_posting.google.com>...
> rgaffuri_at_cox.net (Ryan Gaffuri) wrote
>
> > we push 20-80 GBs of data at a time but have better bandwidth than
> > you. We found that Zip only works on files up to a certain point and
> > TAR or Compress is too slow.
>
> Strange...
>
> > What do you use to do your compression?
>
> compress/uncompress. The way I understand it, it compresses a certain
> buffer/block size at a time. So it works something like this:
> - export writes bytes into pipe
> - compress does a read of a block of x bytes size
> - the block is compressed/zipped
> - the block is written to the NFS mount

which compression program are you using? TAR? I was told its slow.
>
> Thus file size plays no role in the effectiveness of the compression.
>
> BTW, compress uses an adaptive Lempel-Ziv coding scheme.. which is the
> same one that is used by WinZip/pkzip.
>
> > btw, you may get network improvements if its possible to 'packetize'
> > your file. IE break it up into a bunch of smaller files and reassemble
> > at the end. Yes its the same total bandwidth, but it will pass through
> > the routers alot faster.
>
> Does not make sense. At IP packet level, there is no concept of a
> thing such a file. Actual IP packet size however does play a critical
> role. So whether you are transmitting small files in serial or a
> single lathe file.. that simply does not matter.
>
> Pushing a file thru in parallel can be faster if there is sufficient
> bandwidth (check the FTP RFC for the REST command).

i meant parallel. my bad on the language.
>
> > dont know if its practicaly if your sending executable files or .dbfs.
>
> Data is data is data. Whether that is data from an executable, data
> from a binary datafile, or data from a text file. At IP level, that
> does not play anty role.
Received on Mon Jul 28 2003 - 05:46:40 CDT