Oracle-L: Re: Query results to .csv

From: Tanel Poder <tanel.poder.003_at_mail.ee>
Date: Tue, 26 Aug 2003 13:19:27 -0800
Message-ID: <F001.005CD526.20030826131927@fatcity.com>

Hi!

What do you customers care about? Usability and uptime of their app or utilization of your server?

If you use direct exp and imp method:

1) you read data from disk (server process from oracle datafiles)
2) you write data to disk (expfile)
3) you read data from disk (ftp reading expfile)
4) you send data over network (quite efficient if using ftp)
5) you write data to disk
6) you read data from disk (imp process reading expfile)
7) you write data to disk (DBWR in background
8) you write data to LGWR too since there is no direct or nologging imp

method

Using dblinks

1) you read data from disk (server process from oracle datafiles)
2) you transfer data over nework (not so efficient that ftp if untuned)
3) you write data to disk (server process doing direct writes when using

append hint)

Nothing more because of append & nologging hint no redo has to be generated (provided that you build indexes on tables afterwards)

So yes, since dblink transfer can me much more efficient, it won't probably utilize your servers so much. But you always can do the job 10 times in a row, or just start compressing ambigous big files on the same disk, then your utilization should be acceptable as well.

Btw, these silly Oracle methods can provide you valuable knowledge how Oracle works and they don't cost you money - only some time for learning. And learning time, as we all (hopefully) know, isn't wasted time.

Tanel.

Original Message ----- To: "Multiple recipients of list ORACLE-L" <ORACLE-L_at_fatcity.com> Sent: Tuesday, August 26, 2003 10:44 PM

> Tanel
> Thanks for the ideas. My simple mind says that by fully utilizing the
> hardware I can minimize the overall time. But today we were on a
conference
> call with the application vendor and they were touting their utility for
> handling this. Everyone around the table seemed pretty impressed, so maybe
I
> shouldn't be worrying about all these silly Oracle methods. Anyway I said
> that I would test their utility, but since we don't have the server ready,
> that is all a little way off.
>
> Dennis Williams
> DBA, 80%OCP, 100% DBA
> Lifetouch, Inc.
> dwilliams_at_lifetouch.com
>
>
> -----Original Message-----
> Sent: Tuesday, August 26, 2003 12:14 PM
> To: Multiple recipients of list ORACLE-L
>
>
> Hi!
>
> What is your goal? To finish data transfer in the shortest time or keep
> hardware utilization maximum during transfer?
> I think you should concentrate on keeping the time for doing data transfer
> low.
>
> Depending on your network - in case of gigabit (or 10 Gb) you could look
at
> enabling jumbo frames, which enable ethernet packets up to 9000 bytes.
Also
> set (SDU=32768) in your listener and tnsnames.oras (you can set it with
> normal 1500 byte frames as well).
>
> About parallellism, you might want to run several bulk inserts over dblink
> to fully utilize your network (fill the "gaps" when one session is busy
> inserting, thus not using the network). But if your source disk array (or
> CPUs) are slow then they might be the bottleneck.
> If you got SAN and a temporary spare server, do a BCV copy or mirror
split,
> open up several clones of a database and copy data from all of them.
>
> Also, when you have SAN, there's no need for network transfer at all - you
> just mount the filesystem with dump/exportfiles on target database and do
> the load from there. If your operating systems are different, then just
dump
> to raw device, with pipe and tar for example, or completely raw,
remembering
> your data sizes. Note that in some (older) operating systems there were
few
> blocks in beginning of device which were used (and written) by operating
> system (Tru64 had the largest I know - 64k). Thus you had to make sure you
> didn't write anything there (oseek=65536 for dd for example).
>
> If downtime isn't an issue for you, it might not be worth trying above
> recommendations, but in RVLDBs (really very large databases) all of this
can
> help a lot.
>
> Tanel.
> ----- Original Message -----
> To: "Multiple recipients of list ORACLE-L" <ORACLE-L_at_fatcity.com>
> Sent: Tuesday, August 26, 2003 5:54 PM
>
>
> > Taniel, Binley
> > Thanks for the excellent suggestions.
> > At this point we have been testing with two smaller test systems,
> moving
> > a single table at a time, but initial indications are that the
performance
> > order is:
> > 1. Perl dump to CSV / ftp / SQL*Loader
> > 2. Copy across database link
> > 3. Export/ ftp / import
> >
> > I need to re-run the tests once the target production system is
available
> to
> > re-confirm which is faster. I am pretty confident in the ability to run
> > multiple SQL*Loader and import sessions simultaneously. I am a little
> > nervous about the ability of the database link to scale to enough
> > simultaneous sessions to keep the RAID sets maxed out on the target
> system.
> > Several years ago when I was doing a large conversion I hit that limit.
> Some
> > days I wonder if we are the beneficiary or victim of our prior
experience.
> > Oh well thanks for all the good suggestions, on to testing, testing,
> > testing.
> >
> > Dennis Williams
> > DBA, 80%OCP, 100% DBA
> > Lifetouch, Inc.
> > dwilliams_at_lifetouch.com
> >
> >
> > -----Original Message-----
> > Sent: Monday, August 25, 2003 7:09 PM
> > To: Multiple recipients of list ORACLE-L
> >
> >
> > Hi!
> >
> > What about several insert /*+ APPEND NOLOGGING */ commands over database
> > link run in parallel? (Possibly over dedicated network). This is fast
and
> is
> > easier (in case you don't hit any compatibility problems). If you happen
> to
> > be running on Windows for some reason, you could try to use named pipes
> > network protocol instead of TCP as well.
> >
> > Tanel.
> >
> > ----- Original Message -----
> > To: "Multiple recipients of list ORACLE-L" <ORACLE-L_at_fatcity.com>
> > Sent: Monday, August 25, 2003 11:04 PM
> >
> >
> > > Thanks Tanel
> > > We will undoubtedly use export/import for the many small tables. We
> are
> > > looking for alternatives that will perform even faster. The insert
phase
> > > seems to be the slowest part, and that is where SQL*Loader in direct
> path
> > > really shines. Now the next issue is how to produce a CSV file as fast
> as
> > > possible, and so far it looks like Jared's Perl program is the clear
> > winner.
> > >
> > > Dennis Williams
> > > DBA, 80%OCP, 100% DBA
> > > Lifetouch, Inc.
> > > dwilliams_at_lifetouch.com
> > >
> > >
> > > -----Original Message-----
> > > Sent: Monday, August 25, 2003 1:05 PM
> > > To: Multiple recipients of list ORACLE-L
> > >
> > >
> > > Hi!
> > >
> > > Spooling from sqlplus is VERY slow.
> > > Is the source database Oracle? Then use export/import
> > > If not, is there an ODBC driver for source database? Then use Oracle
> > > heterogenous services and do your transfer directly, without any
> > > intermediate files.
> > > Or use some very expensive software for doing this simple job...
> > >
> > > Tanel.
> > > P.S. if you definitely want to spool to textfile fast, Sparky could be
> > what
> > > you want...
> > >
> > > ----- Original Message -----
> > > To: "Multiple recipients of list ORACLE-L" <ORACLE-L_at_fatcity.com>
> > > Sent: Monday, August 25, 2003 7:24 PM
> > >
> > >
> > > > Jared - Thanks for posting this. At the moment, we are preparing to
> move
> > > > large database to a new server. Based on the advice you posted
several
> > > > months ago, we have been testing SQL*Loader and as you predicted, it
> is
> >
> > > > indeed fast. But also as you predicted, using SQL*Plus to create a
CSV
> > > isn't
> > > > very fast. Am I correct in assuming the dump.sql will not be the
best
> > > choice
> > > > for large tables? We are installing perl since you mentioned that
> would
> > > > probably be much faster.
> > > >
> > > > Dennis Williams
> > > > DBA, 80%OCP, 100% DBA
> > > > Lifetouch, Inc.
> > > > dwilliams_at_lifetouch.com
> > > >
> > > >
> > > > -----Original Message-----
> > > > Sent: Monday, August 25, 2003 9:40 AM
> > > > To: Multiple recipients of list ORACLE-L
> > > >
> > > >
> > > > http://www.cybcon.com/~jkstill/util/dump/dump.html
> > > >
> > > > On Mon, 2003-08-25 at 05:39, Imran Ashraf wrote:
> > > > > Hi,
> > > > >
> > > > > Whats the best way to write the results of a SQL query to a CSV
> file?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > _________________________________________________________________
> > > > > Hotmail messages direct to your mobile phone
> > > > http://www.msn.co.uk/msnmobile
> > > > >
> > > > > --
> > > > > Please see the official ORACLE-L FAQ: http://www.orafaq.net
> > > > > --
> > > > > Author: Imran Ashraf
> > > > > INET: imran9a_at_hotmail.com
> > > > >
> > > > > Fat City Network Services -- 858-538-5051
http://www.fatcity.com
> > > > > San Diego, California -- Mailing list and web hosting
> services
> > > >
> > ---------------------------------------------------------------------
> > > > > To REMOVE yourself from this mailing list, send an E-Mail message
> > > > > to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and
in
> > > > > the message BODY, include a line containing: UNSUB ORACLE-L
> > > > > (or the name of mailing list you want to be removed from). You
may
> > > > > also send the HELP command for other information (like
subscribing).
> > > > >
> > > >
> > > >
> > > > --
> > > > Please see the official ORACLE-L FAQ: http://www.orafaq.net
> > > > --
> > > > Author: Jared Still
> > > > INET: jkstill_at_cybcon.com
> > > >
> > > > Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> > > > San Diego, California -- Mailing list and web hosting
services
> > >
> ---------------------------------------------------------------------
> > > > To REMOVE yourself from this mailing list, send an E-Mail message
> > > > to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > > > the message BODY, include a line containing: UNSUB ORACLE-L
> > > > (or the name of mailing list you want to be removed from). You may
> > > > also send the HELP command for other information (like subscribing).
> > > > --
> > > > Please see the official ORACLE-L FAQ: http://www.orafaq.net
> > > > --
> > > > Author: DENNIS WILLIAMS
> > > > INET: DWILLIAMS_at_LIFETOUCH.COM
> > > >
> > > > Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> > > > San Diego, California -- Mailing list and web hosting
services
> > >
> ---------------------------------------------------------------------
> > > > To REMOVE yourself from this mailing list, send an E-Mail message
> > > > to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > > > the message BODY, include a line containing: UNSUB ORACLE-L
> > > > (or the name of mailing list you want to be removed from). You may
> > > > also send the HELP command for other information (like subscribing).
> > > >
> > >
> > >
> > > --
> > > Please see the official ORACLE-L FAQ: http://www.orafaq.net
> > > --
> > > Author: Tanel Poder
> > > INET: tanel.poder.003_at_mail.ee
> > >
> > > Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> > > San Diego, California -- Mailing list and web hosting services
> > > ---------------------------------------------------------------------
> > > To REMOVE yourself from this mailing list, send an E-Mail message
> > > to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > > the message BODY, include a line containing: UNSUB ORACLE-L
> > > (or the name of mailing list you want to be removed from). You may
> > > also send the HELP command for other information (like subscribing).
> > > --
> > > Please see the official ORACLE-L FAQ: http://www.orafaq.net
> > > --
> > > Author: DENNIS WILLIAMS
> > > INET: DWILLIAMS_at_LIFETOUCH.COM
> > >
> > > Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> > > San Diego, California -- Mailing list and web hosting services
> > > ---------------------------------------------------------------------
> > > To REMOVE yourself from this mailing list, send an E-Mail message
> > > to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > > the message BODY, include a line containing: UNSUB ORACLE-L
> > > (or the name of mailing list you want to be removed from). You may
> > > also send the HELP command for other information (like subscribing).
> > >
> >
> >
> > --
> > Please see the official ORACLE-L FAQ: http://www.orafaq.net
> > --
> > Author: Tanel Poder
> > INET: tanel.poder.003_at_mail.ee
> >
> > Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> > San Diego, California -- Mailing list and web hosting services
> > ---------------------------------------------------------------------
> > To REMOVE yourself from this mailing list, send an E-Mail message
> > to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > the message BODY, include a line containing: UNSUB ORACLE-L
> > (or the name of mailing list you want to be removed from). You may
> > also send the HELP command for other information (like subscribing).
> > --
> > Please see the official ORACLE-L FAQ: http://www.orafaq.net
> > --
> > Author: DENNIS WILLIAMS
> > INET: DWILLIAMS_at_LIFETOUCH.COM
> >
> > Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> > San Diego, California -- Mailing list and web hosting services
> > ---------------------------------------------------------------------
> > To REMOVE yourself from this mailing list, send an E-Mail message
> > to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> > the message BODY, include a line containing: UNSUB ORACLE-L
> > (or the name of mailing list you want to be removed from). You may
> > also send the HELP command for other information (like subscribing).
> >
>
>
> --
> Please see the official ORACLE-L FAQ: http://www.orafaq.net
> --
> Author: Tanel Poder
> INET: tanel.poder.003_at_mail.ee
>
> Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> San Diego, California -- Mailing list and web hosting services
> ---------------------------------------------------------------------
> To REMOVE yourself from this mailing list, send an E-Mail message
> to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> the message BODY, include a line containing: UNSUB ORACLE-L
> (or the name of mailing list you want to be removed from). You may
> also send the HELP command for other information (like subscribing).
> --
> Please see the official ORACLE-L FAQ: http://www.orafaq.net
> --
> Author: DENNIS WILLIAMS
> INET: DWILLIAMS_at_LIFETOUCH.COM
>
> Fat City Network Services -- 858-538-5051 http://www.fatcity.com
> San Diego, California -- Mailing list and web hosting services
> ---------------------------------------------------------------------
> To REMOVE yourself from this mailing list, send an E-Mail message
> to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
> the message BODY, include a line containing: UNSUB ORACLE-L
> (or the name of mailing list you want to be removed from). You may
> also send the HELP command for other information (like subscribing).
>

-- 
Please see the official ORACLE-L FAQ: http://www.orafaq.net
-- 
Author: Tanel Poder
  INET: tanel.poder.003_at_mail.ee

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services
---------------------------------------------------------------------
To REMOVE yourself from this mailing list, send an E-Mail message
to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in
the message BODY, include a line containing: UNSUB ORACLE-L
(or the name of mailing list you want to be removed from).  You may
also send the HELP command for other information (like subscribing).

Received on Tue Aug 26 2003 - 16:19:27 CDT