SQL Loader & lookup tables issue [message #74616] |
Fri, 21 January 2005 07:43 |
OptionTrader
Messages: 20 Registered: January 2005 Location: Praha, Czech Republic
|
Junior Member |
|
|
I need to load about 30 million rows into one table of my DWH from a txt files. Every text files takes about 300 000 records.
Then (or perhaps during the loading?) I need to find (for 2 columns) a key for column in a small lookup dimension table and to replace a real value with its foreign key. I have 2 dim tables.
Currently I am doing it perhaps a bit clumsy this way:
I have working and target tables.
First I load a part of my data into ORACLE working table wih SQLLoader.
I then launch a procedure which open a cursor for all current records in the working table, then for every row it searches for that 2 foreign keys in 2 dim tables and then inserts a record into the target table.
SQL Loader works quite swiftly.
But the next step with that procedure is CPU and time consuming.
My question is : Is there any way to load my data and replace 2 values with theirs foreign keys in one ORACLE process step, i.e loading only.
All tables are nollogging and no redo log writing for tablespace. Still a pretty long time to get them in.
Thank you for help.
Svatopluk Solc
|
|
|
|
Re: SQL Loader & lookup tables issue [message #74632 is a reply to message #74616] |
Mon, 24 January 2005 11:21 |
OptionTrader
Messages: 20 Registered: January 2005 Location: Praha, Czech Republic
|
Junior Member |
|
|
Thank you very much Barbara.
Your help with SQL Loader will save me pretty much my time and my PC resources, as well.
Just another question on SQL Loader.
I need to join two fields from source text file and then load into one table column only.
I tried it with filler declaration but got an error message.
Thank you for help once again
Svatopluk
|
|
|
Re: SQL Loader & lookup tables issue [message #74633 is a reply to message #74620] |
Mon, 24 January 2005 11:23 |
OptionTrader
Messages: 20 Registered: January 2005 Location: Praha, Czech Republic
|
Junior Member |
|
|
Thank you very much Barbara.
Your help with SQL Loader will save me pretty much my time and my PC resources, as well.
Just another question on SQL Loader.
I need to join two fields from source text file and then load into one table column only.
I tried it with filler declaration but got an error message.
Thank you for help once again
Svatopluk
|
|
|
Re: SQL Loader & lookup tables issue [message #74636 is a reply to message #74632] |
Mon, 24 January 2005 18:55 |
|
Barbara Boehmer
Messages: 9102 Registered: November 2002 Location: California, USA
|
Senior Member |
|
|
I believe the two columns to be concatenated would actually have to be columns in the table. You could add these columns, load your data, then drop the columns. Please see the following example:
-- contents of abc.txt:
col1;col2;col3;col4
value1;value2;value3;value4
-- contents of test.ctl:
LOAD DATA INFILE 'abc.txt' INTO TABLE ABC
APPEND
FIELDS TERMINATED BY ';'
TRAILING NULLCOLS
(
COLUMN1,
column2,
column3,
COLUMN4,
COLUMN5 ":column2 || :column3"
)
-- table for testing:
scott@ORA92> create table abc
2 (column1 varchar2(10),
3 column4 varchar2(10),
4 column5 varchar2(20))
5 /
Table created.
-- load data:
scott@ORA92> alter table abc add (column2 varchar2(10), column3 varchar2(10))
2 /
Table altered.
scott@ORA92> host sqlldr scott/tiger control=test.ctl log=test.log
scott@ORA92> alter table abc drop column column2
2 /
Table altered.
scott@ORA92> alter table abc drop column column3
2 /
Table altered.
-- check results:
scott@ORA92> select * from abc
2 /
COLUMN1 COLUMN4 COLUMN5
---------- ---------- --------------------
col1 col4 col2col3
value1 value4 value2value3
scott@ORA92>
|
|
|
Re: SQL Loader & lookup tables issue [message #74642 is a reply to message #74636] |
Tue, 25 January 2005 02:12 |
OptionTrader
Messages: 20 Registered: January 2005 Location: Praha, Czech Republic
|
Junior Member |
|
|
Hi Barbara,
thank you very much once again for your snap help, now for this workaround.
If there is this solution to my issue only then I think that SQL Loader designers have simplified theirs job and skipped a direct concatenation Loader feature.
I find it like constructing an Indy winning car with a rung steering wheel.
Thanks and let OraGod (if EXISTS) bless you ...
Svatopluk Solc
Prague
Czech Republic
|
|
|