Home » RDBMS Server » Server Utilities » Load data from thousands of CSV files into one table (Windows 2003 Server, Oracle Database 10GR2)
Load data from thousands of CSV files into one table [message #394234] Thu, 26 March 2009 04:32 Go to next message
gbaliotis
Messages: 8
Registered: April 2007
Location: Greece
Junior Member
Dear all,

I get thousands of CSV files every day with the following name : CDR_<date>_time.csv. All these files have the same format, regarding their contents. 4 columns - VARCHAR2 type. We would like to run a script every hour which will load the data of every CSV file into a specific table. Afterwords, we will cut these csv files and paste them into a new folder.

Our Server's OS is Windows Server 2003 and our Database is Oracle 10GR2.

I would be delighted if you could help me! It is very urgent for me.

Thank you very, very much in advance!

Giorgos Baliotis
Re: Load data from thousands of CSV files into one table [message #394246 is a reply to message #394234] Thu, 26 March 2009 05:20 Go to previous messageGo to next message
babuknb
Messages: 1736
Registered: December 2005
Location: NJ
Senior Member

You need to help us by following the Posting Guidelines as stated below.
http://www.orafaq.com/forum/t/88153/0/

Why duplicate post ??

I hope your already implemented SQL Loader for loding data's from CSV to database.

Just implement in windows scheduler for automatted

>>Afterwords, we will cut these csv files and paste them into a new folder

Write small batch file

Babu
Re: Load data from thousands of CSV files into one table [message #394254 is a reply to message #394246] Thu, 26 March 2009 05:30 Go to previous messageGo to next message
gbaliotis
Messages: 8
Registered: April 2007
Location: Greece
Junior Member
I am so sorry for the 2 same posts. It was a mistake and I couldn't delete one of them.
Re: Load data from thousands of CSV files into one table [message #394257 is a reply to message #394246] Thu, 26 March 2009 05:38 Go to previous messageGo to next message
gbaliotis
Messages: 8
Registered: April 2007
Location: Greece
Junior Member
What do I have to write in the script? Can you give me an example?

I tried the following scripts, but with no success :

#imp_cdr_row.bat file

cd D:\EMM\CDRs

echo "Loading the CSV files......................."

echo ""

for /F %%rep1 in ("D:\EMM\CDRs\CDR*.csv") do

echo %%rep1

echo "......................................................"

sqlldr <user>/<password>@<SID> silent=\(HEADER,FEEDBACK\) data=$rep1 control=D:\EMM\CDRs\Insert_CDRs.ctl log=%%rep1.log bad=%%rep1.bad bindsize=505000 readsize=505000 errors=10000 direct=y

mv $rep1

# Insert_CDRs.ctl

LOAD DATA
INFILE *
APPEND INTO TABLE CDR_ROW
FIELDS TERMINATED BY COMMA
TRAILING NULLCOLS
(IMEI char,
MSISDN char,
LAC char,
CELL_ID char)

I would be delighted if you could help me.

Giorgos Baliotis

Re: Load data from thousands of CSV files into one table [message #394261 is a reply to message #394257] Thu, 26 March 2009 05:41 Go to previous messageGo to next message
gbaliotis
Messages: 8
Registered: April 2007
Location: Greece
Junior Member
The name of every CSV file is different, as it has the current date and time in it.

The files are like the following:

CDR_26032009_1200.csv
CDR_26032009_1213.csv
CDR_26032009_1227.csv
CDR_26032009_1240.csv

etc
Re: Load data from thousands of CSV files into one table [message #394271 is a reply to message #394261] Thu, 26 March 2009 06:25 Go to previous messageGo to next message
babuknb
Messages: 1736
Registered: December 2005
Location: NJ
Senior Member

>>I tried the following scripts, but with no success :

Why? Post your error details.

Post your DDL Sciprt.

Post some sample data's. We try to help you.

PS: Please use code tag for script posting

Babu
Re: Load data from thousands of CSV files into one table [message #394278 is a reply to message #394271] Thu, 26 March 2009 06:35 Go to previous messageGo to next message
gbaliotis
Messages: 8
Registered: April 2007
Location: Greece
Junior Member
The error I get executing the bat file is the following :

'imp_cdr_row.dat' is not recognized as an internal or external command,
operable program or batch file.

D:\EMM\CDRs>imp_cdr_row.bat

'#This' is not recognized as an internal or external command,
operable program or batch file.

D:\EMM\CDRs>cd D:\EMM\CDRs

D:\EMM\CDRs>echo "Loading the CSV files......................."
"Loading the CSV files......................."

D:\EMM\CDRs>echo ""
""
%rep1 was unexpected at this time.
D:\EMM\CDRs>for /F %rep1 in ("D:\EMM\CDRs\CDR*.csv") do

-----------------------------------------------

An example of these csv files is the following :

352287452611079 708937097386 4901 a3ac
354894012359240 900978997618 5701 2d8e
355137001280470 806977636126 3099 cbfb
355509019958026 406721329301 2cd1 d2ff
356503012793478 906990004124 da01 a8ec

I remain at your disposal for any clarifications.

Thank you very much!

[Updated on: Thu, 26 March 2009 06:37]

Report message to a moderator

Re: Load data from thousands of CSV files into one table [message #394280 is a reply to message #394278] Thu, 26 March 2009 06:39 Go to previous messageGo to next message
babuknb
Messages: 1736
Registered: December 2005
Location: NJ
Senior Member

Quote:
>>I tried the following scripts, but with no success :

Why? Post your error details.

Post your DDL Sciprt.


Babu
Re: Load data from thousands of CSV files into one table [message #394290 is a reply to message #394278] Thu, 26 March 2009 06:58 Go to previous messageGo to next message
Mahesh Rajendran
Messages: 10708
Registered: March 2002
Location: oracleDocoVille
Senior Member
Account Moderator
A simple example.
You can get any fancy you want.
C:\Documents and Settings\oracle>load.bat

SQL*Loader: Release 10.1.0.3.1 - Production on Thu Mar 26 07:54:25 2009

Copyright (c) 1982, 2004, Oracle.  All rights reserved.

Commit point reached - logical record count 2

SQL*Loader: Release 10.1.0.3.1 - Production on Thu Mar 26 07:54:26 2009

Copyright (c) 1982, 2004, Oracle.  All rights reserved.

Commit point reached - logical record count 1
Commit point reached - logical record count 2

SQL*Loader: Release 10.1.0.3.1 - Production on Thu Mar 26 07:54:27 2009

Copyright (c) 1982, 2004, Oracle.  All rights reserved.

Commit point reached - logical record count 1
Commit point reached - logical record count 2

C:\Documents and Settings\oracle>sqlplus scott/tiger

SQL*Plus: Release 10.1.0.3.0 - Production on Thu Mar 26 07:54:41 2009

Copyright (c) 1982, 2004, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options

SQL> select * from mytable;

        ID
----------
         1
         2
         3
         4
         5
         6

6 rows selected.

SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options


C:\Documents and Settings\oracle>type somectl.ctl
LOAD DATA
APPEND INTO TABLE mytable
FIELDS TERMINATED BY',' TRAILING NULLCOLS
(id)

C:\Documents and Settings\oracle>type 1.csv
1
2

C:\Documents and Settings\oracle>type 2.csv
3
4
C:\Documents and Settings\oracle>type 3.csv
5
6
C:\Documents and Settings\oracle>type load.bat
@echo off
FOR %%i IN (*.csv) DO sqlldr scott/tiger control=somectl.ctl data=%%i
Re: Load data from thousands of CSV files into one table [message #394309 is a reply to message #394290] Thu, 26 March 2009 07:42 Go to previous message
gbaliotis
Messages: 8
Registered: April 2007
Location: Greece
Junior Member
Thank you very much Mahesh Rajendran!!!

Your solution was the one that I wanted.

With my best regards,

Giorgos Baliotis
Previous Topic: import excel csv file into database
Next Topic: SQL LOADER for multiple table
Goto Forum:
  


Current Time: Mon Dec 23 20:21:27 CST 2024