select does a full table scan. 130.000.000 rows [message #356526] |
Fri, 31 October 2008 00:21 |
mclu
Messages: 3 Registered: October 2008 Location: Germany
|
Junior Member |
|
|
Hi out there!
First I am a java programmer not a DBA.
So I am sorry for my knowledge regarding:
Big Database and select statement performance.
I have a Oracle 9 Database (as installed without any extra performance settings)
There is no Resource Plan active (don't know what this is anyway)
The Table is like this(script generated by toad):
CREATE TABLE mytable
(
ID INTEGER NOT NULL,
A NUMBER(10) NOT NULL,
B VARCHAR2(10 BYTE) NOT NULL,
C NUMBER(32,2) NOT NULL,
D DATE NOT NULL,
E INTEGER NOT NULL,
F INTEGER NOT NULL,
G INTEGER DEFAULT '0'
)
NOLOGGING
NOCOMPRESS
NOCACHE
NOPARALLEL
MONITORING;
I have an index on B
CREATE INDEX mytable_B ON mytable
(B)
NOLOGGING
NOPARALLEL;
There are 130.000.000 entries in the Table
PROBLEM:
I have to process the Data on Java
First I fetch all Distinct(B) of this table
and loop through all B's (around 1.000.000)
The Distinct call takes 8 Minutes.
But Then I have to fetch 1000000 times
select ID,A,C,D from mytable where B = ?
EACH CALL TAKES 5 MINUTES via java/jdbc. (time spend at oracle)
In my EM session monitor I can see:
SQL: SELECT id, a, c, d
FROM mytable
WHERE B = :1
Plan:
select mytable
Master.mytable table access (FULL)
--> so he do a table scan
If I do a query via toad it returns after seconds.
Can you give me performance hints? I thought it should use the index. jumps to the 50 resulting rows and return them.
This should not that slow, or?
Is the reason that I use a prepared statement?
THX A LOT. I want to have this computed before I am old
Markus Lutum
|
|
|
|
Re: select does a full table scan. 130.000.000 rows [message #356683 is a reply to message #356551] |
Sat, 01 November 2008 00:23 |
rleishman
Messages: 3728 Registered: October 2005 Location: Melbourne, Australia
|
Senior Member |
|
|
The :1 that you are passing into the query is a JAVA variable, right? Is it the same data type as the row in the table? ie. Are they both numbers or both strings? If they are different, your SQL will cast the VARCHAR column to a number and you will not be able to use an index.
This is very bad design and will always run poorly. See this article on PL/SQL tuning - the same theory applies to Java. If you really want to tune it properly, you will need to redesign.
Ross Leishman
|
|
|
Re: select does a full table scan. 130.000.000 rows [message #356927 is a reply to message #356683] |
Mon, 03 November 2008 04:20 |
mclu
Messages: 3 Registered: October 2008 Location: Germany
|
Junior Member |
|
|
Thx for reply and sorry because I already found out.
You are right. The table datatype was varchar and the given parameter was Java-Long.
I thought that maybe the java jdbc driver converts the data on the fly...
Thats is also the reason why it was so fast if I not use a prepared statement. In that case java converts the Long parameter to the sql string.
I did Long.toString(myParameter) while setting the parameters to my prepared statement and now it is faster....
OK It still takes 4 Days to process the database but that is ok for me...
Thx!
Markus
|
|
|
Re: select does a full table scan. 130.000.000 rows [message #357215 is a reply to message #356927] |
Tue, 04 November 2008 05:33 |
JRowbottom
Messages: 5933 Registered: June 2006 Location: Sunny North Yorkshire, ho...
|
Senior Member |
|
|
I strongly suspect that you would be better off executing a query like select ID,A,C,D from mytable ORDER BY B and looping throught the values that this returns, taking care to spot when the value of B changes.
From yuor description, you are looking at every row from the table - the quickest way to do this is a Full Table Scan, not by 1,000,000 seperate index scans.
|
|
|
Re: select does a full table scan. 130.000.000 rows [message #357217 is a reply to message #356526] |
Tue, 04 November 2008 05:34 |
michael_bialik
Messages: 621 Registered: July 2006
|
Senior Member |
|
|
Hi, Markus.
Why do you need to access the same data twice:
1. Selecting all distinct B values
2. Selecting 1M times all rows for the specific B.
Use:
It will performs full table scan (as you are doing now probably anyway dusing select distinct), but you process will perform a single pass through the data.
If an order is important - make it
SELECT * FROM mytable ORDER BY b;
HTH.
Michael
|
|
|