A question about how to code a procedure to use dbms_parallel_execute package for a huge insert [message #672876] |
Tue, 30 October 2018 09:45 ![Go to next message Go to next message](/forum/theme/orafaq/images/down.png) |
orausern
Messages: 826 Registered: December 2005
|
Senior Member |
|
|
Hi Experts,
I am on Oracle 11.2.0.4 on Linux. I have a query to tune and a scenario in which the query executes fine for smaller data sets and takes more time for huge data set - when it inserts more than one million rows. Our application timeout is 60 seconds and when the data inserts is more than 1 million we are getting timeout. I want to try out dbms_parallel_execute package for this - to change that query to use this package. This question is specifically about this requirement. I am asking this question to specifically know about how to use DBMS_PARALLEL_EXECUTE package to make an insert go in parallel. (Please do not spend your time on other aspects of tuning as I will raise a new question for it if I need help on that.)
For this I have the following setup: Basically a table with 4 column primary key (to mimic my actual issue) and a simpler insert into it from another table holding 1 million rows. But I am stumped at how to use dbms_parallel_execute for this. So many websites I referred are all giving references on DBMS_PARALLEL_EXECUTE for update statements but not for inserts. If someone can share how the insert can be paralleled - and for a table that does not have a single column primary key but a composite 4 column primary key, that will be very useful to me.
Test data setup ...the sql to be made parallel by dbms_parallel_execute is the last sql at the end of the data setup:
create table test
(owner varchar2(30),
object_name varchar2(128),
object_id number,
object_id_2 number,
create_date date default sysdate);
create table test2
(owner varchar2(30),
object_name varchar2(128),
object_id number,
object_id_2 number,
create_date date default sysdate);
alter table test add constraint test_pk primary key (owner, object_name, object_id,object_id_2);
alter table test2 add constraint test2_pk primary key (owner, object_name, object_id,object_id_2);
insert /*+ append */ into test select owner, object_name,object_id,1 ,sysdate from dba_objects where object_id is not null;
insert /*+ append */ into test select owner, object_name,object_id,2 ,sysdate from dba_objects where object_id is not null;
commit;
insert /*+ append */ into test select owner, object_name,object_id,3 ,sysdate from dba_objects where object_id is not null;
commit;
insert /*+ append */ into test select owner, object_name,object_id,4 ,sysdate from dba_objects where object_id is not null;
commit;
insert /*+ append */ into test select owner, object_name,object_id,5 ,sysdate from dba_objects where object_id is not null;
commit;
17:56:57 testuser@db>select count(*) from test;
COUNT(*)
----------
1141645
--actual sql to be made parallel:..this is to be converted to use dbms_parallel_execute packae
insert into test2 select owner, object_name,object_id,object_id_2 ,sysdate from test where object_id is not null;
|
|
|
|
Re: A question about how to code a procedure to use dbms_parallel_execute package for a huge insert [message #672878 is a reply to message #672877] |
Tue, 30 October 2018 10:33 ![Go to previous message Go to previous message](/forum/theme/orafaq/images/up.png) ![Go to next message Go to next message](/forum/theme/orafaq/images/down.png) |
orausern
Messages: 826 Registered: December 2005
|
Senior Member |
|
|
Thanks Blackswan. I want to try this out - dbms_parallel_execute. But it is only one step in final decision. I will not stick to it and will be open for other solutions - for instance I checked for append (and gave up due to its other locking issues and so on). But at least I want to try this out and I am stuck due to syntax and other issues. So this is a specific syntax related question.
Regarding PK, pls. read the question, I have posted it there , it is not based on a sequence.
|
|
|
|
|
|
Re: A question about how to code a procedure to use dbms_parallel_execute package for a huge insert [message #672882 is a reply to message #672880] |
Tue, 30 October 2018 11:30 ![Go to previous message Go to previous message](/forum/theme/orafaq/images/up.png) ![Go to next message Go to next message](/forum/theme/orafaq/images/down.png) |
cookiemonster
Messages: 13963 Registered: September 2008 Location: Rainy Manchester
|
Senior Member |
|
|
orausern wrote on Tue, 30 October 2018 15:52Thanks cookiemonster. This gives me a lot of guidance. My actual issue at work is a massive three table join so here how to paas row id?
You chunk one of the three, probably whichever is likeliest to be the biggest at any given point in time.
then your insert would be:
insert into tablea
select whatever
from table1
join table2 on ..
join table3 on ..
where table1.rowid between ... and ...
and whatever
orausern wrote on Tue, 30 October 2018 15:52
but I have a way out- What I will try out now is using this example to insert the result set first into a GTT and then use rowid from that GTT. I hope to have it working by tomorrow and will share my finding.
If you want to speed things up enough to avoid your timeout a GTT is unlikely to improve matters.
Can't you bypass the timeout for this operation? You could always submit a job so it runs in the background.
|
|
|
|
|
Re: A question about how to code a procedure to use dbms_parallel_execute package for a huge insert [message #672885 is a reply to message #672883] |
Tue, 30 October 2018 11:44 ![Go to previous message Go to previous message](/forum/theme/orafaq/images/up.png) ![Go to next message Go to next message](/forum/theme/orafaq/images/down.png) |
cookiemonster
Messages: 13963 Registered: September 2008 Location: Rainy Manchester
|
Senior Member |
|
|
orausern wrote on Tue, 30 October 2018 16:36Hi BlackSwan,
I feel bad about your comment. 60 second is not silly. In this days of competition , companies do need good performance. Dont understand why you thought 60 second timeout is silly.
60 seconds for day to day operations (making an order on a website) is fine (possibly on the high side).
60 seconds for processes that move millions of records around in one go though is completely unrealistic.
|
|
|
|
|
Re: A question about how to code a procedure to use dbms_parallel_execute package for a huge insert [message #672889 is a reply to message #672887] |
Wed, 31 October 2018 04:35 ![Go to previous message Go to previous message](/forum/theme/orafaq/images/up.png) ![Go to next message Go to next message](/forum/theme/orafaq/images/down.png) |
cookiemonster
Messages: 13963 Registered: September 2008 Location: Rainy Manchester
|
Senior Member |
|
|
True - I was probably being overly generous with the "possibly on the high side"
That said - networks can be slow, not just DBs, and most websites (if they're sensible) will disable the order button until it gets a response back.
Usually when I order something on amazon it completes in a few seconds but I have seen it take close to a minute a few times.
If the DB end is taking much more than a second or two you've got a serious problem. If the end user occasionally has to wait longer than that, not so much.
So - a timeout on the application of 60 seconds, assuming that timeout is applied on a webserver and not a client device, is probably too high for day to day operations (while being too low for batch processes).
|
|
|
Re: A question about how to code a procedure to use dbms_parallel_execute package for a huge insert [message #672895 is a reply to message #672887] |
Wed, 31 October 2018 11:15 ![Go to previous message Go to previous message](/forum/theme/orafaq/images/up.png) |
orausern
Messages: 826 Registered: December 2005
|
Senior Member |
|
|
To update:
I spent the day today working out the approaches and found the following so far:
1. DBMS_PARALLEL_EXECUTE using row id approach , for some reason was excessively bad. It was taking more than 15 minutes and then I killed the session. The actual query I have is some what complex with EXIST and NOT EXIST clauses etc. and something didn't go right. I couldn't troubleshoot further.
2. Other idea was to insert into a GTT and thereafter do a insert in the main table with select in parallel from GTT - that worked!! the timing came to 44 seconds. This was what BlackSwan suggested and that is working out for me. Thanks a lot to you both. select in parallel from the 3 tables did not work and for some reason the plan was not a parallel plan but select from GTT was a parallel plan. So this approach -of going via GTT looks good to me.
I wish I troubleshoot further and make this work with dbms_parallel_execute but I don't know. I already have a working solution now but I will explore further. Thanks a lot to you both cookiemonster and BlackSwan for your suggestions.
[Updated on: Wed, 31 October 2018 11:19] Report message to a moderator
|
|
|