Re: Datawarehouse Load Performance?

From: Stephen Darlington <stephen_at_zx81.org.uk.nospam>
Date: Tue, 8 Feb 2000 10:29:14 -0000
Message-ID: <87or5t$av3@romeo.logica.co.uk>

That'll teach me to rush writing a question... Thanks for your responses one and all, but I think I wrote the wrong question!

What we have are 1.2 million rows in the main table in the source (normalised) database, I suspect it's going to turn out to be four or five times that in the fact table. The loading, in this case, means a lot of PL/SQL crunching through it all doing some fairly complex stuff (tree traversing, date ranges). Even if it were an option, and it isn't, I don't think coding it in Pro*C would help since it's all data manipulation rather than arithmetic.

I guess this most closely mirrors "Transforming into load record images" and "Migrating from the legacy system to DDW system" (pg 217, The Datawarehouse Toolkit, Kimball). I gather that the processing we're doing is unusually complex. We're not able to do any transformations on the source system, in fact one of the criteria is that we impact them as little as possible.

The main question is what level of performance can I expect in the transformation? Are my figures way off? But the other comments have already been useful.

Thanks again,
--> Stephen

     Stephen Darlington (http://www.zx81.org.uk)
                 "Never put a sock in a toaster"
---------------------------------------------------------------

Stephen Darlington wrote in message <87m7jn$6b7_at_romeo.logica.co.uk>...
>I'm just starting to design a datawarehouse load and was beginning to get a
>little worried about the performance.
>
>What are other people's experience loading a datawarehouse? We're using 8i
>on a 4 processor (Xeon, 550Mhz) NT box and are looking to process around
1.2
>million rows. Some very rough metrics indicated that this could take a week
>to load!
>
>I appreciate that there are too many variables here to get a very precice
>answer, but I'd like to hear any experiences you have.
>
>Cheers,
>--> Stephen
>---------------------------------------------------------------
> Stephen Darlington (http://www.zx81.org.uk)
> "Never put a sock in a toaster"
>---------------------------------------------------------------
>
>
Received on Tue Feb 08 2000 - 04:29:14 CST