Re: Optimal degree of parallelism

"Billy Verreynne" <vslabs_at_onwe.co.za> wrote in message news:1a75df45.0309012200.6d78e0c9_at_posting.google.com...
>
> So just why in hell do many people seem to think that PQ can only be
> used if you have more than one CPU and that the number of PQ slaves
> may not exceed the number of CPUs?

Because "the book" says so? And whatever "the book" says shall not be questioned?

>
> CPU capacity. How many horses are there? If the CPU (or CPUs) is
> running at 80%, then it would be unwise to use PQ as that that will
> redline the CPU (or CPUs).

Unwise but not incorrect. I don't have the slightest problem with redlining a CPU. It's not like it's gonna start sweating! Provided we don't end up with a CPU resource queue 10 deep, nothing wrong with redlining it. Disks are different, of course: they involve a mechanical process in their operation that can be put under stress due to constant overuse.

> bottlenecking the disk i/o subsystem? You will find that for example 5
> threads is faster than 7 threads - and when using 10 or more the
> performance actually goes downs (you will also see an increase in CPU
> waiting for i/o times).

That is indeed the single most important criteria: length of resource queues. If a CPU resource has a 0 length queue and I have a feature that will use it for overall throughput, then I'll use it and stuff if that redlines. If there is a significant queue length, then there is no point in adding much more to it. Simple rule, guaranteed results. What is a "significant" queue length? One that is consistently and constantly above 2. Ie, one that means doubling the CPU resource would bring the queue back to <2.

Of course, one can proceed to then spend 80% of the budget tuning the remaining 20% of performance. Then again, others just add more resource: it's a lot cheaper.

> So to find the optimal degree of parallelism depends on a host of
> factors. And these are unique to your platform and your data. No magic
> wands. No magic numbers.

Well, I'd say there is no such thing as a "fixed" optimal degree of anything. There is a middle ground value that should be clamped at, depending on the nature of the load, and left alone. Anything over that is entering the realm of micro-management. Under that and we're into underuse territory. It's a simple engineering principle too.

Cheers
Nuno Souto
wizofoz2k_at_yahoo.com.au.nospam Received on Tue Sep 02 2003 - 04:35:47 CDT