Re: A kind of... benchmark?

From: Phil <google_at_philgra.org>
Date: 7 Aug 2003 20:13:18 -0700
Message-ID: <6b1894ca.0308071913.5d96bb14@posting.google.com>

Rick Denoire <100.17706_at_germanynet.de> wrote in message news:<5rg5jvgsnhaofispfvbdotpbrtlbj4qruq_at_4ax.com>...

>

> I really don't understand why you guys get that philosophical. As I
> explained, my only concern was to gain OBJECTIVE, REPRODUCIBLE,
> ARTIFICIALLY consistent metrics about how a number of typical
> operations in a database performs.
>
> To put it simple, let's assume that I want to assess the effect of
> using locally managed tablespaces instead of dictionary managed. Why
> the hell would it be senseless to run such a benchmak in databases
> with corresonding tablespaces and compare the results?
>
> Even simpler. If we purchased a new server - something totally
> different (OS, Harddisks, CPU) and we were asked by the managers (who
> understand NOTHING about databases) how much faster this system is, I
> could tell them based on the benchmark. Then, they would know if
> spending the money was a right decision.
>
> The simplest case. Run the benchmark. Reorganize the database. Rerun
> the benchmark. Kill a myth or stick at it.
>
> Run the benchmark. Setup parallelism. Rerun the benchmark. Got it?
>
> By the way, your answers disrespect my declared intention. OK, you
> wouldn't use such a benchmark. But I would, so if you wish to
> contribute dispite your opinion, you're welcome. What I asked was, do
> you know a set of SQL statements compiled specially to probe
> performance of typical database operations? Nothing more than that.
>
> Thanks again
> Rick Denoire

Rick,

I think that the frustration that you have is related to the fact that such a standard benchmark doesn't really exist. There isn't any industry pressure to create such a benchmark, for the reasons that previous posters have outlined (albeit not in those terms).

There are (as you've pointed out) industry-standard benchmarks like TPC, SAP, and Oracle Applications. These are meant to allow comparisons between different platforms, *not* different database configurations, which is what I get the impression you want to know (in addition to different hardware configs).

Your best bet would be to use a "canned" benchmark like what is defined in the TPC and play around with different database configurations. Unfortunately, you need to be a member of TPC in order to gain access to the full toolset which would allow you to put together a "correct" TPC result yourself. Oracle has a "TPC toolkit" that they create so that hardware vendors will end up with the most optimal database configuration. I don't believe this is available to the public...I had a hard enough time getting my hands on it when I still worked for Oracle!

Within TPC, there are a handful of different benchmarks to allow comparisons for different business environments...so there isn't any *one* answer that is right. If all you want to do is compare hardware platforms with like comparisons, TPC is the place to go (www.tpc.org).  But you want more than that...

In which case the answer is "build your own." Ask yourself "what is important to know", and come up with a series of tests that will teach you what you want to know.

Sorry there isn't an easy answer here...

:-Phil Received on Thu Aug 07 2003 - 22:13:18 CDT