distributed performance testing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

distributed performance testing

Andrew Dunstan

A little while ago I rather rashly opined that we could extend the
buildfarm to do (optional) performance testing. After thinking about it
for some time I now think that might not be such a good idea. We can
certainly leverage a lot of the technology used and experience gained in
building the buildfarm to create a distributed performance testing
system, and we know that many among the buildfarm members would be
willing to join such an effort, but I now think it probably needs to be
a separate beast.

Buildfarm tests correctness on various code branches, for a fairly
static config set per member, triggered by code changes.  It has a
number of hoops to  jump through, each of which has a well defined set
of success criteria. By contrast,  performance testing to be useful
needs to be more flexible in several ways - including the code tested
(ideally, we'd be able to try out several patch sets) and the tests
performed. What is more, measurement is on a continuous scale rather
than a set of boolean flags. And the trigger to run tests needs to be
something other than changes in code on a well known branch.

Incidentally,  use of a different SCM system might well make
constructing test sets more simple. Imagine, say, in SVN, you would
create a branch called "test-set-yyyy-mm-dd" or some such, make your
changes there, add a test script under some well known name, and commit
the branch. Then you might put an entry on the server that contains the
name of the branch and a date/time after which we would consider the
tests stale. The performance-farm member would  periodically check with
the server for new entries, switch to the named branch, build, run the
tests, and upload the results.

If someone wants to have a go at implementing such a system, I can give
them some help and advice. Otherwise I will put it on my list of things
to do, but the earliest I could devote any time to it would be very late
this year.

cheers

andrew

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

               http://www.postgresql.org/docs/faq
Reply | Threaded
Open this post in threaded view
|

Re: distributed performance testing

Tom Lane-2
Andrew Dunstan <[hidden email]> writes:
> Incidentally,  use of a different SCM system might well make
> constructing test sets more simple. Imagine, say, in SVN, you would
> create a branch called "test-set-yyyy-mm-dd" or some such, make your
> changes there, add a test script under some well known name, and commit
> the branch.

This seems a pretty unconvincing argument for SVN ... we could perfectly
well do that in CVS, no?

                        regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend
Reply | Threaded
Open this post in threaded view
|

Re: distributed performance testing

Andrew Dunstan


Tom Lane wrote:

>Andrew Dunstan <[hidden email]> writes:
>  
>
>>Incidentally,  use of a different SCM system might well make
>>constructing test sets more simple. Imagine, say, in SVN, you would
>>create a branch called "test-set-yyyy-mm-dd" or some such, make your
>>changes there, add a test script under some well known name, and commit
>>the branch.
>>    
>>
>
>This seems a pretty unconvincing argument for SVN ... we could perfectly
>well do that in CVS, no?
>
>
>  
>

You could well be right.  And I should know better than to engage in
speculation like that and possibly cause us to be inundated by
proponents of SCMfoo, most of whom will never write a line of code for
postgres let alone ever have to commit anything ;-) Pardon my weakness.

cheers

andrew



---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster
Reply | Threaded
Open this post in threaded view
|

Re: distributed performance testing

Jim C. Nasby
In reply to this post by Tom Lane-2
On Sat, Aug 13, 2005 at 06:29:58PM -0400, Tom Lane wrote:
> Andrew Dunstan <[hidden email]> writes:
> > Incidentally,  use of a different SCM system might well make
> > constructing test sets more simple. Imagine, say, in SVN, you would
> > create a branch called "test-set-yyyy-mm-dd" or some such, make your
> > changes there, add a test script under some well known name, and commit
> > the branch.
>
> This seems a pretty unconvincing argument for SVN ... we could perfectly
> well do that in CVS, no?

In any case, I think it probably makes more sense to specify tests as
'pull from CVS as of this date/tag and then (optionally) apply these
patches'. It doesn't seem to make sense to clutter up CVS just to be
able to run performance tests.

In any case, I agree. I've been wondering if it makes sense to setup a
result repository for dbt* where people running dbt tests could submit
results (along with machine config, etc). ISTM that having that would be
beneficial on it's own, and we could then build an additional framework
for pushing desired tests out to a set of machines.

Of course we could use pgbench for this instead of dbt*, but ISTM that
dbt is a better choice since it's useful for a broader set of people.
The downside is it requires dbt, but that doesn't seem to be a major
issue. Also, using dbt means we can test different use cases (dbt2 ~=
TPC-C, dbt3 ~= TPC-H, etc), while pgbench is just a single benchmark.
--
Jim C. Nasby, Sr. Engineering Consultant      [hidden email]
Pervasive Software        http://pervasive.com        512-569-9461

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

               http://www.postgresql.org/docs/faq
Reply | Threaded
Open this post in threaded view
|

Re: distributed performance testing

Darcy Buskermolen
On Monday 22 August 2005 13:13, Jim C. Nasby wrote:

> On Sat, Aug 13, 2005 at 06:29:58PM -0400, Tom Lane wrote:
> > Andrew Dunstan <[hidden email]> writes:
> > > Incidentally,  use of a different SCM system might well make
> > > constructing test sets more simple. Imagine, say, in SVN, you would
> > > create a branch called "test-set-yyyy-mm-dd" or some such, make your
> > > changes there, add a test script under some well known name, and commit
> > > the branch.
> >
> > This seems a pretty unconvincing argument for SVN ... we could perfectly
> > well do that in CVS, no?
>
> In any case, I think it probably makes more sense to specify tests as
> 'pull from CVS as of this date/tag and then (optionally) apply these
> patches'. It doesn't seem to make sense to clutter up CVS just to be
> able to run performance tests.
>
> In any case, I agree. I've been wondering if it makes sense to setup a
> result repository for dbt* where people running dbt tests could submit
> results (along with machine config, etc). ISTM that having that would be
> beneficial on it's own, and we could then build an additional framework
> for pushing desired tests out to a set of machines.
>
> Of course we could use pgbench for this instead of dbt*, but ISTM that
> dbt is a better choice since it's useful for a broader set of people.
> The downside is it requires dbt, but that doesn't seem to be a major
> issue. Also, using dbt means we can test different use cases (dbt2 ~=
> TPC-C, dbt3 ~= TPC-H, etc), while pgbench is just a single benchmark.

And there is always http://pgfoundry.org/projects/tpc-w-php/  for a ~= TPC-W
workload.

--
Darcy Buskermolen
Wavefire Technologies Corp.

http://www.wavefire.com
ph: 250.717.0200
fx: 250.763.1759

---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

               http://www.postgresql.org/docs/faq
Reply | Threaded
Open this post in threaded view
|

Re: distributed performance testing

Jim C. Nasby
On Mon, Aug 22, 2005 at 02:28:54PM -0700, Darcy Buskermolen wrote:
> On Monday 22 August 2005 13:13, Jim C. Nasby wrote:
> > Of course we could use pgbench for this instead of dbt*, but ISTM that
> > dbt is a better choice since it's useful for a broader set of people.
> > The downside is it requires dbt, but that doesn't seem to be a major
> > issue. Also, using dbt means we can test different use cases (dbt2 ~=
> > TPC-C, dbt3 ~= TPC-H, etc), while pgbench is just a single benchmark.
>
> And there is always http://pgfoundry.org/projects/tpc-w-php/  for a ~= TPC-W
> workload.

True, but then you don't get TPC-C, and dbt1 is ~= TPC-W. So with a
package of the full dbt suite (doesn't exist yet, but I suspect it
wouldn't be hard to change that), you get W, C, H, and eventually
TPC-App. Plus, much of what needs to be developed for our use-case would
benefit all dbt users, whereas pgbench is only of use to us internally.
dbt is also more flexable, since you can vary workload ratios. For
example, you can run dbt2 in a 90% read environment if you wanted.
--
Jim C. Nasby, Sr. Engineering Consultant      [hidden email]
Pervasive Software        http://pervasive.com        512-569-9461

---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings