Buildfarm alarms

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Buildfarm alarms

Dave Page
Hi Andrew,

I'm just investigating a problem with beta 1 running on Windows 2K and
XP, and noticed that neither Snake or Bandicoot have built -HEAD for
nearly 3 weeks. I'm investigating why and will fix the problem, but it
strikes me that what would be useful is an alarm email from the server
to note that a run hasn't been reported for a while would have helped
spot this earlier. This could be configured with an admin-specified
maximum number of days between reports to allow for those machines that
connect far less frequently.

Does that sound feasible to you?

Regards, Dave.

Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Andrew Dunstan
Dave Page wrote:

>
> I'm just investigating a problem with beta 1 running on Windows 2K and
> XP, and noticed that neither Snake or Bandicoot have built -HEAD for
> nearly 3 weeks. I'm investigating why and will fix the problem, but it
> strikes me that what would be useful is an alarm email from the server
> to note that a run hasn't been reported for a while would have helped
> spot this earlier. This could be configured with an admin-specified
> maximum number of days between reports to allow for those machines that
> connect far less frequently.
>
> Does that sound feasible to you?
>
>


It could certainly be done. In general, I have generally taken the view
that owners have the responsibility for monitoring their own machines.
I'll think about it some more.

cheers

andrew


Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Tom Lane-2
"Andrew Dunstan" <[hidden email]> writes:
> It could certainly be done. In general, I have generally taken the view
> that owners have the responsibility for monitoring their own machines.

Sure, but providing them tools to do that seems within buildfarm's
purview.

For some types of failure, the buildfarm script could make a local
notification without bothering the server --- but a timeout on the
server side would cover a wider variety of failures, including "this
machine is dead and ought to be removed from the farm".

                        regards, tom lane

Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Dave Page
In reply to this post by Andrew Dunstan
 

> -----Original Message-----
> From: Andrew Dunstan [mailto:[hidden email]]
> Sent: 24 September 2006 03:13
> To: Dave Page
> Cc: [hidden email]
> Subject: Re: Buildfarm alarms
>
> It could certainly be done. In general, I have generally
> taken the view
> that owners have the responsibility for monitoring their own machines.
> I'll think about it some more.

We are monitoring the machine, however in this case nothing appeared
wrong to the monitoring processes - what had happened was that both had
hung or got in an inifinite loop in ECPG-check, the machine was running
just fine, and a glance at the process list showed everything I'd expect
to see during a normal run. A system for detecting lack of reports from
a member would definitely have helped in this case.

Regards, Dave

Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Andrew Dunstan
In reply to this post by Tom Lane-2
Tom Lane wrote:

> "Andrew Dunstan" <[hidden email]> writes:
>> It could certainly be done. In general, I have generally taken the view
>> that owners have the responsibility for monitoring their own machines.
>
> Sure, but providing them tools to do that seems within buildfarm's
> purview.
>
> For some types of failure, the buildfarm script could make a local
> notification without bothering the server --- but a timeout on the
> server side would cover a wider variety of failures, including "this
> machine is dead and ought to be removed from the farm".
>

Nothing gets removed. If a machine does not report on a branch for 30 days
it drops off the dashboard, but apart from that it is a retained historic
aretfact. This buildup in history has been gradually slowing down the
dashboard, in fact, but Ian Barwick tells me that he has rewritten my
lousy SQL to make it fast again, so we'll soon get that working better.

Anyway, I think we can do something fairly simply for these alarms. We'll
just have a special stanza in the config file, and a cron job that checks,
say, once a day, to see if we have exceeded the alarm period on any
machine/branch combination.

cheers

andrew




Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Joachim Wieland
In reply to this post by Dave Page
On Sun, Sep 24, 2006 at 11:51:49AM +0100, Dave Page wrote:
> wrong to the monitoring processes - what had happened was that both had
> hung or got in an inifinite loop in ECPG-check, the machine was running
> just fine

Is this still an issue? Can you provide more information? What happens if you
run ecpg-check manually? Which test hangs?


Joachim

--
Joachim Wieland                                              [hidden email]
                                                           GPG key available

Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Dave Page
 

> -----Original Message-----
> From: Joachim Wieland [mailto:[hidden email]]
> Sent: 25 September 2006 13:25
> To: Dave Page
> Cc: Andrew Dunstan; [hidden email];
> [hidden email]
> Subject: Re: [HACKERS] Buildfarm alarms
>
> On Sun, Sep 24, 2006 at 11:51:49AM +0100, Dave Page wrote:
> > wrong to the monitoring processes - what had happened was
> that both had
> > hung or got in an inifinite loop in ECPG-check, the machine
> was running
> > just fine
>
> Is this still an issue? Can you provide more information?
> What happens if you
> run ecpg-check manually? Which test hangs?

Dt_test is the one that hangs - though in actual fact what is happening
is that it's crashing and popping up a 'do you wanna debug' dialogue
which doesn't get seen in a non-interactive buildfarm run. After saying
no to that, the complete list of failed tests is (see Snake/Bandicoot's
logs for more info):

testing connect/test1.pgc                      ... FAILED (log)
testing compat_informix/dec_test.pgc           ... FAILED (output)
testing preproc/variable.pgc                   ... FAILED (log, output)
testing pgtypeslib/dt_test.pgc                 ... FAILED (log, output)
testing pgtypeslib/num_test.pgc                ... FAILED (output)
testing pgtypeslib/num_test2.pgc               ... FAILED (output)

Regards, Dave.

Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Joachim Wieland
On Mon, Sep 25, 2006 at 02:23:39PM +0100, Dave Page wrote:
> testing connect/test1.pgc                      ... FAILED (log)
> testing compat_informix/dec_test.pgc           ... FAILED (output)
> testing preproc/variable.pgc                   ... FAILED (log, output)
> testing pgtypeslib/dt_test.pgc                 ... FAILED (log, output)
> testing pgtypeslib/num_test.pgc                ... FAILED (output)
> testing pgtypeslib/num_test2.pgc               ... FAILED (output)

All should be fine now. I tested successfully with both cygwin and MinGW.


Joachim

--
Joachim Wieland                                              [hidden email]
                                                           GPG key available

Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Andrew Dunstan
In reply to this post by Andrew Dunstan
I wrote:

> Tom Lane wrote:
>  
>> "Andrew Dunstan" <[hidden email]> writes:
>>    
>>> It could certainly be done. In general, I have generally taken the view
>>> that owners have the responsibility for monitoring their own machines.
>>>      
>> Sure, but providing them tools to do that seems within buildfarm's
>> purview.
>>
>> For some types of failure, the buildfarm script could make a local
>> notification without bothering the server --- but a timeout on the
>> server side would cover a wider variety of failures, including "this
>> machine is dead and ought to be removed from the farm".
>>
>>    
>
> Nothing gets removed. If a machine does not report on a branch for 30 days
> it drops off the dashboard, but apart from that it is a retained historic
> aretfact. This buildup in history has been gradually slowing down the
> dashboard, in fact, but Ian Barwick tells me that he has rewritten my
> lousy SQL to make it fast again, so we'll soon get that working better.
>
> Anyway, I think we can do something fairly simply for these alarms. We'll
> just have a special stanza in the config file, and a cron job that checks,
> say, once a day, to see if we have exceeded the alarm period on any
> machine/branch combination.
>
>  

OK, I have a gadget to do this in place.


It looks at the config of the last build registered on each branch for a
stanza called 'alerts' that would look like this:

  alerts => {
    HEAD => { alert_after => 24, alert_every => 48 },
    REL8_1_STABLE => { alert_after => 168, alert_every => 48 },
  }

The settings are in hours, so this says that if we haven't seen  a HEAD
build in 1 day or a stable branch build in 1 week, alert the owner by
email, and keep repeating the alert in each case every 2 days.

If some intrepid buildfarm owner wants to test this out by using low
settings that would trigger an alert that would be good - the cron job
runs every hour.

cheers

andrew


Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Dave Page
 

> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of
> Andrew Dunstan
> Sent: 27 September 2006 14:56
> To: [hidden email]
> Cc: [hidden email]
> Subject: Re: [HACKERS] Buildfarm alarms
>
> If some intrepid buildfarm owner wants to test this out by using low
> settings that would trigger an alert that would be good - the
> cron job
> runs every hour.

Dunno about intrepid, but I've added the following to Snake:

alerts => {
     HEAD => { alert_after => 1, alert_every => 2 },
     REL8_1_STABLE => { alert_after => 168, alert_every => 48 },
     REL8_0_STABLE => { alert_after => 168, alert_every => 48 },
}

Thanks for your work on this.

Regards, Dave

Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Kris Jurka
In reply to this post by Andrew Dunstan


On Wed, 27 Sep 2006, Andrew Dunstan wrote:

> The settings are in hours, so this says that if we haven't seen  a HEAD build
> in 1 day or a stable branch build in 1 week, alert the owner by email, and
> keep repeating the alert in each case every 2 days.
>

How does this know if there wasn't a build because nothing in CVS changed
over that time period?  Especially on the back branches it is normal to go
weeks without a build.

Kris Jurka

Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Andrew Dunstan
Kris Jurka wrote:

>
>
> On Wed, 27 Sep 2006, Andrew Dunstan wrote:
>
>> The settings are in hours, so this says that if we haven't seen  a
>> HEAD build in 1 day or a stable branch build in 1 week, alert the
>> owner by email, and keep repeating the alert in each case every 2 days.
>>
>
> How does this know if there wasn't a build because nothing in CVS
> changed over that time period?  Especially on the back branches it is
> normal to go weeks without a build.
>
> Kris Jurka
>

Indeed. The short answer is it doesn't. But there is a buildfarm config
option to allow you to force a build every so often even if there hasn't
been a CVS change, and I'm thinking of providing an option for this to
be branch specific. The you would make this setting shorter than your
alarm period for any branch you had an alarm set for.

cheers

andrew

Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Jim Nasby-2
On Wed, Sep 27, 2006 at 01:55:21PM -0400, Andrew Dunstan wrote:

> Kris Jurka wrote:
> >
> >
> > On Wed, 27 Sep 2006, Andrew Dunstan wrote:
> >
> >> The settings are in hours, so this says that if we haven't seen  a
> >> HEAD build in 1 day or a stable branch build in 1 week, alert the
> >> owner by email, and keep repeating the alert in each case every 2 days.
> >>
> >
> > How does this know if there wasn't a build because nothing in CVS
> > changed over that time period?  Especially on the back branches it is
> > normal to go weeks without a build.
> >
> > Kris Jurka
> >
>
> Indeed. The short answer is it doesn't. But there is a buildfarm config
> option to allow you to force a build every so often even if there hasn't
> been a CVS change, and I'm thinking of providing an option for this to
> be branch specific. The you would make this setting shorter than your
> alarm period for any branch you had an alarm set for.

Another possibility is just having the client report "no CVS changes
detected" to the server, as a form of a ping.
--
Jim Nasby                                            [hidden email]
EnterpriseDB      http://enterprisedb.com      512.569.9461 (cell)

Reply | Threaded
Open this post in threaded view
|

Re: Buildfarm alarms

Andrew Dunstan
Jim C. Nasby wrote:
>
> Another possibility is just having the client report "no CVS changes
> detected" to the server, as a form of a ping.
>  

I am not going to re-architect the buildfarm client and server for this.
I think what I have done will be quite sufficient. I suspect most people
will only want alarms on HEAD anyway.

cheers

andrew

Reply | Threaded
Open this post in threaded view
|

Re: [HACKERS] Buildfarm alarms

Andrew Dunstan
In reply to this post by Dave Page
Dave Page wrote:

>  
>
>  
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]] On Behalf Of
>> Andrew Dunstan
>> Sent: 27 September 2006 14:56
>> To: [hidden email]
>> Cc: [hidden email]
>> Subject: Re: [HACKERS] Buildfarm alarms
>>
>> If some intrepid buildfarm owner wants to test this out by using low
>> settings that would trigger an alert that would be good - the
>> cron job
>> runs every hour.
>>    
>
> Dunno about intrepid, but I've added the following to Snake:
>
> alerts => {
>      HEAD => { alert_after => 1, alert_every => 2 },
>      REL8_1_STABLE => { alert_after => 168, alert_every => 48 },
>      REL8_0_STABLE => { alert_after => 168, alert_every => 48 },
> }
>
> Thanks for your work on this.
>
>
>  

It won't work until you first run with these settings, and report the
results. snake hasn't reported for > 1 day currently.

cheers

andrew


Reply | Threaded
Open this post in threaded view
|

Re: [HACKERS] Buildfarm alarms

Dave Page
 

> -----Original Message-----
> From: Andrew Dunstan [mailto:[hidden email]]
> Sent: 28 September 2006 15:17
> To: Dave Page
> Cc: [hidden email]
> Subject: Re: [HACKERS] Buildfarm alarms
>
> > alerts => {
> >      HEAD => { alert_after => 1, alert_every => 2 },
> >      REL8_1_STABLE => { alert_after => 168, alert_every => 48 },
> >      REL8_0_STABLE => { alert_after => 168, alert_every => 48 },
> > }
> >
> > Thanks for your work on this.
> >
> >
> >  
>
> It won't work until you first run with these settings, and report the
> results. snake hasn't reported for > 1 day currently.

Rats - typo in the config file. Fixed now - running a build.

Regards, Dave.