BuildFarm client release 4.19

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

BuildFarm client release 4.19

Andrew Dunstan-8

I have released version 4.19 of the PostgreSQL Buildfarm client. It can
be downloaded from
<https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz>


Apart from some minor bug fixes, the following changes are made:

  * Include the script's path in @INC. That means you can usually run
    the script from anywhere rather than just its own directory.
  * Set TZ after "make check" is run. this makes for slightly faster
    initdb runs in later stages.
  * make TAP tests run with --timer
  * change default log_line_prefix in config file to use %p instead of
    %c. That makes it easier to tie log messages to other messages that
    mention a pid
  * Add a module to log running commands to a file as it runs and
    replace critical uses of `` with the new procedure. That means we
    have better traces if the buildfarm process crashes.
  * Improvements to TAP tests logging and coverage. Each test set (i.e
    each /t directory) is now logged separately. Coverage is extended to
    remaining test sets in src/test except SSL tests.
  * Add a module for testing ICU
  * change target URL in config file sample to use canonical name
    matching SSL certificate
  * default build_root to a directory in the same place that the script
    is found, and create it if it doesn't exist.
  * add a command-line flag to allow setting config settings
  * from-source path can now be relative
  * from-source doesn't need to specify a branch, if not clear it
    defaults to HEAD
  * from-source can now do vpath builds


These changes mean that the client is more useful for testing
development code, and also that testing config settings is much simpler.
An initial test run on a fresh installation is now as simple as:

    cp buildfarm.conf.sample build-farm.conf

    ./run_build.pl --test --verbose

To test development code, the from-source option is now much more
flexible and friendly. For example, one might do something like:

    ./run_build.pl --from-source=/path/to/postgresql \

       --config-set use_vpath=1 \

       --config-set config_opts+=--enable-tap-tests \

       --config-set locales+=en_US.utf8


If you run something like this against your development code, and it
succeeds, you can be reasonably confident that if committed it won't
sent the buildfarm all red.

cheers

andrew

--
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



--
Sent via buildfarm-members mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members

Reply | Threaded
Open this post in threaded view
|

Re: BuildFarm client release 4.19

Tom Lane-2
Andrew Dunstan <[hidden email]> writes:
> I have released version 4.19 of the PostgreSQL Buildfarm client. It can
> be downloaded from
> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz>

Nice work!

>   * Improvements to TAP tests logging and coverage. Each test set (i.e
>     each /t directory) is now logged separately. Coverage is extended to
>     remaining test sets in src/test except SSL tests.

I was going to plead with buildfarm owners who use --enable-tap-tests to
update sooner rather than later, because this separate-logging behavior
is going to make it a lot less painful to decipher failures during the
TAP tests.  However, in view of my results in
https://www.postgresql.org/message-id/21358.1492622881%40sss.pgh.pa.us
I can't really recommend that owners of slower machines update just yet.
Hopefully we can do something to trim down the added runtime.

> These changes mean that the client is more useful for testing
> development code, and also that testing config settings is much simpler.
> An initial test run on a fresh installation is now as simple as:
> ...
> To test development code, the from-source option is now much more
> flexible and friendly. For example, one might do something like:

These things are pretty cool, but maybe they need to be documented at
https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto
?

                        regards, tom lane


--
Sent via buildfarm-members mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members

Reply | Threaded
Open this post in threaded view
|

Re: [buildfarm-members] BuildFarm client release 4.19

Andrew Dunstan-8


On 04/19/2017 01:38 PM, Tom Lane wrote:
> Andrew Dunstan <[hidden email]> writes:
>> I have released version 4.19 of the PostgreSQL Buildfarm client. It can
>> be downloaded from
>> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz>
> Nice work!
>


Thank you.


>>   * Improvements to TAP tests logging and coverage. Each test set (i.e
>>     each /t directory) is now logged separately. Coverage is extended to
>>     remaining test sets in src/test except SSL tests.
> I was going to plead with buildfarm owners who use --enable-tap-tests to
> update sooner rather than later, because this separate-logging behavior
> is going to make it a lot less painful to decipher failures during the
> TAP tests.  However, in view of my results in
> https://www.postgresql.org/message-id/21358.1492622881%40sss.pgh.pa.us
> I can't really recommend that owners of slower machines update just yet.
> Hopefully we can do something to trim down the added runtime.


As I mentioned in reply to that thread, you can disable the extra TAP
tests with this addition to the command line:

    --skip-steps=misc-check


>
>> These changes mean that the client is more useful for testing
>> development code, and also that testing config settings is much simpler.
>> An initial test run on a fresh installation is now as simple as:
>> ...
>> To test development code, the from-source option is now much more
>> flexible and friendly. For example, one might do something like:
> These things are pretty cool, but maybe they need to be documented at
> https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto
> ?
>
>



Indeed. I am planning to.

cheers

andrew


--
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Reply | Threaded
Open this post in threaded view
|

Re: [buildfarm-members] BuildFarm client release 4.19

Mikael Kjellström
In reply to this post by Andrew Dunstan-8
On 2017-04-19 15:59, Andrew Dunstan wrote:


> I have released version 4.19 of the PostgreSQL Buildfarm client. It can
> be downloaded from
> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz>

I don't know if it's only me or if others have noticed this also but I
have the buildfarm client set up like this:

/home/pgbf
/home/pgbf/build-farm-4
/home/pgbf/buildfarm@ -> build-farm-4.19/
/home/pgbf/buildroot

and then I have a cron job that does this:

x y * * * cd /home/pgbf/buildfarm && ./run_build.pl REL9_6_STABLE

and when it tries to run I get the following error:

cannot create
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
No such file or directory
gmake is not GNU Make - please fix config file at ./run_build.pl line 343.

if I manually create the directory loach.lastrun-logs the script then
continues normally.

I am no perl guru so I can't figure out really what is going on here.  I
think that the directory
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs needs to be
created before the PGBuild/Utils.pm sub run_log tries to write to it.

/Mikael


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Reply | Threaded
Open this post in threaded view
|

Re: [buildfarm-members] BuildFarm client release 4.19

Andrew Dunstan-8


On 05/01/2017 02:13 PM, Mikael Kjellström wrote:

> On 2017-04-19 15:59, Andrew Dunstan wrote:
>
>
>> I have released version 4.19 of the PostgreSQL Buildfarm client. It can
>> be downloaded from
>> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz>
>>
>
> I don't know if it's only me or if others have noticed this also but I
> have the buildfarm client set up like this:
>
> /home/pgbf
> /home/pgbf/build-farm-4
> /home/pgbf/buildfarm@ -> build-farm-4.19/
> /home/pgbf/buildroot
>
> and then I have a cron job that does this:
>
> x y * * * cd /home/pgbf/buildfarm && ./run_build.pl REL9_6_STABLE
>
> and when it tries to run I get the following error:
>
> cannot create
> /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
> No such file or directory
> gmake is not GNU Make - please fix config file at ./run_build.pl line
> 343.
>
> if I manually create the directory loach.lastrun-logs the script then
> continues normally.
>
> I am no perl guru so I can't figure out really what is going on here.
> I think that the directory
> /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs needs to be
> created before the PGBuild/Utils.pm sub run_log tries to write to it.
>


I will check on it. Thanks for the report.

cheers

andrew

--
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Reply | Threaded
Open this post in threaded view
|

Re: BuildFarm client release 4.19

Tom Lane-2
In reply to this post by Mikael Kjellström
=?UTF-8?Q?Mikael_Kjellstr=c3=b6m?= <[hidden email]> writes:
> I don't know if it's only me or if others have noticed this also but I
> have the buildfarm client set up like this:
> ...
> and when it tries to run I get the following error:
> cannot create
> /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
> No such file or directory
> gmake is not GNU Make - please fix config file at ./run_build.pl line 343.

> if I manually create the directory loach.lastrun-logs the script then
> continues normally.

FWIW, I haven't seen that, but probably that's because the directory
already exists in each branch's working directory.  I bet zapping the
whole working directory for a branch would reproduce it.

                        regards, tom lane


--
Sent via buildfarm-members mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members

Reply | Threaded
Open this post in threaded view
|

Re: BuildFarm client release 4.19

Andrew Dunstan-8
In reply to this post by Mikael Kjellström


On 05/01/2017 02:13 PM, Mikael Kjellström wrote:

> On 2017-04-19 15:59, Andrew Dunstan wrote:
>
>
>> I have released version 4.19 of the PostgreSQL Buildfarm client. It can
>> be downloaded from
>> <https://buildfarm.postgresql.org/downloads/releases/build-farm-4_19.tgz>
>>
>
> I don't know if it's only me or if others have noticed this also but I
> have the buildfarm client set up like this:
>
> /home/pgbf
> /home/pgbf/build-farm-4
> /home/pgbf/buildfarm@ -> build-farm-4.19/
> /home/pgbf/buildroot
>
> and then I have a cron job that does this:
>
> x y * * * cd /home/pgbf/buildfarm && ./run_build.pl REL9_6_STABLE
>
> and when it tries to run I get the following error:
>
> cannot create
> /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
> No such file or directory
> gmake is not GNU Make - please fix config file at ./run_build.pl line
> 343.
>
> if I manually create the directory loach.lastrun-logs the script then
> continues normally.
>
> I am no perl guru so I can't figure out really what is going on here.
> I think that the directory
> /usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs needs to be
> created before the PGBuild/Utils.pm sub run_log tries to write to it.
>
>


OK, that's a bug. Mea culpa.

the quick fix is this patch:


    diff --git a/run_build.pl b/run_build.pl
    index aeb8966..822b4de 100755
    --- a/run_build.pl
    +++ b/run_build.pl
    @@ -1008,7 +1008,8 @@ sub writelog
     
     sub check_make
     {
    -    my @out = run_log("$make -v");
    +   # don't use run_log here - it's too early in the process
    +    my @out = `$make -v 2>&1`;
         return undef unless ($? == 0 && grep {/GNU Make/} @out);
         return 'OK';
     }



cheers

andrew




--
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



--
Sent via buildfarm-members mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members

Reply | Threaded
Open this post in threaded view
|

Re: BuildFarm client release 4.19

Mikael Kjellström
In reply to this post by Tom Lane-2


On 2017-05-01 20:19, Tom Lane wrote:

>> if I manually create the directory loach.lastrun-logs the script then
>> continues normally.
>
> FWIW, I haven't seen that, but probably that's because the directory
> already exists in each branch's working directory.  I bet zapping the
> whole working directory for a branch would reproduce it.

Yes, zapping the whole working directory for each branch reproduces it
for me every time.

/Mikael


--
Sent via buildfarm-members mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members

Reply | Threaded
Open this post in threaded view
|

Re: BuildFarm client release 4.19

Mikael Kjellström
In reply to this post by Andrew Dunstan-8


On 2017-05-01 20:25, Andrew Dunstan wrote:

> OK, that's a bug. Mea culpa.
>
> the quick fix is this patch:
>
>
>     diff --git a/run_build.pl b/run_build.pl
>     index aeb8966..822b4de 100755
>     --- a/run_build.pl
>     +++ b/run_build.pl
>     @@ -1008,7 +1008,8 @@ sub writelog
>
>      sub check_make
>      {
>     -    my @out = run_log("$make -v");
>     +   # don't use run_log here - it's too early in the process
>     +    my @out = `$make -v 2>&1`;
>          return undef unless ($? == 0 && grep {/GNU Make/} @out);
>          return 'OK';
>      }

Nope, that didn't do it.

If I zap REL9_6_STABLE and then run:

./run_build.pl REL9_6_STABLE

I get the following output:

cannot create
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
No such file or directory
cannot create
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
No such file or directory
cannot create
/usr/home/pgbf/buildroot/REL9_6_STABLE/loach.lastrun-logs/lastcomand.log:
No such file or directory
Buildfarm member loach failed on REL9_6_STABLE stage pgsql-Git

/Mikael



--
Sent via buildfarm-members mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members

Reply | Threaded
Open this post in threaded view
|

Re: BuildFarm client release 4.19

Mikael Kjellström


On 2017-05-01 20:44, Mikael Kjellström wrote:
> Nope, that didn't do it.

Or well.  It fixed the check_make bug but not the other bug with that
the loach.lastrun-logs-directory isn't created before trying to write to it.

/Mikael


--
Sent via buildfarm-members mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members

Reply | Threaded
Open this post in threaded view
|

Re: BuildFarm client release 4.19

Andrew Dunstan-8


On 05/01/2017 02:46 PM, Mikael Kjellström wrote:
>
>
> On 2017-05-01 20:44, Mikael Kjellström wrote:
>> Nope, that didn't do it.
>
> Or well.  It fixed the check_make bug but not the other bug with that
> the loach.lastrun-logs-directory isn't created before trying to write
> to it.
>


OK, coming up with a more comprehensive fix.

The obvious workaround for now is to create the directory and dont zap
it or its parents. You should only have to do it once (per branch)

cheers

andrew

--
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



--
Sent via buildfarm-members mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members

Reply | Threaded
Open this post in threaded view
|

Re: [buildfarm-members] BuildFarm client release 4.19

Mikael Kjellström

On 2017-05-01 20:56, Andrew Dunstan wrote:
> OK, coming up with a more comprehensive fix.

Ok.

> The obvious workaround for now is to create the directory and dont zap
> it or its parents. You should only have to do it once (per branch)

Yes, I know.  That is what I have been doing so far.  But if you want to
rerun a branch from scratch it's much easier to just do:

rm -rf buildroot/HEAD
or
rm -rf buildroot/REL9_6_STABLE

and then the buildfarm script should do the right thing and create all
the directories that it needs.

/Mikael


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Reply | Threaded
Open this post in threaded view
|

Re: [buildfarm-members] BuildFarm client release 4.19

Andrew Dunstan-8


On 05/01/2017 03:00 PM, Mikael Kjellström wrote:

>
> On 2017-05-01 20:56, Andrew Dunstan wrote:
>> OK, coming up with a more comprehensive fix.
>
> Ok.
>
>> The obvious workaround for now is to create the directory and dont zap
>> it or its parents. You should only have to do it once (per branch)
>
> Yes, I know.  That is what I have been doing so far.  But if you want
> to rerun a branch from scratch it's much easier to just do:
>
> rm -rf buildroot/HEAD
> or
> rm -rf buildroot/REL9_6_STABLE
>
> and then the buildfarm script should do the right thing and create all
> the directories that it needs.
>


Not sure I understand what "rerun a branch" from scratch means. If you
zap the branch directory you lose all its state. That's generally a bad
thing.

Anyway, this patch should fix it for all uses. It creates the directory
if it doesn't exist.

    diff --git a/PGBuild/Utils.pm b/PGBuild/Utils.pm
    index 91c1362..175eaa7 100644
    --- a/PGBuild/Utils.pm
    +++ b/PGBuild/Utils.pm
    @@ -14,6 +14,8 @@ See accompanying License file for license details
     use strict;
     use warnings;
     
    +use File::Path;
    +
     use Exporter   ();
     our (@ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
     
    @@ -30,8 +32,9 @@ use vars qw($VERSION); $VERSION = 'REL_4.19';
     sub run_log
     {
         my $command = shift;
    -    my $file=
    -    
    "$main::branch_root/$main::st_prefix$main::logdirname/lastcomand.log";
    +   my $filedir =
    "$main::branch_root/$main::st_prefix$main::logdirname";
    +   mkpath($filedir);
    +   my $file= "$filedir/lastcomand.log";
         unlink $file;
         system("$command > $file 2>&1");
         my @loglines;


cheers

andrew


--
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Reply | Threaded
Open this post in threaded view
|

Re: [buildfarm-members] BuildFarm client release 4.19

Mikael Kjellström

On 2017-05-01 21:10, Andrew Dunstan wrote:
> Not sure I understand what "rerun a branch" from scratch means. If you
> zap the branch directory you lose all its state. That's generally a bad
> thing.

I mean like the first time when you set up the buildfarm client /
branch.  And it's not something I recomend doing all the time but some
times when the buildfarm client doesn't clean up after it self and there
are a lot of crap left it's "easier" to just zap everything and start
over to get it running again.


> Anyway, this patch should fix it for all uses. It creates the directory
> if it doesn't exist.
>
>     diff --git a/PGBuild/Utils.pm b/PGBuild/Utils.pm
>     index 91c1362..175eaa7 100644
>     --- a/PGBuild/Utils.pm
>     +++ b/PGBuild/Utils.pm
>     @@ -14,6 +14,8 @@ See accompanying License file for license details
>      use strict;
>      use warnings;
>
>     +use File::Path;
>     +
>      use Exporter   ();
>      our (@ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
>
>     @@ -30,8 +32,9 @@ use vars qw($VERSION); $VERSION = 'REL_4.19';
>      sub run_log
>      {
>          my $command = shift;
>     -    my $file=
>     -
>     "$main::branch_root/$main::st_prefix$main::logdirname/lastcomand.log";
>     +   my $filedir =
>     "$main::branch_root/$main::st_prefix$main::logdirname";
>     +   mkpath($filedir);
>     +   my $file= "$filedir/lastcomand.log";
>          unlink $file;
>          system("$command > $file 2>&1");
>          my @loglines;

Ok.

Thanks.  Will try it out.

/Mikael


--
Sent via pgsql-hackers mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Reply | Threaded
Open this post in threaded view
|

Re: BuildFarm client release 4.19

Mikael Kjellström


On 2017-05-01 21:19, Mikael Kjellström wrote:

> Thanks.  Will try it out.

Just wanted to report that I've tried it and it works as expected.

Thanks for the really fast fixes.

/Mikael


--
Sent via buildfarm-members mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members