docker instances of pgbuildfarm

I've dumped here the very basics on how I'm running the critters in docker.

https://github.com/linux-on-ibm-power/docker_pgbuildfarm

Might get a chance to write more doco next week.

Daniel,


Your config is seriously awry on pretty much all your animals. I see this:


                    'invocation_args' => [
                                           '--verbose=2',
                                           '--keepall',
                                           '--from-source=/source',
                                           'HEAD'
                                         ],


from-source should never be used for a reporting animal. Quite apart
from any other reason, it inhibits git updates. It's designed for
offline use. I use it sometimes to check patches by running them through
a very complete set of tests. I'm mildly surprised that it actually
reports at all with this option, and I will stop it from doing so in the
next release.


Please fix this ASAP.


You also probably don't want --keepall either, nor --verbose=2, but
those are less serious.


cheers


andrew

On 09/09/2016 09:54 AM, Andrew Dunstan wrote:
>
> I'm mildly surprised that it actually reports at all with this option,
> and I will stop it from doing so in the next release.


Actually, the script has this already. $nosend is set if $from_source is
used. Have you hacked the run script? If so, please don't.

cheers

andrew

Andrew,

Relax, the git updates are done outside of the pgbuildfarm scripts.
Below is the start of the log that triggered the build this morning.

The keepall has no effect as anything done in the run stage is removed.
I'll reexamine verbose, I think that was debugging only.

Perhaps I can get $scm->log_id() into the build to prove this a bit more.

As a server side thought, maybe even check this value server side to
ensure that results aren't getting submitting when/if people's mirror
scripts break for some reason.


07:59:28 Started by upstream project "Pgbuildfarm/REL9___6___STABLE"
build number 14
07:59:28 originally caused by:
07:59:28  Branch indexing
07:59:28  Branch indexing
07:59:28  Branch indexing
07:59:28  Branch indexing
07:59:28  Branch indexing
07:59:29 [lockable-resources] acquired lock on [ka4-perf-1]
07:59:29 [EnvInject] - Loading node environment variables.
07:59:29 Building remotely on p84-vm (ubuntu ppc64le_docker_build power
ppc64le_docker docker ppc64le) in workspace
/var/lib/jenkins/workspace/Pgbuildfarm/REL9___6___STABLE/CC/gcc/CONFIG/ppc64le/HOST_OS/debian-8.4
07:59:29 [WS-CLEANUP] Deleting project workspace...
07:59:29 [WS-CLEANUP] Done
07:59:29 Using checkout strategy: Default
07:59:29 Last Built Revision: Revision
f6802936a53671c9db24273ab2ae94291a68afcd (origin/REL9_6_STABLE)
07:59:29 Cloning the remote Git repository
07:59:30 Cloning repository
git://gitlab.ozlabs.ibm.com/mirror/postgresql.git
07:59:30  > git init
/var/lib/jenkins/workspace/Pgbuildfarm/REL9___6___STABLE/CC/gcc/CONFIG/ppc64le/HOST_OS/debian-8.4
# timeout=30
07:59:30 ERROR: Reference path does not exist:
/var/lib/jenkins/git/postgresql.git
07:59:30 Fetching upstream changes from
git://gitlab.ozlabs.ibm.com/mirror/postgresql.git
07:59:30  > git --version # timeout=30
07:59:30  > git -c core.askpass=true fetch --tags --progress
git://gitlab.ozlabs.ibm.com/mirror/postgresql.git
+refs/heads/*:refs/remotes/origin/*
08:00:22  > git config remote.origin.url
git://gitlab.ozlabs.ibm.com/mirror/postgresql.git # timeout=30
08:00:22  > git config --add remote.origin.fetch
+refs/heads/*:refs/remotes/origin/* # timeout=30
08:00:22  > git config remote.origin.url
git://gitlab.ozlabs.ibm.com/mirror/postgresql.git # timeout=30
08:00:22 Fetching upstream changes from
git://gitlab.ozlabs.ibm.com/mirror/postgresql.git
08:00:22  > git -c core.askpass=true fetch --tags --progress
git://gitlab.ozlabs.ibm.com/mirror/postgresql.git
+refs/heads/*:refs/remotes/origin/*
08:00:22  > git config remote.origin1.url
git://gitlab.ozlabs.ibm.com/mirror/postgresql.git # timeout=30
08:00:22 Fetching upstream changes from
git://gitlab.ozlabs.ibm.com/mirror/postgresql.git
08:00:22  > git -c core.askpass=true fetch --tags --progress
git://gitlab.ozlabs.ibm.com/mirror/postgresql.git
+refs/remotes/*:refs/remotes/*
08:00:22 Checking out Revision fa7b3a88dd5d7f6d18614eefc6cc0662e31bd451
(origin/REL9_6_STABLE)
08:00:22  > git config core.sparsecheckout # timeout=30
08:00:22  > git checkout -f fa7b3a88dd5d7f6d18614eefc6cc0662e31bd451
08:00:23  > git rev-list f6802936a53671c9db24273ab2ae94291a68afcd #
timeout=30
08:00:24 Run condition [Always] enabling prebuild for step [Archive the
artifacts]
08:00:24 Set build name.
08:00:24 New build name is
'#14_origin/REL9_6_STABLE_fa7b3a88dd5d7f_gcc_ppc64le'
08:00:24 [debian-8.4] $ /bin/bash -xe /tmp/hudson1484895481194831141.sh
08:00:24 + mkdir .source
08:00:24 + mv aclocal.m4 config configure configure.in contrib COPYRIGHT
doc GNUmakefile.in HISTORY Makefile README README.git src .git .source
08:00:24 + mv .source source
08:00:24 + mkdir artifacts
08:00:24 + export https_proxy=http://proxy:3142/
http_proxy=http://proxy:3142/ ftp_proxy=http://proxy:3128/
08:00:24 + https_proxy=http://proxy:3142/
08:00:24 + http_proxy=http://proxy:3142/
08:00:24 + ftp_proxy=http://proxy:3128/
08:00:24 + git clone --branch secret --recursive
https://gitlab.ozlabs.ibm.com/CI/pgbuildfarm-script.git scripts
08:00:24 Cloning into 'scripts'...
08:00:24 + cd scripts
08:00:24 + ./docker_run_build.sh
08:00:24 + set -euo pipefail
08:00:24 + DOC=jenkins-pgbuildfarm-debian-8.4-gcc
08:00:24 + mkdir -m 0777 -p
/var/lib/jenkins/workspace/Pgbuildfarm/REL9___6___STABLE/CC/gcc/CONFIG/ppc64le/HOST_OS/debian-8.4/artifacts
/var/lib/jenkins/workspace/Pgbuildfarm/REL9___6___STABLE/CC/gcc/CONFIG/ppc64le/HOST_OS/debian-8.4/ccache
08:00:24 ++ id -u
08:00:24 + docker build --build-arg=BUILD_UID=1000 --build-arg=CC=gcc -t
jenkins-pgbuildfarm-debian-8.4-gcc --file Docker.debian-8.4 .
08:00:25 Sending build context to Docker daemon 150.5 kB

08:00:25 Step 1 : FROM debian:8.4
08:00:25  ---> e0a06a89ed81
08:00:25 Step 2 : ARG BUILD_UID
08:00:25  ---> Using cache
08:00:25  ---> 1e5c3052ee81
08:00:25 Step 3 : ARG CC
08:00:25  ---> Using cache
08:00:25  ---> 112d62a74f5a
08:00:25 Step 4 : ENV DEBIAN_FRONTEND noninteractive
08:00:25  ---> Using cache
08:00:25  ---> 1916fcad9ff1
08:00:25 Step 5 : ENV http_proxy http://proxy:3142/
08:00:25  ---> Using cache
08:00:25  ---> 424d4ef4f6ae
08:00:25 Step 6 : ENV https_proxy http://proxy:3142/
08:00:25  ---> Using cache
08:00:25  ---> 1086c2a06ab5
08:00:25 Step 7 : ENV ftp_proxy http://proxy:3128/
08:00:25  ---> Using cache
08:00:25  ---> 8aa0d6c5635d
08:00:25 Step 8 : VOLUME /ccache /artifacts /source
08:00:25  ---> Using cache
08:00:25  ---> 623f4bdc87c3
08:00:25 Step 9 : RUN for p in http https ftp; do echo
Acquire::${p}::proxy '"http://proxy:3128";' ; done >>
/etc/apt/apt.conf.d/proxy
08:00:25  ---> Using cache
08:00:25  ---> d2d59ab6f116
08:00:25 Step 10 : RUN sed -e 's/^deb/deb-src/g' /etc/apt/sources.list >
/etc/apt/sources.list.d/src.list &&     apt-get update && apt-get
upgrade -y &&      apt-get build-dep -y postgresql &&     apt-get
install -y curl software-properties-common ccache sudo
libwww-perl libhttp-message-perl libdigest-sha-perl libbsd-resource-perl
             python-dev libreadline-dev zlib1g-dev libgss-dev libssl-dev
libxml2-dev libxslt1-dev              libldap2-dev libkrb5-dev
tcl8.6-dev tcl-dev libperl-dev bison flex
08:00:25  ---> Using cache
08:00:25  ---> becbb0b8e4f1
08:00:25 Step 11 : RUN curl  --proxy proxy:3128
http://www.pgbuildfarm.org/downloads/latest-client.tgz | tar -zxf - &&
ln -s build-farm-* build-farm
08:00:25  ---> Using cache
08:00:25  ---> f36f872dcc21
08:00:25 Step 12 : RUN /bin/bash -c "[ \"${CC}\" == 'clang' ] && apt-get
install -y clang || true"
08:00:25  ---> Using cache
08:00:25  ---> 19d88445b121
08:00:25 Step 13 : RUN /bin/bash -c "set -v; if [[ `uname -m` ==
\"ppc64le\" &&  \"${CC}\" =~ ^at ]]; then                       curl
--proxy proxy:3128
ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/debian/dists/jessie/6976a827.gpg.key
| apt-key add -;                       [[ \"${CC}\" =~
^at[[:digit:]]+\.[[:digit:]]+$ ]] &&                          echo 'deb
ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/debian jessie '${CC}
>> /etc/apt/sources.list ||                          echo 'deb
http://ausgsa.ibm.com/projects/p/ppctoolchain/web/public/at/debian
jessie '${CC%-*} >> /etc/apt/sources.list;                       apt-get
update && apt-get install -y advance-toolchain-${CC}-devel;
     fi"
08:00:25  ---> Using cache
08:00:25  ---> a9c673d7d4d4
08:00:25 Step 14 : RUN /bin/bash -c "set -v ; if [[ \"${CC}\" =~ 'gcc-'
]]; then                     echo 'deb
http://httpredir.debian.org/debian unstable main' >
/etc/apt/sources.list.d/unstable.list &&                     apt-get
update && apt-get install -y ${CC} ;                   fi"
08:00:25  ---> Using cache
08:00:25  ---> 308e80392e08
08:00:25 Step 15 : RUN /bin/bash -c "if [[ \"${CC}\" =~ clang- ]]; then
                   if [[ \"${CC}\" =~ clang-4 ]]; then
    echo \"deb http://apt.llvm.org/jessie/ llvm-toolchain-jessie main\"
>> /etc/apt/sources.list;                     else
echo \"deb http://apt.llvm.org/jessie/
llvm-toolchain-jessie-${CC#clang-} main\" >> /etc/apt/sources.list;
               fi ;                     curl --proxy proxy:3128
http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - && apt-get
update &&                     apt-get install -y \"${CC}\" || exit 1 ;
                fi"
08:00:25  ---> Using cache
08:00:25  ---> bbb0d5ca6c81
08:00:25 Step 16 : RUN useradd -m -u $BUILD_UID --home-dir /build build
&& mkdir /build/postgresql_build && chown build: /build/postgresql_build
08:00:25  ---> Using cache
08:00:25  ---> abbfe7451e48
08:00:25 Step 17 : COPY build_pg.sh /build/build_pg.sh
08:00:25  ---> Using cache
08:00:25  ---> b2af0217bbc0
08:00:25 Step 18 : COPY build-farm.conf build-farm/build-farm.conf
08:00:25  ---> Using cache
08:00:25  ---> cd701f1536c2
08:00:25 Step 19 : COPY secrets.conf build-farm/secrets.conf
08:00:25  ---> Using cache
08:00:25  ---> b705bec5e6c8
08:00:25 Step 20 : RUN sed -i -e '/nosend=1/d' -e  '/nostatus=1/d'
/build-farm/run_build.pl
08:00:25  ---> Using cache
08:00:25  ---> f1f5643e2fc1
08:00:25 Step 21 : WORKDIR /build
08:00:25  ---> Using cache
08:00:25  ---> d62538c8ef65
08:00:25 Step 22 : CMD /usr/bin/sudo -E -H -u build CC=${CC}
GIT_BRANCH=${GIT_BRANCH} ./build_pg.sh
08:00:25  ---> Using cache
08:00:25  ---> 3c45d6937cd9
08:00:25 Successfully built 3c45d6937cd9
08:00:25 + DOCKER_OPT=
08:00:25 + docker run --rm=true --cpu-shares=2 --shm-size=32G --name
jenkins-pgbuildfarm-debian-8.4-gcc_REL9_6_STABLE --volume
/var/lib/jenkins/workspace/Pgbuildfarm/REL9___6___STABLE/CC/gcc/CONFIG/ppc64le/HOST_OS/debian-8.4/source:/source
--volume
/var/lib/jenkins/workspace/Pgbuildfarm/REL9___6___STABLE/CC/gcc/CONFIG/ppc64le/HOST_OS/debian-8.4/ccache:/ccache
--volume
/var/lib/jenkins/workspace/Pgbuildfarm/REL9___6___STABLE/CC/gcc/CONFIG/ppc64le/HOST_OS/debian-8.4/artifacts:/artifacts
--env CC=gcc --env HOST_OS=debian-8.4 --env
GIT_BRANCH=origin/REL9_6_STABLE -t jenkins-pgbuildfarm-debian-8.4-gcc
08:00:25 + set -v
08:00:25
08:00:25 # keep caches in different directories
08:00:25 CCACHE=$CC
08:00:25 + CCACHE=gcc
08:00:25
08:00:25 # for secret lookup
08:00:25 COMPILER=${CC}
08:00:25 + COMPILER=gcc
08:00:25
08:00:25 case $CC in
08:00:25 at*)
08:00:25   P=$(echo /opt/${CC/-/-*}/bin)
08:00:25   CC=${P}/gcc
08:00:25   AR=${P}/ar
08:00:25   PATH=${P}:${PATH}
08:00:25   ;;
08:00:25 xlc*)
08:00:25   CC=/usr/bin/xlc_r
08:00:25   ;;
08:00:25 *)
08:00:25   which ${CC} || CC=${CC%-*}
08:00:25   ;;
08:00:25 esac
08:00:25 + case $CC in
08:00:25 + which gcc
08:00:25 /usr/bin/gcc
08:00:25
08:00:25 export PERL5LIB=/build-farm
08:00:25 + export PERL5LIB=/build-farm
08:00:25 + PERL5LIB=/build-farm
08:00:25 export CC AR PATH CCACHE COMPILER
08:00:25 + export CC AR PATH CCACHE COMPILER
08:00:25
08:00:25 # Debug
08:00:25 # if ! /build-farm/run_build.pl --nosend --nostatus --verbose=2
--keepall --from-source=/source ${GIT_BRANCH#*/};
08:00:25
08:00:25 if ! /build-farm/run_build.pl --verbose=2 --keepall
--from-source=/source ${GIT_BRANCH#*/};
08:00:25 then
08:00:25   cp ~/postgresql_build/*/*.fromsource-logs/* /artifacts/
08:00:25   exit 1
08:00:25 fi
08:00:25 + /build-farm/run_build.pl --verbose=2 --keepall
--from-source=/source REL9_6_STABLE
08:00:25 For a ppc64le, on debian-8.4 using gcc I am a bleak
08:00:25 Sun Sep 11 22:00:25 2016: buildfarm run for bleak:REL9_6_STABLE
starting
08:00:25 [22:00:25] running configure ...
08:00:55 ======== configure output ===========
08:00:55 checking build system type... powerpc64le-unknown-linux-gnu
08:00:55 checking host system type... powerpc64le-unknown-linux-gnu
08:00:55 checking which template to use... linux
08:00:55 checking whether to build with 64-bit integer date/time
support... yes


On 09/09/16 23:54, Andrew Dunstan wrote:

Daniel Black <[hidden email]> writes:
> Relax, the git updates are done outside of the pgbuildfarm scripts.
> Below is the start of the log that triggered the build this morning.

The problem with inventing your own approach here is that your buildfarm
members' reports do not provide any evidence of what they are building
against.  Please examine

http://buildfarm.postgresql.org/cgi-bin/show_status.pl

and note that your animals are not showing last-commit-id in their lines
on the summary page, and if you drill down to their build reports, those
don't show anything about "changes since last run" either, eg compare

http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=demoiselle&dt=2016-09-12%2000%3A36%3A03
http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=guaibasaurus&dt=2016-09-12%2000%3A17%3A01

That greatly reduces the usefulness of the reports.

                        regards, tom lane

On 09/11/2016 08:49 PM, Tom Lane wrote:

I also don't see any point in doing this.

Please run this software as intended.

(BTW, telling someone to relax is quite offensively patronizing.)


cheers

andre

On 12/09/16 11:08, Andrew Dunstan wrote:
so I see.

> I also don't see any point in doing this.

Mainly fitting in with current infrastructure.

> Please run this software as intended.

Will look into it.

> (BTW, telling someone to relax is quite offensively patronizing.)

Wasn't intended. Sorry.

On 09/11/2016 09:25 PM, Daniel Black wrote:
>
>
>> Please run this software as intended.
> Will look into it.
>
>


These machines have not reported for a week. I am going to place them in
suspended state pending assurances that they have been fixed so that the
client runs as designed.

Except in extraordinary cases, there should be no need to change
anything except the client's configuration file.

cheers

andrew

On 19/09/16 21:47, Andrew Dunstan wrote:
Apologies - really busy + was ill for a few days.

> I am going to place them in
> suspended state pending assurances that they have been fixed so that the
> client runs as designed.

Almost there.

> Except in extraordinary cases, there should be no need to change
> anything except the client's configuration file.

The builds don't always run on the same hardware however the
architecture and build environment is identical.

Daniel Black wrote:

Any news on this?

--
Álvaro Herrera                PostgreSQL Expert, https://www.2ndQuadrant.com/


--
Sent via buildfarm-members mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members