== PostgreSQL Weekly News - September 1, 2019 ==
== PostgreSQL Product News ==
pg_dumpbinary 1.0, a program used to dump a PostgreSQL database in binary
Slony 2.2.8, a trigger-based logical replication system for PostgreSQL,
Ajqvue Version 3.0, a java-based UI which supports PostgreSQL, released.
pgbouncer 1.11.0, a connection pooler and more for PostgreSQL, released.
== PostgreSQL Jobs for September ==
== PostgreSQL Local ==
The first Austrian pgDay, will take place September 6, 2019 at the Hilton Garden
Inn in Wiener Neustadt.
PostgresOpen will be September 11th - 13th, 2019 in Orlando, Florida at the
Rosen Centre Hotel.
PostgresConf South Africa 2019 will take place in Johannesburg on October 8-9, 2019
PostgreSQL Conference Europe 2019 will be held on October 15-18, 2019 in Milan,
2Q PGConf 2019 will be held December 4 & 5 in Chicago.
The CFP is open through August 30, 2019.
pgDay Paris 2020 will be held in Paris, France on March 26, 2020
at Espace Saint-Martin.
Nordic PGDay 2020 will be held in Helsinki, Finland at the Hilton Helsinki
Strand Hotel on March 24, 2020. The CfP is open through December 31, 2019 at
PGConf India 2020 will be on February 26-28, 2020 in Bengaluru, Karnataka.
== PostgreSQL in the News ==
Planet PostgreSQL: http://planet.postgresql.org/
PostgreSQL Weekly News is brought to you this week by David Fetter
Submit news and announcements by Sunday at 3:00pm PST8PDT to [hidden email].
== Applied Patches ==
Michaël Paquier pushed:
- Fix error handling of vacuumdb and reindexdb when running out of fds. When
trying to use a high number of jobs, vacuumdb (and more recently reindexdb)
has only checked for a maximum number of jobs used, causing confusing failures
when running out of file descriptors when the jobs open connections to
Postgres. This commit changes the error handling so as we do not check
anymore for a maximum number of allowed jobs when parsing the option value
with FD_SETSIZE, but check instead if a file descriptor is within the
supported range when opening the connections for the jobs so as this is
detected at the earliest time possible. Also, improve the error message to
give a hint about the number of jobs recommended, using a wording given by the
reviewers of the patch. Reported-by: Andres Freund Author: Michael Paquier
Reviewed-by: Andres Freund, Álvaro Herrera, Tom Lane Discussion:
- Fix failure of --jobs with reindexdb and vacuumdb on Windows. FD_SETSIZE needs
to be declared before winsock2.h, or it is possible to run into buffer
overflow issues when using --jobs. This is similar to pgbench's solution done
in a23c641. This has been introduced by 71d84ef, and older versions have been
using the default value of FD_SETSIZE, defined at 64. Per buildfarm member
jacana, but this impacts all Windows animals running the TAP tests. I have
reproduced the failure locally to check the patch. Author: Michael Paquier
Reviewed-by: Andrew Dunstan Discussion:
https://postgr.es/m/20190826054000.GE7005@... Backpatch-through: 9.5
- Disable timeouts when running pg_rewind with online source cluster. In this
case, the transfer uses a libpq connection, which is subject to the timeout
parameters set at system level, and this can make the rewind operation
suddenly canceled which is not good for automation. One workaround to such
issues would be to use PGOPTIONS to enforce the wanted timeout parameters, but
that's annoying, and for example pg_dump, which can run potentially
long-running queries disables all types of timeouts. lock_timeout and
statement_timeout are the ones which can cause problems now. Note that
pg_rewind does not use transactions, so disabling
idle_in_transaction_session_timeout is optional, but it feels safer to do so
for the future. This is back-patched down to 9.5.
idle_in_transaction_session_timeout is only present since 9.6. Author:
Alexander Kukushkin Discussion:
- Improve coverage of utils/float.h. check_float4_val() checks after underflow
and overflow of values converted from float8 to float4, but there has never
been any regression tests for that. This brings the coverage of float.h to
100%. Author: Movead Li Discussion:
Andrew Dunstan pushed:
- Treat MINGW and MSYS the same in pg_upgrade test script. On msys2, 'uname -s'
reports a string starting MSYS instead on MINGW as happens on msys1. Treat
these both the same way. This reverts 608a710195a4b in favor of a more general
solution. Backpatch to all live branches.
- Adjust to latest Msys2 kernel release number. Previously 'uname -r' on Msys2
reported a kernele release starting with 2. The latest version starts with 3.
In commit 1638623f we specifically looked for one starting with 2. This is now
changed to look for any digit between 2 and 9. backpatch to release 10.
Peter Eisentraut pushed:
- Fix gettext triggers specification. In
cc8d41511721d25d557fc02a46c053c0a602fed0, the arguments of
warn_or_exit_horribly() were changed but this was not updated.
- Error out on too many command-line arguments. Fix up oid2name, pg_upgrade, and
pgbench to error out on too many command-line arguments. This makes it match
the behavior of other PostgreSQL programs. Author: Peter Eisentraut, Ibrar
Tom Lane pushed:
- Make comment in fmgr.h match the one in fmgr.c. Incompletely quoting an API
spec does nobody any good. Noted by Paul Jungwirth. Looks like the
discrepancy was my fault originally :-( Discussion:
- Fix postmaster state machine to handle dead_end child crashes better. A report
from Alvaro Herrera shows that if we're in PM_STARTUP state, and we spawn a
dead_end child to reject some incoming connection request, and that child dies
with an unexpected exit code, the postmaster does not respond well. We
correctly send SIGQUIT to the startup process, but then: * if the startup
process exits with nonzero exit code, as expected, we thought that that
indicated a crash and aborted startup. * if the startup process exits with
zero exit code, which is possible due to the inherent race condition, we'd
advance to PM_RUN state which is fine --- but the code forgot that
AbortStartTime would be nonzero in this situation. We'd either die on the
Asserts saying that it was zero, or perhaps misbehave later on. (A quick look
suggests that the only misbehavior might be busy-waiting due to
DetermineSleepTime doing the wrong thing.) To fix the first point, adjust the
state-machine logic to recognize that a nonzero exit code is expected after
sending SIGQUIT, and have it transition to a state where we can restart the
startup process. To fix the second point, change the Asserts to clear the
variable rather than just claiming it should be clear already. Perhaps we
could improve this further by not treating a crash of a dead_end child as a
reason for panic'ing the database. However, since those child processes are
connected to shared memory, that seems a bit risky. There are few good
reasons for a dead_end child to report failure anyway (the cause of this in
Alvaro's report is quite unclear). On balance, therefore, a minimal fix seems
best. This is an oversight in commit 45811be94. While that was back-patched,
I'm hesitant to back-patch this change. The lack of reasons for a dead_end
child to fail suggests that the case should be very rare in the field, which
squares with the lack of reports; so it seems like this might not be worth the
risk of introducing new issues. In any case we can let it bake awhile in HEAD
before considering a back-patch. Discussion:
- Fix 007_sync_rep.pl to notice failures in ALTER SYSTEM SET. If a test case
tried to set an invalid value of synchronous_standby_names, the test script
didn't detect that, which seems like a bad idea. Noticed while testing a
proposed patch that broke some of these test cases.
- Reject empty names and recursion in config-file include directives. An empty
file name or subdirectory name leads join_path_components() to just produce
the parent directory name, which leads to weird failures or recursive
inclusions. Let's throw a specific error for that. It takes only slightly
more code to detect all-blank names, so do so. Also, detect direct recursion,
ie a file calling itself. As coded this will also detect recursion via
"include_dir '.'", which is perhaps more likely than explicitly including the
file itself. Detecting indirect recursion would require API changes for
guc-file.l functions, which seems not worth it since extensions might call
them. The nesting depth limit will catch such cases eventually, just not with
such an on-point error message. In passing, adjust the example usages in
postgresql.conf.sample to perhaps eliminate the problem at the source: there's
no reason for the examples to suggest that an empty value is valid. Per a
trouble report from Brent Bates. Back-patch to 9.5; the issue is old, but the
code in 9.4 is enough different that the patch doesn't apply easily, and it
doesn't seem worth the trouble to fix there. Ian Barwick and Tom Lane
- Add missing newline in help output. Daniel Gustafsson Discussion:
- Doc: clarify behavior of standard aggregates for null inputs. Section 4.2.7
says that unless otherwise specified, built-in aggregates ignore rows in which
any input is null. This is not true of the JSON aggregates, but it wasn't
documented. Fix that. Of the other entries in table 9.55, some were explicit
about ignoring nulls, and some weren't; for consistency and
self-contained-ness, make them all say it explicitly. Per bug #15884 from Tim
Möhlmann. Back-patch to all supported branches. Discussion:
- Improve what pg_strsignal prints if we haven't got strsignal(3). Turns out
that returning "unrecognized signal" is confusing. Make it explicit that the
platform lacks any support for signal names. (At least of the machines in the
buildfarm, only HPUX lacks it.) Back-patch to v12 where we invented this
function. Discussion: https://postgr.es/m/3067.1566870481@...
- Doc: improve documentation of pg_signal_backend default role. Give it an
explanatory para like the other default roles have. Don't imply that it can
send any signal whatever. In passing, reorder the table entries and
explanatory paras for the default roles into some semblance of consistency.
Ian Barwick, tweaked a bit by me. Discussion:
- Set application_name per-test in isolation and ecpg tests. Commit a4327296d
taught pg_regress proper to do this, but missed the opportunity to do likewise
in the isolationtester and ecpg variants of pg_regress. Seems like this might
be helpful for tracking down issues exposed by those tests.
- Add .gitignore file forgotten in commit bde7493d1.
- Doc: remove some long-obsolete information from installation.sgml. Section
16.2 pointed to platform-specific FAQ files that we removed way back in 8.4.
Section 16.7 contained a bunch of information about AIX and HPUX bugs that
were squashed decades ago, plus discussions of old compiler versions that are
certainly moot now that we require C99 support. Since we're obviously not
maintaining this stuff carefully, just remove it. The HPUX sub-section seems
like it can go away entirely, since everything it said that was still
applicable was redundant with material elsewhere in the chapter. In passing,
I couldn't resist the temptation to do a small amount of copy-editing on
nearby text. Back-patch to v12, since this stuff is surely obsolete in any
branch that requires C99. Discussion:
- Doc: restructure documentation of the configure script's options. The list of
configure options has grown long, and there was next to no organization to it,
never mind any indication of which options were interesting to most people.
Break it into several sub-sections to provide a bit of structure, and add some
introductory text where it seems helpful to point people to particular
options. I failed to resist the temptation to do a small amount of
word-smithing on some of the option descriptions, too. But mostly this is
reorganization and addition of intro text. Discussion:
- Cosmetic improvements for options-handling code in ECPGconnect(). The comment
describing the string format was a lie. Make it agree with reality,
add/improve some other comments, fix coding style for loops with empty bodies.
Also add an Assert that we counted parameters correctly, because the
spread-out logic for that looks pretty fragile. No actual bugs fixed here, so
no need to back-patch. Discussion:
- Doc: describe the "options" allowed in an ECPG connection target string. These
have been there a long time, but their format was never explained in the docs.
Per complaint from Yusuke Egashira. Discussion:
Peter Geoghegan pushed:
- Remove obsolete nbtree page deletion comment. Commit efada2b8e92, which made
the nbtree page deletion algorithm more robust, removed the concept of a
half-dead internal page. Remove a comment about half dead parent pages that
Thomas Munro pushed:
- Avoid catalog lookups in RelationAllowsEarlyPruning().
RelationAllowsEarlyPruning() performed a catalog scan, but is used in two
contexts where that was a bad idea: 1. In heap_page_prune_opt(), which runs
very frequently in some large scans. This caused major performance
problems in a field report that was easy to reproduce. 2. In
TestForOldSnapshot(), which runs while we hold a buffer content lock.
It's not clear if this was guaranteed to be free of buffer deadlock risk.
The check was introduced in commit 2cc41acd8 and defended against a real
problem: 9.6's hash indexes have no page LSN and so we can't allow early
pruning (ie the snapshot-too-old feature). We can remove the check from all
later releases though: hash indexes are now logged, and there is no way to
create UNLOGGED indexes on regular logged tables. If a future release allows
such a combination, it might need to put a similar check in place, but it'll
need some more thought. Back-patch to 10. Author: Thomas Munro Reviewed-by:
Tom Lane, who spotted the second problem Discussion:
Heikki Linnakangas pushed:
- Fix overflow check and comment in GIN posting list encoding. The comment did
not match what the code actually did for integers with the 43rd bit set. You
get an integer like that, if you have a posting list with two adjacent TIDs
that are more than 2^31 blocks apart. According to the comment, we would store
that in 6 bytes, with no continuation bit on the 6th byte, but in reality, the
code encodes it using 7 bytes, with a continuation bit on the 6th byte as
normal. The decoding routine also handled these 7-byte integers correctly,
except for an overflow check that assumed that one integer needs at most 6
bytes. Fix the overflow check, and fix the comment to match what the code
actually does. Also fix the comment that claimed that there are 17 unused bits
in the 64-bit representation of an item pointer. In reality, there are
64-32-11=21. Fitting any item pointer into max 6 bytes was an important
property when this was written, because in the old pre-9.4 format, item
pointers were stored as plain arrays, with 6 bytes for every item pointer. The
maximum of 6 bytes per integer in the new format guaranteed that we could
convert any page from the old format to the new format after upgrade, so that
the new format was never larger than the old format. But we hardly need to
worry about that anymore, and running into that problem during upgrade, where
an item pointer is expanded from 6 to 7 bytes such that the data doesn't fit
on a page anymore, is implausible in practice anyway. Backpatch to all
supported versions. This also includes a little test module to test these
large distances between item pointers, without requiring a 16 TB table. It is
not backpatched, I'm including it more for the benefit of future development
of new posting list formats. Discussion:
Reviewed-by: Masahiko Sawada, Alexander Korotkov
Etsuro Fujita pushed:
- Fix typos in regression test comments.
== Pending Patches ==
Sergei Kornilov sent in another revision of a patch to change the ereport level
Gareth Palmer sent in another revision of a patch to implement INSERT ... SET.
Magnus Hagander sent in another revision of a patch to implement online
checksums, this time with global barriers.
Peter Eisentraut sent in another revision of a patch to prevent getting errors
of the form "Could not read from file ...: Success."
Anastasia Lubennikova sent in a patch to improve the BTScanPos comments.
Fabien COELHO sent in another revision of a patch to enable pgbench to create
Movead Li sent in two more revisions of a patch to improve the test coverage.
Andrey Borodin sent in two revisions of a patch to build GiST indexes faster
using z-order sorting.
Alexander Korotkov sent in another revision of a patch to add support for the
jsonpath .datetime() method.
Asim R P sent in another revision of a patch to implement a fault injection
Kyotaro HORIGUCHI sent in two more revisions of a patch to fix the WAL-skipping
Dmitry Dolgov sent in a patch to add copy freeze and maintenance_work_mem
options to pgbench.
Thomas Munro sent in a patch to stop performing catalog lookups in
Euler Taveira de Oliveira and Aleksey Kondratov traded patches to make it
possible to do row filtering for logical replication.
Sergei Kornilov sent in another revision of a patch to make it possible to
change primary_conninfo online.
Etsuro Fujita sent in another revision of a patch to improve the
partition-matching algorithm for partition-wise JOINs.
Heikki Linnakangas sent in a patch to fix a bug in BRIN inclusion
Fabien COELHO sent in a patch to increase the psql test coverage from 41% to
Dmitry Dolgov sent in another revision of a patch to implement index skip scans.
Sergei Kornilov sent in a patch to implement pg_get_databasebyid(oid).
Joe Conway sent in a patch to add support for seccomp-bpf.
fn_ln sent in three more revisions of a patch to fix some inconsistent behavior
in chained transactions.
Michaël Paquier and Tom Lane traded patches to improve error detection in TAP
Taylor Vesely sent in another revision of a patch to implement memory-bounded
Jeevan Chalke sent in two revisions of a patch to fix a bug where basebackup.c's
sendFile() ignores read errors.
Aleksey Kondratov sent in another revision of a patch to ensure that the
BGWorkers pool for streamed transactions applies without spilling to disk.
Fabien COELHO and Michaël Paquier traded patches to recombine the str2*int64
functions to be in one place.
Masahiko Sawada sent in another revision of a patch to add a RESUME option to
VACUUM and autovacuum.
Moon Insung sent in a patch to fix a bug in GIN metapages.
Richard Guo sent in another revision of a patch to create any EC-derived
joinclauses of form 'outer_em = inner_em'.
Andres Freund sent in a WIP patch to implement generic functions for Node types
using generated metadata.
Anastasia Lubennikova sent in another revision of a patch to compress storage of
duplicates in B-tree indexes.
Ian Barwick sent in a patch to update the documents for the PL/pgSQL sample
Thomas Munro sent in a patch to consolidate code that makes a sorted array
Ian Barwick sent in a patch to psql which adds tab completion for \df slash
Masahiko Sawada sent in another revision of a patch to add a CLUSTER command
Nikhil Sontakke sent in another revision of a patch to add batch and pipelining
support to libpq.
Fabien COELHO sent in a patch to add a TAP test for the bug fix to pg_upgrade
which ensured that it errors out on too many command-line arguments.
Peter Eisentraut sent in another revision of a patch to implement the base
backup client as an auxiliary backend process.
Rajkumar Raghuwanshi sent in a patch to add some TAP tests for pg_combinebackup.
Aleksey Kondratov sent in another revision of a patch to enable CLUSTER, VACUUM
FULL and REINDEX to change tablespaces on the fly.
Noah Misch sent in a patch to add some tests for AIX's atomic fetches.
Thomas Munro sent in a patch to add an SQL type xid8 to expose FullTransactionId
to users, and introduce xid8 variants of the txid_XXX() fmgr functions.
Aleksey Kondratov sent in a patch to write the visibility map during
Pierre Ducroquet sent in a patch to add a reset_computed_values function to
|Free forum by Nabble||Edit this page|