PostgreSQL@SCaLE is a two day, two track event which takes place on
March 8-9, 2018, at Pasadena Convention Center, as part of SCaLE 16X.
The CfP is open through October 31, 2017 at
Nordic PGDay 2018 will be held in Oslo, Norway, at the Radisson Blu Hotel
Nydalen, on March 13, 2018. The CfP is open through December 31, 2017 at
- Pass InitPlan values to workers via Gather (Merge). If a PARAM_EXEC parameter
is used below a Gather (Merge) but the InitPlan that computes it is attached
to or above the Gather (Merge), force the value to be computed before starting
parallelism and pass it down to all workers. This allows us to use
parallelism in cases where it previously would have had to be rejected as
unsafe. We do - in this case - lose the optimization that the value is only
computed if it's actually used. An alternative strategy would be to have the
first worker that needs the value compute it, but one downside of that
approach is that we'd then need to select a parallel-safe path to compute the
parameter value; it couldn't for example contain a Gather (Merge) node. At
some point in the future, we might want to consider both approaches.
Independent of that consideration, there is a great deal more work that could
be done to make more kinds of PARAM_EXEC parameters parallel-safe. This
infrastructure could be used to allow a Gather (Merge) on the inner side of a
nested loop (although that's not a very appealing plan) and cases where the
InitPlan is attached below the Gather (Merge) could be addressed as well using
various techniques. But this is a good start. Amit Kapila, reviewed and
revised by me. Reviewing and testing from Kuntal Ghosh, Haribabu Kommi, and
Tushar Ahuja. Discussion:
- Allow running just selected steps of pgbench's initialization sequence. This
feature caters to specialized use-cases such as running the normal pgbench
scenario with nonstandard indexes, or inserting other actions between steps of
the initialization sequence. The normal sequence of initialization actions is
broken down into half a dozen steps which can be executed in a user-specified
order, to the extent to which that's sensible. The actions themselves aren't
changed, except to make them more robust against nonstandard uses: * all four
tables are now dropped in one DROP command, to reduce assumptions about what
foreign key relationships exist; * all four tables are now truncated at the
start of the data load step, for consistency; * the foreign key creation
commands now specify constraint names, to prevent accidentally creating
duplicate constraints by executing the 'f' step twice. Make some cosmetic
adjustments in the messages emitted by pgbench so that it's clear which steps
are getting run, and so that the messages agree with the documented names of
the steps. In passing, fix failure to enforce that the -v option is used only
in benchmarking mode. Masahiko Sawada, reviewed by Fabien Coelho,
editorialized a bit by me Discussion:
- Rearrange c.h to create a "compiler characteristics" section. Generalize
section 1 to handle stuff that is principally about the compiler (not
libraries), such as attributes, and collect stuff there that had been dropped
into various other parts of c.h. Also, push all the gettext macros into
section 8, so that section 0 is really just inclusions rather than inclusions
and random other stuff. The primary goal here is to get
pg_attribute_aligned() defined before section 3, so that we can use it with
int128. But this seems like good cleanup anyway. This patch just moves macro
definitions around, and shouldn't result in any changes in generated code.
But I'll push it out separately to see if the buildfarm agrees. Discussion:
- Prevent int128 from requiring more than MAXALIGN alignment. Our initial work
with int128 neglected alignment considerations, an oversight that came back to
bite us in bug #14897 from Vincent Lachenal. It is unsurprising that int128
might have a 16-byte alignment requirement; what's slightly more surprising is
that even notoriously lax Intel chips sometimes enforce that. Raising
MAXALIGN seems out of the question: the costs in wasted disk and memory space
would be significant, and there would also be an on-disk compatibility break.
Nor does it seem very practical to try to allow some data structures to have
more-than-MAXALIGN alignment requirement, as we'd have to push knowledge of
that throughout various code that copies data structures around. The only way
out of the box is to make type int128 conform to the system's alignment
assumptions. Fortunately, gcc supports that via its __attribute__(aligned())
pragma; and since we don't currently support int128 on non-gcc-workalike
compilers, we shouldn't be losing any platform support this way. Although we
could have just done pg_attribute_aligned(MAXIMUM_ALIGNOF) and called it a
day, I did a little bit of extra work to make the code more portable than
that: it will also support int128 on compilers without
__attribute__(aligned()), if the native alignment of their 128-bit-int type is
no more than that of int64. Add a regression test case that exercises the one
known instance of the problem, in parallel aggregation over a bigint column.
This will need to be back-patched, along with the preparatory commit
91aec93e6. But let's see what the buildfarm makes of it first. Discussion:
- Further refactoring of c.h and nearby files. This continues the work of
commit 91aec93e6 by getting rid of a lot of Windows-specific funny business in
"section 0". Instead of including pg_config_os.h in different places
depending on platform, let's standardize on putting it before the system
headers, and in consequence reduce win32.h to just what has to appear before
the system headers or the body of c.h (the latter category seems to include
only PGDLLIMPORT and PGDLLEXPORT). The rest of what was in win32.h is moved
to a new sub-include of port.h, win32_port.h. Some of what was in port.h
seems to better belong there too. It's possible that I missed some
declaration ordering dependency that needs to be preserved, but hopefully the
buildfarm will find that out in short order. Unlike the previous commit, no
back-patch, since this is just cleanup not a prerequisite for a bug fix.
Discussion: https://postgr.es/m/29650.1510761080@... https://git.postgresql.org/pg/commitdiff/ed9b3606dadb461aac57e41ac509f3892095a394
- Define _WINSOCK_DEPRECATED_NO_WARNINGS in all MSVC builds. Commit 0fb54de9a
thought that this was only needed in VS2015 and later, but buildfarm member
woodlouse shows that at least VS2013 whines as well. Let's just define it
regardless of MSVC version; it should be harmless enough in older releases.
Also, in the wake of ed9b3606d, it seems better to put it in win32_port.h
where <winsock2.h> is included. Since this is only suppressing a pedantic
compiler warning, I don't feel a need for a back-patch. Discussion:
- Make PL/Python handle domain-type conversions correctly. Fix PL/Python so
that it can handle domains over composite, and so that it enforces domain
constraints correctly in other cases that were not always done properly
before. Notably, it didn't do arrays of domains right (oversight in commit
c12d570fa), and it failed to enforce domain constraints when returning a
composite type containing a domain field, and if a transform function is being
used for a domain's base type then it failed to enforce domain constraints on
the result. Also, in many places it missed checking domain constraints on
null values, because the plpy_typeio code simply wasn't called for Py_None.
Rather than try to band-aid these problems, I made a significant refactoring
of the plpy_typeio logic. The existing design of recursing for array and
composite members is extended to also treat domains as containers requiring
recursion, and the APIs for the module are cleaned up and simplified. The
patch also modifies plpy_typeio to rely on the typcache more than it did
before (which was pretty much not at all). This reduces the need for
repetitive lookups, and lets us get rid of an ad-hoc scheme for detecting
changes in composite types. I added a couple of small features to typcache to
help with that. Although some of this is fixing bugs that long predate v11, I
don't think we should risk a back-patch: it's a significant amount of code
churn, and there've been no complaints from the field about the bugs. Tom
Lane, reviewed by Anthony Bykov Discussion:
- Clean up warnings in MinGW builds. Experimentation with modern MinGW
(specifically the 5.0.2 version packaged for Fedora 26) shows that its version
of sys/stat.h *does* provide S_IRGRP and friends, contrary to the expectation
of win32_port.h. This results in an astonishing number of compiler warnings,
and perhaps in incorrect code --- I'm not sure if the nonzero values supplied
by MinGW's header actually do anything. Hence, adjust win32_port.h to only
define these macros if <sys/stat.h> doesn't. This might be worth
back-patching, but given the lack of complaints so far, I'm not too excited
- Prevent to_number() from losing data when template doesn't match exactly.
Non-data template patterns would consume characters whether or not those
characters were what the pattern expected, for example SELECT
TO_NUMBER('1234', '9,999'); produced 134 because the '2' got eaten by the
comma pattern. This seems undesirable, not least because it doesn't happen in
Oracle. For the ',' and 'G' template patterns, we can fix this by consuming
characters only if they match what the pattern would output. For non-data
patterns such as 'L' and 'TH', it seems impractical to tighten things up to
the point of consuming only exact matches to what the pattern would output;
but we can improve matters quite a lot by redefining the behavior as "consume
only characters that aren't digits, signs, decimal point, or comma". Also,
fix it so that the behavior is to consume the number of *characters* the
pattern would output, not the number of *bytes*. The old coding would do
surprising things with non-ASCII currency symbols, for example. (It would be
good to apply that rule for literal text as well, but this commit only fixes
it for non-data patterns.) Oliver Ford, reviewed by Thomas Munro and Nathan
Wagner, and whacked around a bit more by me Discussion:
- Provide modern examples of how to auto-start Postgres on macOS. The scripts
in contrib/start-scripts/osx don't work at all on macOS 10.10 (Yosemite) or
later, because they depend on SystemStarter which Apple deprecated long ago
and removed in 10.10. Add a new subdirectory contrib/start-scripts/macos with
scripts that use the newer launchd infrastructure. Since this problem is
independent of which Postgres version you're using, back-patch to all
supported branches. Discussion:
- Fix quoted-substring handling in format parsing for to_char/to_number/etc.
This code evidently intended to treat backslash as an escape character within
double-quoted substrings, but it was sufficiently confused that cases like
..."foo\\"... did not work right: the second backslash managed to quote the
double-quote after it, despite being quoted itself. Rewrite to get that
right, while preserving the existing behavior outside double-quoted
substrings, which is that backslash isn't special except in the combination
\". Comparing to Oracle, it seems that their version of to_char() for
timestamps allows literal alphanumerics only within double quotes, while
non-alphanumerics are allowed outside quotes; backslashes aren't special
anywhere; there is no way at all to emit a literal double quote. (Bizarrely,
their to_char() for numbers is different; it doesn't allow literal text at all
AFAICT.) The fact that they don't treat backslash as special justifies our
existing behavior for backslash outside double quotes. I considered making
backslash inside double quotes act the same way (ie, special only if before
"), which in a green field would be a more consistent behavior. But that
would likely break more existing SQL code than what this patch does. Add some
test cases illustrating this behavior. (Only the last new case actually
changes behavior in this commit.) Little of this behavior was documented,
either, so fix that. Discussion:
- Improve to_date/to_number/to_timestamp behavior with multibyte characters.
The documentation says that these functions skip one input character per
literal (non-pattern) format character. Actually, though, they skipped one
input *byte* per literal *byte*, which could be hugely confusing if either
data or format contained multibyte characters. To fix, adjust the FormatNode
representation and parse_format() so that multibyte format characters are
stored as one FormatNode not several, and adjust the data-skipping bits to
advance by pg_mblen() not necessarily one byte. There's no user-visible
behavior change on the to_char() side, although the internal representation
changes. Commit e87d4965b had already fixed most places where we skip
characters on the basis of non-literal format patterns to advance by
characters not bytes, but this gets one more place, the SKIP_THth macro. I
think everything in formatting.c gets that right now. It'd be nice to have
some regression test cases covering this behavior; but of course there's no
way to do so in an encoding-agnostic way, and many of the interesting aspects
would also require unportable locale selections. So I've not bothered here.
Discussion: https://postgr.es/m/28186.1510957703@... https://git.postgresql.org/pg/commitdiff/976a1a48fc35cde3c750982be64f872c4de4d343
- Merge near-duplicate code in RI triggers. Merge ri_restrict_del and
ri_restrict_upd into one function ri_restrict. Create a function ri_setnull
that is the common implementation of RI_FKey_setnull_del and
RI_FKey_setnull_upd. Likewise create a function ri_setdefault that is the
common implementation of RI_FKey_setdefault_del and RI_FKey_setdefault_upd.
All of these pairs of functions were identical except for needing to check for
no-actual-key-change in the UPDATE cases; the one extra if-test is a small
price to pay for saving so much code. Aside from removing about 400 lines of
essentially duplicate code, this allows us to recognize that we were uselessly
caching two identical plans whenever there were pairs of triggers using these
duplicated functions (which is likely very common). Ildar Musin, reviewed by
Ildus Kurbangaliev Discussion:
- Support channel binding 'tls-unique' in SCRAM. This is the basic feature set
using OpenSSL to support the feature. In order to allow the frontend and the
backend to fetch the sent and expected TLS Finished messages, a PG-like API is
added to be able to make the interface pluggable for other SSL
implementations. This commit also adds a infrastructure to facilitate the
addition of future channel binding types as well as libpq parameters to
control the SASL mechanism names and channel binding names. Those will be
added by upcoming commits. Some tests are added to the SSL test suite to test
SCRAM authentication with channel binding. Author: Michael Paquier
<[hidden email]> Reviewed-by: Peter Eisentraut
- Consistently catch errors from Python _New() functions. Python Py*_New()
functions can fail and return NULL in out-of-memory conditions. The previous
code handled that inconsistently or not at all. This change organizes that
better. If we are in a function that is called from Python, we just check for
failure and return NULL ourselves, which will cause any exception information
to be passed up. If we are called from PostgreSQL, we consistently create an
"out of memory" error. Reviewed-by: Tom Lane <[hidden email]>
- Disable installcheck tests for test_session_hooks. The module requires a
preloaded library and the defect can't be cured by a LOAD instruction in the
test script. To achieve this we override the installcheck target in the
module's Makefile, and exclude ithe module in vcregress.pl. Along the way,
revert commit 9989f92aabd.
- Back out the session_start and session_end hooks feature. It's become
apparent during testing that there are problems with at least the testing
regime. I don't think we should have it without a working test regime, and the
difficulties might indicate implementation problems anyway, so I'm backing out
the whole thing until that's sorted out. This reverts commits 7459484 9989f92
Thomas Munro sent in another revision of a patch to implement ldaps.
Noah Misch sent in a patch to fix building PL/Perl with ActiveState Perl 5.22
Mark Rofail sent in another revision of a patch to implement foriegn key arrays.
Anthony Bykov sent in another revision of a patch to add a PL/PythonU TRANSFORM
Dean Rasheed sent in another revision of a patch to add a RLS summary table to
Fabien COELHO sent in another revision of a patch to fix a pgbench regression
Konstantin Knizhnik sent in a patch to the PostgrSQL FDW which enables
concurrent execution of aggregates by all remote servers.
Álvaro Herrera sent in three more revisions of a patch to implement indexes for
Oliver Ford sent in another revision of a patch to fix number skipping in
Thomas Munro sent in another revision of a patch to implement parallel hash.
Masahiko Sawada sent in two revisions of a patch to fix a bug where
ginInsertCleanup called from vacuum could still miss tuples to be deleted.
Andres Freund sent in two more revisions of a patch to fix freezing of a dead
Moon Insung sent in a patch to add a buffer state column to pg_buffercache and
add a function to decode buffer state.
Andrew Dunstan sent in another revision of a patch to add a toast_tuple_target
parameter to tables.
Peter Eisentraut sent in another revision of a patch to implement SQL
Anthony Bykov and Pavel Stěhule traded patches to add PL/Perl TRANSFORMs for
Alexander Korotkov sent in another revision of a patch to implement incremental
Rushabh Lathia sent in another revision of a patch to implement parallel B-tree
index build sorting.
Beena Emerson sent in another revision of a patch to implement runtime partition
Ildus Kurbangaliev sent in another revision of a patch to implement custom
David Rowley sent in another revision of a patch to implement bms_add_range().
Nikita Glukhov sent in another revision of a patch to implement SQL/JSON.
David Rowley sent in another revision of a patch to remove [Merge]Append nodes
which contain a single subpath.
Robert Haas sent in another revision of a patch to implement wire protocol
Masahiko Sawada sent in a patch to fix a bug that manifested as a failure with
user-defined data types in logical replication.
Michaël Paquier and Masahiko Sawada traded patches to fix an assertion failure
when the non-exclusive pg_stop_backup aborted.
Jeevan Chalke sent in another revision of a patch to implement partition-wise
Peter Eisentraut sent in another revision of a patch to implement transaction
control in PL procedures.
Michaël Paquier sent in another revision of a patch to add timeline ID in
Stas Kelvich sent in a patch to create a flag in pgproc indicating that hw lock
is acquired and include in RunningXacts only transaction with this flag set.
Brian Cloutier sent in a patch to add PGDLLIMPORT lines to some variables for
use on Windows.
Kyotaro HORIGUCHI sent in another revision of a patch to add columns
Kyotaro HORIGUCHI sent in another revision of a patch to fix walsender timeouts
when decoding a large transaction.
Masahiko Sawada sent in a doc patch to add a missing wal_receiver_status_interval
to the subscribers section.
Fabrízio de Royes Mello sent in another revision of a patch to add hooks for
session start and session end.
Amit Langote sent in another revision of a patch to prune partitions faster.
Robert Haas sent in a patch to fix multiple problems with
Amit Langote sent in a patch to fix a bug in default range partition and
Amit Kapila sent in another revision of a patch to pass eflags to workers.
Tomas Vondra sent in another revision of a patch to implement multivariate
histograms and MCV lists.
Tomas Vondra sent in another revision of a patch to implement BRIN multi-range
GaoZengqi (高增琦) sent in another revision of a patch to add a dependency
between client executes and static libraries.
Alik Khilazhev sent in another revision of a patch to add a zipfian distribution