== PostgreSQL Weekly News - August 10, 2019 ==

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

== PostgreSQL Weekly News - August 10, 2019 ==

David Fetter
== PostgreSQL Weekly News - August 10, 2019 ==

PostgreSQL security releases 11.5, 10.10, 9.6.15, 9.5.19, and 9.4.24 released.
Upgrade as soon as possible.
https://www.postgresql.org/about/news/1960/

== PostgreSQL Jobs for August ==

http://archives.postgresql.org/pgsql-jobs/2019-08/

== PostgreSQL Local ==

The first Austrian pgDay, will take place September 6, 2019 at the Hilton Garden
Inn in Wiener Neustadt.
https://pgday.at/en/

PostgresOpen will be September 11th - 13th, 2019 in Orlando, Florida at the
Rosen Centre Hotel.  The CfP is open at https://2019.postgresopen.org/callforpapers/
https://2019.postgresopen.org/

PostgresConf South Africa 2019 will take place in Johannesburg on October 8-9, 2019
https://postgresconf.org/conferences/SouthAfrica2019

PostgreSQL Conference Europe 2019 will be held on October 15-18, 2019 in Milan,
Italy.
https://2019.pgconf.eu/

2Q PGConf 2019 will be held December 4 & 5 in Chicago.
The CFP is open through August 30, 2019.
https://www.2qpgconf.com/

pgDay Paris 2020 will be held in Paris, France on March 26, 2020
at Espace Saint-Martin.
http://2020.pgday.paris/

Nordic PGDay 2020 will be held in Helsinki, Finland at the Hilton Helsinki
Strand Hotel on March 24, 2020.  The CfP is open through December 31, 2019 at
https://2020.nordicpgday.org/cfp/


== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to [hidden email].

== Applied Patches ==

Tom Lane pushed:

- Improve test coverage for LISTEN/NOTIFY. We had no actual end-to-end test of
  NOTIFY message delivery.  In the core async.sql regression test, testing this
  is problematic because psql traditionally prints the PID of the sending
  backend, making the output unstable.  We also have an isolation test script,
  but it likewise failed to prove that delivery worked, because
  isolationtester.c had no provisions for detecting/reporting NOTIFY messages.
  Hence, add such provisions to isolationtester.c, and extend async-notify.spec
  to include direct tests of basic NOTIFY functionality.  I also added tests
  showing that NOTIFY de-duplicates messages normally, but not across
  subtransaction boundaries.  (That's the historical behavior since we
  introduced subtransactions, though perhaps we ought to change it.)  Patch by
  me, with suggestions/review by Andres Freund.  Discussion:
  https://postgr.es/m/31304.1564246011@...
  https://git.postgresql.org/pg/commitdiff/b10f40bf0e4516d7832db8ccbe5f76319ad08682

- Fix busted logic for parallel lock grouping in TopoSort(). A "break" statement
  erroneously left behind by commit a1c1af2a1 caused TopoSort to do the wrong
  thing if a lock's wait list contained multiple members of the same locking
  group.  Because parallel workers don't normally need any locks not already
  taken by their leader, this is very hard --- maybe impossible --- to hit in
  production.  Still, if it did happen, the queries involved in an
  otherwise-resolvable deadlock would block until canceled.  In addition to
  removing the bogus "break", add an Assert showing that the conflicting uses of
  the beforeConstraints[] array (for both counts and flags) don't overlap, and
  add some commentary explaining why not; because it's not obvious without
  explanation, IMHO.  Original report and patch from Rui Hai Jiang; additional
  assert and commentary by me.  Back-patch to 9.6 where the bug came in.
  Discussion:
  https://postgr.es/m/CAEri+mLd3bpHLyW+a9pSe1y=aEkeuJpwBSwvo-+m4n7-ceRmXw@...
  https://git.postgresql.org/pg/commitdiff/3420851a2c2d2ac49b8ba53ccec5d02aa1e6a272

- Fix pg_dump's handling of dependencies for custom opclasses. Since pg_dump
  doesn't treat the member operators and functions of operator classes/families
  (that is, the pg_amop and pg_amproc entries, not the underlying
  operators/functions) as separate dumpable objects, it missed their dependency
  information.  I think this was safe when the code was designed, because the
  default object sorting rule emits operators and functions before opclasses,
  and there were no dependency types that could mess that up.  However, the
  introduction of range types in 9.2 broke it: now a type can have a dependency
  on an opclass, allowing dependency rules to push the opclass before the type
  and hence before custom operators. Lacking any information showing that it
  shouldn't do so, pg_dump emitted the objects in the wrong order.  Fix by
  teaching getDependencies() to translate pg_depend entries for pg_amop/amproc
  rows to look like dependencies for their parent opfamily.  I added a
  regression test for this in HEAD/v12, but not further back; life is too short
  to fight with 002_pg_dump.pl.  Per bug #15934 from Tom Gottfried.  Back-patch
  to all supported branches.  Discussion:
  https://postgr.es/m/15934-58b8c8ab7a09ea15@...
  https://git.postgresql.org/pg/commitdiff/07b39083c2aca003c4b1f289d7dc2368b5e2297a

- Mark advisory-lock functions as parallel restricted, not parallel unsafe.
  There seems no good reason not to allow a parallel leader to execute these
  functions.  (The workers still can't, though.  Although the code would work,
  any such lock would go away at worker exit, which is not the documented
  behavior of advisory locks.)  Discussion:
  https://postgr.es/m/11847.1564496844@...
  https://git.postgresql.org/pg/commitdiff/4886da8327507dddd830786b0c7aaa9cfc480b4b

- Add an isolation test to exercise parallel-worker deadlock resolution. Commit
  a1c1af2a1 added logic in the deadlock checker to handle lock grouping, but it
  was very poorly tested, as evidenced by the bug fixed in 3420851a2.  Add a
  test case that exercises that a bit better (and catches the bug --- if you
  revert 3420851a2, this will hang).  Since it's pretty hard to get parallel
  workers to take exclusive regular locks that their parents don't already have,
  this test operates by creating a deadlock among advisory locks taken in
  parallel workers. To make that happen, we must override the parallel-safety
  labeling of the advisory-lock functions, which we do by putting them in
  mislabeled, non-inlinable wrapper functions.  We also have to remove the
  redundant PreventAdvisoryLocksInParallelMode checks in lockfuncs.c.  That
  seems fine though; if some user accidentally does what this test is
  intentionally doing, not much harm will ensue. (If there are any remaining
  bugs that are reachable that way, they're probably reachable in other ways
  too.)  Discussion: https://postgr.es/m/3243.1564437314@...
  https://git.postgresql.org/pg/commitdiff/da9456d22a7697ef2c5ba9dd1402d948b2ec7f09

- Allow functions-in-FROM to be pulled up if they reduce to constants. This
  allows simplification of the plan tree in some common usage patterns: we can
  get rid of a join to the function RTE.  In principle we could pull up any
  immutable expression, but restricting it to Consts avoids the risk that
  multiple evaluations of the expression might cost more than we can save.
  (Possibly this could be improved in future --- but we've more or less promised
  people that putting a function in FROM guarantees single evaluation, so we'd
  have to tread carefully.)  To do this, we need to rearrange when
  eval_const_expressions() happens for expressions in function RTEs.  I moved it
  to inline_set_returning_functions(), which already has to iterate over every
  function RTE, and in consequence renamed that function to
  preprocess_function_rtes().  A useful consequence is that
  inline_set_returning_function() no longer has to do this for itself,
  simplifying that code.  In passing, break out pull_up_simple_subquery's code
  that knows where everything that needs pullup_replace_vars() processing is, so
  that the new pull_up_constant_function() routine can share it.  We'd gotten
  away with one-and-a-half copies of that code so far, since
  pull_up_simple_values() could assume that a lot of cases didn't apply to it
  --- but I don't think pull_up_constant_function() can make any simplifying
  assumptions.  Might as well make pull_up_simple_values() use it too.
  (Possibly this refactoring should go further: maybe we could share some of the
  code to fill in the pullup_replace_vars_context struct? For now, I left it
  that the callers fill that completely.)  Note: the one existing test case that
  this patch changes has to be changed because inlining its function RTEs would
  destroy the point of the test, namely to check join order.  Alexander
  Kuzmenkov and Aleksandr Parfenov, reviewed by Antonin Houska and Anastasia
  Lubennikova, and whacked around some more by me  Discussion:
  https://postgr.es/m/402356c32eeb93d4fed01f66d6c7fe2d@...
  https://git.postgresql.org/pg/commitdiff/7266d0997dd2a0632da38a594c78e25ff21df67e

- Avoid picking already-bound TCP ports in kerberos and ldap test suites.
  src/test/kerberos and src/test/ldap need to run a private authentication
  server of the relevant type, for which they need a free TCP port. They were
  just picking a random port number in 48K-64K, which works except when
  something's already using the particular port.  Notably, the probability of
  failure rises dramatically if one simply runs those tests in a tight loop,
  because each test cycle leaves behind a bunch of high ports that are
  transiently in TIME_WAIT state.  To fix, split out the code that
  PostgresNode.pm already had for identifying a free TCP port number, so that it
  can be invoked to choose a port for the KDC or LDAP server.  This isn't 100%
  bulletproof, since conceivably something else on the machine could grab the
  port between the time we check and the time we actually start the server.  But
  that's a pretty short window, so in practice this should be good enough.
  Back-patch to v11 where these test suites were added.  Patch by me, reviewed
  by Andrew Dunstan.  Discussion:
  https://postgr.es/m/3397.1564872168@...
  https://git.postgresql.org/pg/commitdiff/803466b6ffaa2e5b94d8ce4d7fffa8185f2a0184

- Fix handling of "undef" in contrib/jsonb_plperl. Perl has multiple internal
  representations of "undef", and just testing for SvTYPE(x) == SVt_NULL doesn't
  recognize all of them, leading to "cannot transform this Perl type to jsonb"
  errors. Use the approved test SvOK() instead.  Report and patch by Ivan
  Panchenko.  Back-patch to v11 where this module was added.  Discussion:
  https://postgr.es/m/1564783533.324795401@...
  https://git.postgresql.org/pg/commitdiff/e0f5048851ff88a53630a0c121a1cd15f6a2f1cd

- Fix choice of comparison operators for cross-type hashed subplans. Commit
  bf6c614a2 rearranged the lookup of the comparison operators needed in a hashed
  subplan, and in so doing, broke the cross-type case: it caused the original
  LHS-vs-RHS operator to be used to compare hash table entries too (which of
  course are all of the RHS type). This leads to C functions being passed a
  Datum that is not of the type they expect, with the usual hazards of crashes
  and unauthorized server memory disclosure.  For the set of hashable cross-type
  operators present in v11 core Postgres, this bug is nearly harmless on 64-bit
  machines, which may explain why it escaped earlier detection.  But it is a
  live security hazard on 32-bit machines; and of course there may be extensions
  that add more hashable cross-type operators, which would increase the risk.
  Reported by Andreas Seltenreich.  Back-patch to v11 where the problem came in.
  Security: CVE-2019-10209
  https://git.postgresql.org/pg/commitdiff/4766dce0dd1a1a26db253dfc81773a2c55cd2555

- Save Kerberos and LDAP daemon logs where the buildfarm can find them.
  src/test/kerberos and src/test/ldap try to run private authentication servers,
  which of course might fail.  The logs from these servers were being dropped
  into the tmp_check/ subdirectory, but they should be put in tmp_check/log/,
  because the buildfarm will only capture log files in that subdirectory.
  Without the log output there's little hope of diagnosing buildfarm failures
  related to these servers.  Backpatch to v11 where these test suites were
  added.  Discussion: https://postgr.es/m/16017.1565047605@...
  https://git.postgresql.org/pg/commitdiff/4ecd05cb771313d8e4d4007ec02408e5f18de5d2

- Fix intarray's GiST opclasses to not fail for empty arrays with <@.
  contrib/intarray considers "arraycol <@ constant-array" to be indexable, but
  its GiST opclass code fails to reliably find index entries for empty array
  values (which of course should trivially match such queries). This is because
  the test condition to see whether we should descend through a non-leaf node is
  wrong.  Unfortunately, empty array entries could be anywhere in the index, as
  these index opclasses are currently designed.  So there's no way to fix this
  except by lobotomizing <@ indexscans to scan the whole index ... which is what
  this patch does.  That's pretty unfortunate: the performance is now actually
  worse than a seqscan, in most cases. We'd be better off to remove <@ from the
  GiST opclasses entirely, and perhaps a future non-back-patchable patch will do
  so.  In the meantime, applications whose performance is adversely impacted
  have a couple of options.  They could switch to a GIN index, which doesn't
  have this bug, or they could replace "arraycol <@ constant-array" with
  "arraycol <@ constant-array AND arraycol && constant-array". That will provide
  about the same performance as before, and it will find all non-empty subsets
  of the given constant-array, which is all that could reliably be expected of
  the query before.  While at it, add some more regression test cases to improve
  code coverage of contrib/intarray.  In passing, adjust resize_intArrayType so
  that when it's returning an empty array, it uses construct_empty_array for
  that rather than cowboy hacking on the input array.  While the hack produces
  an array that looks valid for most purposes, it isn't bitwise equal to empty
  arrays produced by other code paths, which could have subtle odd effects.  I
  don't think this code path is performance-critical enough to justify such
  shortcuts.  (Back-patch this part only as far as v11; before commit 01783ac36
  we were not careful about this in other intarray code paths either.)
  Back-patch the <@ fixes to all supported versions, since this was broken from
  day one.  Patch by me; thanks to Alexander Korotkov for review.  Discussion:
  https://postgr.es/m/458.1565114141@...
  https://git.postgresql.org/pg/commitdiff/efc77cf5f1deb96a5233c04d287ead01ee7b260d

- Doc: document permissions required for ANALYZE. VACUUM's reference page had
  this text, but ANALYZE's didn't.  That's a clear oversight given that section
  5.7 explicitly delegates the responsibility to define permissions requirements
  to the individual commands' man pages.  Per gripe from Isaac Morland.
  Back-patch to all supported branches.  Discussion:
  https://postgr.es/m/CAMsGm5fp3oBUs-2iRfii0iEO=fZuJALVyM2zJLhNTjG34gpAVQ@...
  https://git.postgresql.org/pg/commitdiff/e94cd0a8eb831d3ad756ad5f1e69b05392038355

- Cosmetic improvements in setup of planner's per-RTE arrays. Merge
  setup_append_rel_array into setup_simple_rel_arrays.  There's no particularly
  good reason to keep them separate, and it's inconsistent with the lack of
  separation in expand_planner_arrays.  The only apparent benefit was that the
  fast path for trivial queries in query_planner() doesn't need to set up the
  append_rel_array; but all we're saving there is an if-test and NULL
  assignment, which surely ought to be negligible.  Also improve some obsolete
  comments.  Discussion: https://postgr.es/m/17220.1565301350@...
  https://git.postgresql.org/pg/commitdiff/1661a4050593a472c369a6660ffec05b6b837c57

- Fix SIGSEGV in pruning for ScalarArrayOp with constant-null array. Not much to
  be said here: commit 9fdb675fc should have checked constisnull, didn't.  Per
  report from Piotr Włodarczyk.  Back-patch to v11 where bug was introduced.
  Discussion:
  https://postgr.es/m/CAP-dhMr+vRpwizEYjUjsiZ1vwqpohTm+3Pbdt6Pr7FEgPq9R0Q@...
  https://git.postgresql.org/pg/commitdiff/0662eb6219f1f4bafc1916a0bf2813cdc58e5eca

- Fix "ANALYZE t, t" inside a transaction block. This failed with either "tuple
  already updated by self" or "duplicate key value violates unique constraint",
  depending on whether the table had previously been analyzed or not.  The
  reason is that ANALYZE tried to insert or update the same pg_statistic rows
  twice, and there was no CommandCounterIncrement between.  So add one.  The
  same case works fine outside a transaction block, because then there's a whole
  transaction boundary between, as a consequence of the way VACUUM works.  This
  issue has been latent all along, but the problem was unreachable before commit
  11d8d72c2 added the ability to specify multiple tables in ANALYZE.  We could,
  perhaps, alternatively fix it by adding code to de-duplicate the list of
  VacuumRelations --- but that would add a lot of overhead to work around dumb
  commands, so it's not attractive.  Per bug #15946 from Yaroslav Schekin.
  Back-patch to v11.  (Note: in v11 I also back-patched the test added by commit
  23224563d; otherwise the problem doesn't manifest in the test I added, because
  "vactst" is empty when the tests for multiple ANALYZE targets are reached.
  That seems like not a very good thing anyway, so I did this rather than
  rethinking the choice of test case.)  Discussion:
  https://postgr.es/m/15946-5c7570a2884a26cf@...
  https://git.postgresql.org/pg/commitdiff/cabe0f298ea7efade11d8171c617e668934d0d09

Thomas Munro pushed:

- Avoid macro clash with LLVM 9. Early previews of LLVM 9 reveal that our Min()
  macro causes compiler errors in LLVM headers reached by the #include
  directives in llvmjit_inline.cpp.  Let's just undefine it.  Per buildfarm
  animal seawasp.  Back-patch to 11.  Reviewed-by: Fabien Coelho, Tom Lane
  Discussion: https://postgr.es/m/20190606173216.GA6306%40alvherre.pgsql
  https://git.postgresql.org/pg/commitdiff/a2a777d011971ace3a349a3f02b1bf6eeea07bf2

Michaël Paquier pushed:

- Fix handling of expressions and predicates in REINDEX CONCURRENTLY. When
  copying the definition of an index rebuilt concurrently for the new entry, the
  index information was taken directly from the old index using the relation
  cache.  In this case, predicates and expressions have some post-processing to
  prepare things for the planner, which loses some information including the
  collations added in any of them.  This inconsistency can cause issues when
  attempting for example a table rewrite, and makes the new indexes rebuilt
  concurrently inconsistent with the old entries.  In order to fix the problem,
  fetch expressions and predicates directly from the catalog of the old entry,
  and fill in IndexInfo for the new index with that.  This makes the process
  more consistent with DefineIndex(), and the code is refactored with the
  addition of a routine to create an IndexInfo node.  Reported-by: Manuel Rigger
  Author: Michael Paquier Discussion:
  https://postgr.es/m/CA+u7OA5Hp0ra235F3czPom_FyAd-3+XwSJmX95r1+sRPOJc9VQ@...
  Backpatch-through: 12
  https://git.postgresql.org/pg/commitdiff/7cce159349ccdb39ade07f869f08e4929ef2fe0b

- Fix inconsistencies and typos in the tree. This is numbered take 8, and
  addresses again a set of issues with code comments, variable names and
  unreferenced variables.  Author: Alexander Lakhin Discussion:
  https://postgr.es/m/b137b5eb-9c95-9c2f-586e-38aba7d59788@...
  https://git.postgresql.org/pg/commitdiff/eb43f3d19324d7e5376b1f57fc2e5c142a6b5f3d

- Fix memory leak coming from simple lists built in reindexdb. When building a
  list of relations for a parallel processing of a schema or a database (or just
  a single-entry list for the non-parallel case with the database name), the
  list is allocated and built on-the-fly for each database processed, leaking
  after one database-level reindex is done.  This accumulates leaks when
  processing all databases, and could become a visible issue with thousands of
  relations.  This is fixed by introducing a new routine in simple_list.c to
  free all the elements in a simple list made of strings or OIDs.  The header of
  the list may be using a variable declaration or an allocated pointer, so we
  don't have a routine to free this part to keep the interface simple.  Per
  report from coverity for an issue introduced by 5ab892c, and valgrind
  complains about the leak as well.  The idea to introduce a new routine in
  simple_list.c is from Tom Lane.  Author: Michael Paquier Reviewed-by: Tom Lane
  https://git.postgresql.org/pg/commitdiff/04cf0bfc90dfae89a794d2bdd88fe3b8e313798e

- Remove orphaned structure member in pgcrypto. int_name has never been used for
  digest lookups since its introduction in e94dd6a.  Author: Daniel Gustafsson
  Discussion: https://postgr.es/m/386C26CB-628B-4A4C-8879-D8BF190F2C77@...
  https://git.postgresql.org/pg/commitdiff/652a8947d981db0367bcff5b123545eba0049878

- Fix handling of previous password hooks in passwordcheck. When piling up
  loading of modules using check_password_hook_type, loading passwordcheck would
  remove any trace of a previously-loaded hook.  Unloading the module would also
  cause previous hooks to be entirely gone.  Reported-by: Rafael Castro Author:
  Michael Paquier Reviewed-by: Daniel Gustafsson Discussion:
  https://postgr.es/m/15932-78f48f9ef166778c@... Backpatch-through:
  9.4
  https://git.postgresql.org/pg/commitdiff/b2a3d706b8d76b9d65e953942fc1ccafe892f692

- Fix format truncation issue from ECPG test. This fixes one warning generated
  by GCC and present in the test case array part of ECPG.  This likely got
  missed in past fixes like 3a4b891 because the compilation of those tests is
  not done by default.  Reported-by: Sergei Kornilov Discussion:
  https://postgr.es/m/14951331562847675@...
  https://git.postgresql.org/pg/commitdiff/a9f301df0e76c38d4544477c1b3e5e29d57904e6

- Refactor BuildIndexInfo() with the new makeIndexInfo(). This portion of the
  code got forgotten in 7cce159 which has introduced a new routine to build this
  node, and this finishes the unification of the places where IndexInfo is
  initialized.  Author: Michael Paquier Discussion:
  https://postgr.es/m/20190801041322.GA3435@...
  https://git.postgresql.org/pg/commitdiff/69edf4f8802247209e77f69e089799b3d83c13a4

- Fix inconsistencies and typos in the tree, take 9. This addresses more issues
  with code comments, variable names and unreferenced variables.  Author:
  Alexander Lakhin Discussion:
  https://postgr.es/m/7ab243e0-116d-3e44-d120-76b3df7abefd@...
  https://git.postgresql.org/pg/commitdiff/8548ddc61b5858b6466e69f66a6b1a7ea9daef06

- Fix tab completion for ALTER LANGUAGE in psql. OWNER_TO was used for the
  completion, which is not a supported grammar, but OWNER TO is.  This error has
  been introduced by d37b816, so backpatch down to 9.6.  Author: Alexander
  Lakhin Discussion:
  https://postgr.es/m/7ab243e0-116d-3e44-d120-76b3df7abefd@...
  Backpatch-through: 9.6
  https://git.postgresql.org/pg/commitdiff/05ba8370b8e4b5c8f3dd51986b9fdeb43fed5610

- Add safeguards in LSN, numeric and float calculation for custom errors. Those
  data types use parsing and/or calculation wrapper routines which can generate
  some generic error messages in the event of a failure.  The caller of these
  routines can also pass a pointer variable settable by the routine to track if
  an error has happened, letting the caller decide what to do in the event of an
  error and what error message to generate.  Those routines have been slacking
  the initialization of the tracking flag, which can be confusing when reading
  the code, so add some safeguards against calls of these parsing routines which
  could lead to a dubious result.  The LSN parsing gains an assertion to make
  sure that the tracking flag is set, while numeric and float paths initialize
  the flag to a saner state.  Author: Jeevan Ladhe Reviewed-by: Álvaro Herrera,
  Michael Paquier Discussion:
  https://postgr.es/m/CAOgcT0NOM9oR0Hag_3VpyW0uF3iCU=BDUFSPfk9JrWXRcWQHqw@...
  https://git.postgresql.org/pg/commitdiff/a76cfba663ceab79b891cc81a5f941051755b3b0

- Fix typo in pathnode.c. Author: Amit Langote Discussion:
  https://postgr.es/m/CA+HiwqFhZ6ABoz-i=JZ5wMMyz-orx4asjR0og9qBtgEwOww6Yg@...
  https://git.postgresql.org/pg/commitdiff/940c8b01b071b777ed56c086c383e054fbd0456e

- Adjust tuple data lookup logic in multi-insert logical decoding. As of now,
  logical decoding of a multi-insert has been scanning all xl_multi_insert_tuple
  entries only if XLH_INSERT_CONTAINS_NEW_TUPLE was getting set in the record.
  This is not an issue on HEAD as multi-insert records are not used for system
  catalogs, but the logical decoding logic includes all the code necessary to
  handle that properly, except that the code missed to iterate correctly over
  all xl_multi_insert_tuple entries when the flag is not set.  Hence, when
  trying to use multi-insert for system catalogs, an assertion would be
  triggered.  An upcoming patch is going to make use of multi-insert for system
  catalogs, and this fixes the logic to make sure that all entries are scanned
  correctly without softening the existing assertions.  Reported-by: Daniel
  Gustafsson Author: Michael Paquier Reviewed-by: Daniel Gustafsson Discussion:
  https://postgr.es/m/CBFFD532-C033-49EB-9A5A-F67EAEE9EB0B@...
  https://git.postgresql.org/pg/commitdiff/75c1921cd6c868c5995b88113b4463a4830b9a27

- Fix some incorrect parsing of time with time zone strings. When parsing a
  timetz string with a dynamic timezone abbreviation or a timezone not
  specified, it was possible to generate incorrect timestamps based on a date
  which uses some non-initialized variables if the input string did not specify
  fully a date to parse.  This is already checked when a full timezone spec is
  included in the input string, but the two other cases mentioned above missed
  the same checks.  This gets fixed by generating an error as this input is
  invalid, or in short when a date is not fully specified.  Valgrind was
  complaining about this problem.  Bug: #15910 Author: Alexander Lakhin
  Discussion: https://postgr.es/m/15910-2eba5106b9aa0c61@...
  Backpatch-through: 9.4
  https://git.postgresql.org/pg/commitdiff/64579be64a3811d08ea7ccf92b1a443d76b96412

- Refactor logic to remove trailing CR/LF characters from strings. b654714 has
  reworked the way trailing CR/LF characters are removed from strings.  This
  commit introduces a new routine in common/string.c and refactors the code so
  as the logic is in a single place, mostly.  Author: Michael Paquier
  Reviewed-by: Bruce Momjian Discussion:
  https://postgr.es/m/20190801031820.GF29334@...
  https://git.postgresql.org/pg/commitdiff/b8f2da0ac5a24f669c8d320c58646759b8fc69a5

Peter Eisentraut pushed:

- Handle fsync failures in pg_receivewal and pg_recvlogical. It is not safe to
  simply report an fsync error and continue.  We must exit the program instead.
  Reviewed-by: Michael Paquier <[hidden email]> Reviewed-by: Sehrope
  Sarkuni <[hidden email]> Discussion:
  https://www.postgresql.org/message-id/flat/9b49fe44-8f3e-eca9-5914-29e9e99030bf@...
  https://git.postgresql.org/pg/commitdiff/1e2fddfa33d3c7cc93ca3ee0f32852699bd3e012

- Run UTF8-requiring collation tests by default. The tests collate.icu.utf8 and
  collate.linux.utf8 were previously only run when explicitly selected via
  EXTRA_TESTS.  They require a UTF8 database, because the error messages in the
  expected files refer to that, and they use some non-ASCII characters in the
  tests.  Since users can select any locale and encoding for the regression test
  run, it was not possible to include these tests automatically.  To fix, use
  psql's \if facility to check various prerequisites such as platform and the
  server encoding and quit the tests at the very beginning if the configuration
  is not adequate.  We then need to maintain alternative expected files for
  these tests, but they are very tiny and never need to change after this.
  These two tests are now run automatically as part of the regression tests.
  Reviewed-by: Tom Lane <[hidden email]> Discussion:
  https://www.postgresql.org/message-id/flat/052295c2-a2e1-9a21-bd36-8fbff8686cf3%402ndquadrant.com
  https://git.postgresql.org/pg/commitdiff/f140007050a2ba874b85c4578d8417828f4b64b6

- Add error codes to some corruption log messages. In some cases we have
  elog(ERROR) while corruption is certain and we can give a clear error code
  ERRCODE_DATA_CORRUPTED or ERRCODE_INDEX_CORRUPTED.  Author: Andrey Borodin
  <[hidden email]> Discussion:
  https://www.postgresql.org/message-id/flat/25F6C686-6442-4A6B-BAF8-A6F7B84B16DE@...
  https://git.postgresql.org/pg/commitdiff/fd6ec93bf890314ac694dc8a7f3c45702ecc1bbd

- initdb: Use varargs macro for PG_CMD_PRINTF. I left PG_CMD_PUTS around even
  though it could be handled by PG_CMD_PRINTF since PG_CMD_PUTS is sometimes
  called with non-literal arguments, and so that would create a potential
  problem if such a string contained percent signs.  Reviewed-by: Tom Lane
  <[hidden email]>
  https://git.postgresql.org/pg/commitdiff/43211c2a02f39d6568496168413dc00e0399dc2e

Tomáš Vondra pushed:

- Don't build extended statistics on inheritance trees. When performing ANALYZE
  on inheritance trees, we collect two samples for each relation - one for the
  relation alone, and one for the inheritance subtree (relation and its child
  relations). And then we build statistics on each sample, so for each relation
  we get two sets of statistics.  For regular (per-column) statistics this works
  fine, because the catalog includes a flag differentiating statistics built
  from those two samples. But we don't have such flag in the extended statistics
  catalogs, and we ended up updating the same row twice, triggering this error:
  ERROR:  tuple already updated by self  The simplest solution is to disable
  extended statistics on inheritance trees, which is what this commit is doing.
  In the future we may need to do something similar to per-column statistics,
  but that requires adding a flag to the catalog - and that's not backpatchable.
  Moreover, the current selectivity estimation code only works with individual
  relations, so building statistics on inheritance trees would be pointless
  anyway.  Author: Tomas Vondra Backpatch-to: 10- Discussion:
  https://postgr.es/m/20190618231233.GA27470@... Reported-by: Justin
  Pryzby
  https://git.postgresql.org/pg/commitdiff/14ef15a22246ca17c949e7a9d1abe14c8874d743

- Revert "Silence compiler warning". This reverts commit
  9dc122585551516309c9362e673effdbf3bd79bd.  As committed, statement sampling
  used the existing duration threshold (log_min_duration_statement) when decide
  which statements to sample. The issue is that even the longest statements are
  subject to sampling, and so may not end up logged. An improvement was
  proposed, introducing a second duration threshold, but it would not be
  backwards compatible. So we've decided to revert this feature - the separate
  threshold should be part of the feature itself.  Discussion:
  https://postgr.es/m/CAFj8pRDS8tQ3Wviw9%3DAvODyUciPSrGeMhJi_WPE%2BEB8%2B4gLL-Q%40mail.gmail.com
  https://git.postgresql.org/pg/commitdiff/4f9ed8f3c5ef0034c98dd34549f85d8c72aab9ad

- Revert "Add log_statement_sample_rate parameter". This reverts commit
  88bdbd3f746049834ae3cc972e6e650586ec3c9d.  As committed, statement sampling
  used the existing duration threshold (log_min_duration_statement) when decide
  which statements to sample. The issue is that even the longest statements are
  subject to sampling, and so may not end up logged. An improvement was
  proposed, introducing a second duration threshold, but it would not be
  backwards compatible. So we've decided to revert this feature - the separate
  threshold should be part of the feature itself.  Discussion:
  https://postgr.es/m/CAFj8pRDS8tQ3Wviw9%3DAvODyUciPSrGeMhJi_WPE%2BEB8%2B4gLL-Q%40mail.gmail.com
  https://git.postgresql.org/pg/commitdiff/75506195da81d75597a4025b72f8367e6c45f60d

Heikki Linnakangas pushed:

- Print WAL position correctly in pg_rewind error message. This has been wrong
  ever since pg_rewind was added. The if-branch just above this, where we print
  the same error with an extra message supplied by XLogReadRecord() got this
  right, but the variable name was wrong in the else-branch. As a consequence,
  the error printed the WAL position as 0/0 if there was an error reading a WAL
  file.  Backpatch to 9.5, where pg_rewind was added.
  https://git.postgresql.org/pg/commitdiff/d8b094dabb0fa16388340ca823d0a38285d2d6ce

- Allow table AM's to use rd_amcache, too. The rd_amcache allows an index AM to
  cache arbitrary information in a relcache entry. This commit moves the cleanup
  of rd_amcache so that it can also be used by table AMs. Nothing takes
  advantage of that yet, but I'm sure it'll come handy for anyone writing new
  table AMs.  Backpatch to v12, where table AM interface was introduced.
  Reviewed-by: Julien Rouhaud
  https://git.postgresql.org/pg/commitdiff/a29834beb1deeb0aa06742dd77ba1d21b444ca44

- Fix predicate-locking of HOT updated rows. In serializable mode,
  heap_hot_search_buffer() incorrectly acquired a predicate lock on the root
  tuple, not the returned tuple that satisfied the visibility checks. As
  explained in README-SSI, the predicate lock does not need to be copied or
  extended to other tuple versions, but for that to work, the correct, visible,
  tuple version must be locked in the first place.  The original SSI commit had
  this bug in it, but it was fixed back in 2013, in commit 81fbbfe335. But
  unfortunately, it was reintroduced a few months later in commit b89e151054.
  Wising up from that, add a regression test to cover this, so that it doesn't
  get reintroduced again. Also, move the code that sets 't_self', so that it
  happens at the same time that the other HeapTuple fields are set, to make it
  more clear that all the code in the loop operate on the "current" tuple in the
  chain, not the root tuple.  Bug spotted by Andres Freund, analysis and
  original fix by Thomas Munro, test case and some additional changes to the fix
  by Heikki Linnakangas. Backpatch to all supported versions (9.4).  Discussion:
  https://www.postgresql.org/message-id/20190731210630.nqhszuktygwftjty%40alap3.anarazel.de
  https://git.postgresql.org/pg/commitdiff/1169fcf129f41762fa8a2d82b52eabd788b3a4de

Andres Freund pushed:

- Remove superfluous semicolon. Author: Andres Freund
  https://git.postgresql.org/pg/commitdiff/6384e87be28ee8d69ef11e49413b115506a3c6d3

- Remove superfluous newlines in function prototypes. These were introduced by
  pgindent due to fixe to broken indentation (c.f. 8255c7a5eeba8). Previously
  the mis-indentation of function prototypes was creatively used to reduce
  indentation in a few places.  As that formatting only exists in master and
  REL_12_STABLE, it seems better to fix it in both, rather than having some odd
  indentation in v12 that somebody might copy for future patches or such.
  Author: Andres Freund Discussion:
  https://postgr.es/m/20190728013754.jwcbe5nfyt3533vx@...
  Backpatch: 12-
  https://git.postgresql.org/pg/commitdiff/870b1d6800cc2173ab672449047efbc30bdc1b57

- Fix representation of hash keys in Hash/HashJoin nodes. In 5f32b29c1819 I
  changed the creation of HashState.hashkeys to actually use HashState as the
  parent (instead of HashJoinState, which was incorrect, as they were executed
  below HashState), to fix the problem of hashkeys expressions otherwise relying
  on slot types appropriate for HashJoinState, rather than HashState as would be
  correct. That reliance was only introduced in 12, which is why it previously
  worked to use HashJoinState as the parent (although I'd be unsurprised if
  there were problematic cases).  Unfortunately that's not a sufficient
  solution, because before this commit, the to-be-hashed expressions referenced
  inner/outer as appropriate for the HashJoin, not Hash. That didn't have
  obvious bad consequences, because the slots containing the tuples were put
  into ecxt_innertuple when hashing a tuple for HashState (even though Hash
  doesn't have an inner plan).  There are less common cases where this can cause
  visible problems however (rather than just confusion when inspecting such
  executor trees). E.g. "ERROR: bogus varno: 65000", when explaining queries
  containing a HashJoin where the subsidiary Hash node's hash keys reference a
  subplan. While normally hashkeys aren't displayed by EXPLAIN, if one of those
  expressions references a subplan, that subplan may be printed as part of the
  Hash node - which then failed because an inner plan was referenced, and Hash
  doesn't have that.  It seems quite possible that there's other broken cases,
  too.  Fix the problem by properly splitting the expression for the HashJoin
  and Hash nodes at plan time, and have them reference the proper subsidiary
  node. While other workarounds are possible, fixing this correctly seems easy
  enough. It was a pretty ugly hack to have ExecInitHashJoin put the expression
  into the already initialized HashState, in the first place.  I decided to not
  just split inner/outer hashkeys inside make_hashjoin(), but also to separate
  out hashoperators and hashcollations at plan time. Otherwise we would have
  ended up having two very similar loops, one at plan time and the other during
  executor startup. The work seems to more appropriately belong to plan time,
  anyway.  Reported-By: Nikita Glukhov, Alexander Korotkov Author: Andres Freund
  Reviewed-By: Tom Lane, in an earlier version Discussion:
  https://postgr.es/m/CAPpHfdvGVegF_TKKRiBrSmatJL2dR9uwFCuR+teQ_8tEXU8mxg@...
  Backpatch: 12-
  https://git.postgresql.org/pg/commitdiff/2abd7ae9b20bcd810d4f19d28aefb97048813825

Peter Geoghegan pushed:

- Add sort support routine for the inet data type. Add sort support for inet,
  including support for abbreviated keys. Testing has shown that this reduces
  the time taken to sort medium to large inet/cidr inputs by ~50-60% in
  realistic cases.  Author: Brandur Leach Reviewed-By: Peter Geoghegan, Edmund
  Horner Discussion:
  https://postgr.es/m/CABR_9B-PQ8o2MZNJ88wo6r-NxW2EFG70M96Wmcgf99G6HUQ3sw@...
  https://git.postgresql.org/pg/commitdiff/71dcd7438664d81235c72337cbbbfa780f7a0630

- Bump catversion. Oversight in commit 71dcd743.
  https://git.postgresql.org/pg/commitdiff/a8d6a95eb992e942838e41029537564d81c4a50e

- Show specific OID suggestion in unused_oids output. Commit a6417078
  established a new project policy around OID assignment: new patches are
  encouraged to choose a random OID in the 8000..9999 range when a
  manually-assigned OID is required (if multiple OIDs are required, a
  consecutive block of OIDs starting from the random point should be used).
  Catalog entries added by committed patches that use OIDs from this "unstable"
  range are renumbered after feature freeze. This practice minimizes OID
  collisions among concurrently-developed patches.  Show a specific random OID
  suggestion when the unused_oids script is run.  This makes it easy for patch
  authors to use a random OID from the unstable range, per the new policy.
  Author: Julien Rouhaud, Peter Geoghegan Reviewed-By: Tom Lane Discussion:
  https://postgr.es/m/CAH2-WzkkRs2ScmuBQ7xWi7xzp7fC1B3w0Nt8X+n4rBw5k+Z=zA@...
  https://git.postgresql.org/pg/commitdiff/98eab30b93d52a114bd65e154cda3402d0630667

- Update obsolete tuplesort READTUP() comment. READTUP() routines do not and
  cannot use the resettable "tuplecontext" memory context, since it is deleted
  when merging begins.  Update an obsolete comment that claimed otherwise.  This
  was an oversight in commit e94568ecc10.  In passing, fix an unrelated
  tuplesort typo.
  https://git.postgresql.org/pg/commitdiff/28b901f73a3924187988bfaac57d20e422a432c3

- Rename tuplesort.c's SortTuple.tupindex field. Rename the "tupindex" field
  from tuplesort.c's SortTuple struct to "srctape", since it can only ever be
  used to store a source/input tape number when merging external sort runs.
  This has been the case since commit 8b304b8b72b, which removed replacement
  selection sort from tuplesort.c.
  https://git.postgresql.org/pg/commitdiff/d8cd68c8d472292ef8943a765bd1c69c0d4d61d8

Jeff Davis pushed:

- Allow simplehash to use already-calculated hash values. Add _lookup_hash and
  _insert_hash functions for callers that have already calculated the hash value
  of the key.  The immediate use case is for hash algorithms that write to disk
  in partitions. The hash value can be calculated once, used to perform a
  lookup, used to select the partition, then written to the partition along with
  the tuple. When the tuple is read back, the hash value does not need to be
  recalculated.  Author: Jeff Davis Reviewed-by: Andres Freund Discussion:
  https://postgr.es/m/48abe675e1330f0c264ab2fe0d4ff23eb244f9ef.camel%40j-davis.com
  https://git.postgresql.org/pg/commitdiff/6ae4e8eae78e0781633f7b40a1b5cc189bc40923

Álvaro Herrera pushed:

- Improve pruning of a default partition. When querying a partitioned table
  containing a default partition, we were wrongly deciding to include it in the
  scan too early in the process, failing to exclude it in some cases.  If we
  reinterpret the PruneStepResult.scan_default flag slightly, we can do a better
  job at detecting that it can be excluded.  The change is that we avoid setting
  the flag for that pruning step unless the step absolutely requires the default
  partition to be scanned (in contrast with the previous arrangement, which was
  to set it unless the step was able to prune it). So get_matching_partitions()
  must explicitly check the partition that each returned bound value corresponds
  to in order to determine whether the default one needs to be included, rather
  than relying on the flag from the final step result.  Author: Yuzuko Hosoya
  <[hidden email]> Reviewed-by: Amit Langote
  <[hidden email]> Discussion:
  https://postgr.es/m/00e601d4ca86$932b8bc0$b982a340$@lab.ntt.co.jp
  https://git.postgresql.org/pg/commitdiff/489247b0e615592111226297a0564e11616361a5

- Apply constraint exclusion more generally in partitioning. We were applying
  constraint exclusion on the partition constraint when generating pruning steps
  for a clause, but only for the rather restricted situation of them being
  boolean OR operators; however it is possible to have differently shaped
  clauses that also benefit from constraint exclusion.  This applies
  particularly to the default partition since their constraints are in essence a
  long list of OR'ed subclauses ... but it applies to other cases too.  So in
  certain cases we're scanning partitions that we don't need to.  Remove the
  specialized code in OR clauses, and add a generally applicable test of the
  clause refuting the partition constraint; mark the whole pruning operation as
  contradictory if it hits.  This has the unwanted side-effect of testing some
  (most? all?) constraints more than once if constraint_exclusion=on.  That
  seems unavoidable as far as I can tell without some additional work, but
  that's not the recommended setting for that parameter anyway. However, because
  this imposes additional processing cost for all queries using partitioned
  tables, I decided not to backpatch this change.  Author: Amit Langote, Yuzuko
  Hosoya, Álvaro Herrera Reviewers: Shawn Wang, Thibaut Madeleine, Yoshikazu
  Imai, Kyotaro Horiguchi; they were also uncredited reviewers for commit
  489247b0e615. Discussion:
  https://postgr.es/m/9bb31dfe-b0d0-53f3-3ea6-e64b811424cf@...
  https://git.postgresql.org/pg/commitdiff/4e85642d935eb13341584df7776eb76585d45819

- Add comment on no default partition with hash partitioning. Discussion:
  https://postgr.es/m/20190806222735.GA9535@...
  https://git.postgresql.org/pg/commitdiff/12afc7145c03c212f26fea3a99e016da6a1c919c

- Remove unnecessary #include <limits.h>. This include was probably copied from
  tuplestore.c, but it's not needed.  Extracted from a larger patch submitted by
  vignesh C <[hidden email]>  Discussion:
  https://postgr.es/m/CALDaNm1B9naPDTm3ox1m_yZvOm3KA5S4kZQSWWAeLHAQ=3gV1Q@...
  https://git.postgresql.org/pg/commitdiff/e1f4c481b995d0f4ef9570aaf9638fe06bc5d742

- Clarify the default partition's role. Reviewed by Tom Lane and Amit Langote
  Discussion: https://postgr.es/m/20190806222735.GA9535@...
  https://git.postgresql.org/pg/commitdiff/956451e8bc9face2241d9a785e0d236a92f8e210

Noah Misch pushed:

- Require the schema qualification in pg_temp.type_name(arg). Commit
  aa27977fe21a7dfa4da4376ad66ae37cb8f0d0b5 introduced this restriction for
  pg_temp.function_name(arg); do likewise for types created in temporary
  schemas.  Programs that this breaks should add "pg_temp." schema qualification
  or switch to arg::type_name syntax. Back-patch to 9.4 (all supported
  versions).  Reviewed by Tom Lane.  Reported by Tom Lane.  Security:
  CVE-2019-10208
  https://git.postgresql.org/pg/commitdiff/ffa2d37e5fbd1243f918f622113d6e371667e5a0

Etsuro Fujita pushed:

- Fix typos in comments.
  https://git.postgresql.org/pg/commitdiff/68343b4ad75305391b38f4b42734dc07f2fe7ee2

Alexander Korotkov pushed:

- Fix some typos in jsonpath documentation. Discussion:
  https://postgr.es/m/8B7FA3B4-328D-43D7-95A8-37B8891B8C78%40winand.at Author:
  Markus Winand Backpatch-through: 12
  https://git.postgresql.org/pg/commitdiff/1f33f211bc531d257f84fefb9208dd43e8718972

Andrew Dunstan pushed:

- Fix certificate subjects in ldap test. openssl doesn't like lower case subject
  attribute names. Error observed in buildfarm results.  Backpatch to release
  11.
  https://git.postgresql.org/pg/commitdiff/15077ab63f29fd85ff519d1c456fda614774d28b

== Pending Patches ==

Alexander Lakhin sent in a patch to fix typos and inconsistencies.

Konstantin Knizhnik sent in another revision of a patch to implement
auto_prepare.

Daniel Gustafsson sent in another revision of a patch to implement catalog
multi-insert.

Fabien COELHO sent in a patch to speed up pgbench using a symbol table.

Andrey V. Lepikhov and Konstantin Knizhnik traded patches to remove unneeded
self-joins.

Ashwin Agrawal, Thomas Munro, and Heikki Linnakangas traded patches to fix
predicate-locking of HOT updated rows.

Amit Langote and Etsuro Fujita traded patches to fix a partition routing
layering violation in nodeModifyTable.c.

Alexander Korotkov sent in another revision of a patch to add \dA (access
methods) to psql.

Peter Geoghegan sent in another revision of a patch to store duplicates more
efficiently in B-trees.

Anastasia Lubennikova sent in another revision of a patch to overwrite the
lastright item with highkey.

Alexander Korotkov sent in another revision of a patch to ensure that file
renames are atomic on Windows.

Joshua Drake sent in a patch to clean up intro.sgml.

Dmitry Igrishin sent in four revisions of a patch to fix Windows builds.

Amit Langote and Álvaro Herrera traded patches to improve
RelOptInfo.partition_qual usage, and improve constraint exclusion usage in
partprune.c.

Konstantin Knizhnik sent in two more revisions of a patch to implement global
temporary tables.

Álvaro Herrera sent in a patch to remove 'msg' param from convert_tuples_by_name
and friends.

Pavel Stěhule  sent in another revision of a patch to add type info support
functions for functions that use "any" type.

Peter Eisentraut sent in a patch to support Unix domain sockets on Windows.

Melanie Plageman sent in another revision of a patch to test additional
speculative conflict scenarios.

Álvaro Herrera sent in another revision of a patch to fix a duplicated LSN in
ReorderBuffer.

Tom Lane sent in another revision of a patch to avoid GIN full scan for empty
ALL keys.

Konstantin Knizhnik sent in another revision of a patch to implement a built-in
connection pooler.

Álvaro Herrera sent in another revision of a patch to mention the fact that
there is no default partition for hash-partitioned tables.

Michaël Paquier sent in two revisions of a patch to fix a regression test
failure in regression test temp.sql.

Vigneshwaran C sent in another revision of a patch to fix an issue that caused a
segmentation fault in rescan.

Markus Winand and Alexander Korotkov traded patches to use Unicode codepoint
collation in jsonpath.

Thomas Munro, Amit Kapila, and Robert Haas traded patches to clean up orphaned
files using undo logs.

Peter Eisentraut sent in a patch to remove obsolete locale handling in initdb.

Amul Sul sent in a patch to ensure that all memory is freed at the exit of
RelationBuildPartitionDesc().

Tom Lane sent in a patch to implement aforeach().

Tom Lane sent in a patch to remove simple_rte_array in favor of just accessing
the query's rtable directly.

Tom Lane sent in a patch to remove es_range_table_array.

Jeff Davis sent in a patch to add a password_protocol connection parameter to
libpq.

Pavel Stěhule sent in another revision of a patch to implement schema variables.