Upgrade streaming replication and log-shipping standby servers

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgrade streaming replication and log-shipping standby servers

Victor Sudakov
Dear Colleagues,

When upgrading to a new major version, pg_upgrade documentation
https://www.postgresql.org/docs/current/pgupgrade.html#PGUPGRADE-STEP-REPLICAS
states that pg_upgrade can be used on a master server. After upgrading
the PostgreSQL version on standbys, however, one must either pull the
data from scratch with pg_basebackup, or run rsync from master to
standby.

Is running pg_upgrade on replicas not supported and why?

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/


Reply | Threaded
Open this post in threaded view
|

Re: Upgrade streaming replication and log-shipping standby servers

Greg Spiegelberg
On Mon, Jun 15, 2020 at 4:46 AM Victor Sudakov <[hidden email]> wrote:
Dear Colleagues,

When upgrading to a new major version, pg_upgrade documentation
https://www.postgresql.org/docs/current/pgupgrade.html#PGUPGRADE-STEP-REPLICAS
states that pg_upgrade can be used on a master server. After upgrading
the PostgreSQL version on standbys, however, one must either pull the
data from scratch with pg_basebackup, or run rsync from master to
standby.

Is running pg_upgrade on replicas not supported and why?


Hi Victor,

I agree it would be extraordinarily convenient if pg_upgrade could operate on standbys.

Via experience and a quick code read of pg_upgrade source confirms that the new, initialized and as yet unmodified cluster is modified in preparation for an upgrade.  Some of those new cluster modifications include execution of analyze, freezing rows, resetting WAL, modification of controldata(?) and restoration of the old schema.  These and others are evident running the utility in verbose mode. Setting aside a standby must be promoted for these modifications, there is no current way to guarantee these modifications will be consistently done on both primary and all standbys. A standby inconsistent with the primary is no standby at all.

Perhaps not the right place but I would argue that IF pg_upgrade could 
1) pause after all these new cluster modifications at or soon after "Restoring database schemas in the new cluster",
2) permit standbys to replicate the new cluster,
3) continue on primary, and
4) run on standbys picking up from where it left off at step 2 perhaps in coordination with primary
then it would be very nice but I'm making assumptions and painting with a wide brush.  :)

-Greg 
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade streaming replication and log-shipping standby servers

Victor Sudakov
Greg Spiegelberg wrote:

>
> >
> > When upgrading to a new major version, pg_upgrade documentation
> >
> > https://www.postgresql.org/docs/current/pgupgrade.html#PGUPGRADE-STEP-REPLICAS
> > states that pg_upgrade can be used on a master server. After upgrading
> > the PostgreSQL version on standbys, however, one must either pull the
> > data from scratch with pg_basebackup, or run rsync from master to
> > standby.
> >
> > Is running pg_upgrade on replicas not supported and why?
> >
> >
> Hi Victor,
>
> I agree it would be extraordinarily convenient if pg_upgrade could operate
> on standbys.
>
> Via experience and a quick code read of pg_upgrade source confirms that the
> new, initialized and as yet unmodified cluster is modified in preparation
> for an upgrade.  Some of those new cluster modifications include execution
> of analyze, freezing rows, resetting WAL, modification of
> controldata(?) and restoration of the old schema.  These and others are
> evident running the utility in verbose mode. Setting aside a standby must
> be promoted for these modifications, there is no current way to
> guarantee these modifications will be consistently done on both primary and
> all standbys. A standby inconsistent with the primary is no standby at all.

Hi Greg,

Thank you for the detailed explanation.

>
> Perhaps not the right place but I would argue that IF pg_upgrade could
> 1) pause after all these new cluster modifications at or soon after
> "Restoring database schemas in the new cluster",
> 2) permit standbys to replicate the new cluster,
> 3) continue on primary, and
> 4) run on standbys picking up from where it left off at step 2 perhaps in
> coordination with primary
> then it would be very nice but I'm making assumptions and painting with a
> wide brush.  :)
>

Sounds very tricky.

--
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
2:5005/49@fidonet http://vas.tomsk.ru/


Reply | Threaded
Open this post in threaded view
|

Re: Upgrade streaming replication and log-shipping standby servers

Bruce Momjian
In reply to this post by Greg Spiegelberg
On Tue, Jun 16, 2020 at 05:48:10AM -0600, Greg Spiegelberg wrote:

> On Mon, Jun 15, 2020 at 4:46 AM Victor Sudakov <[hidden email]> wrote:
>
>     Dear Colleagues,
>
>     When upgrading to a new major version, pg_upgrade documentation
>     https://www.postgresql.org/docs/current/pgupgrade.html#
>     PGUPGRADE-STEP-REPLICAS
>     states that pg_upgrade can be used on a master server. After upgrading
>     the PostgreSQL version on standbys, however, one must either pull the
>     data from scratch with pg_basebackup, or run rsync from master to
>     standby.
>
>     Is running pg_upgrade on replicas not supported and why?
>
>
>
> Hi Victor,
>
> I agree it would be extraordinarily convenient if pg_upgrade could operate on
> standbys.

Uh, pg_upgrade is already documented as being able to upgrade standby
servers:

        https://www.postgresql.org/docs/12/pgupgrade.html
       
        8. Prepare for standby server upgrades

Am I missing something?  The sync is not sync'ing the file contents,
just the hard links.  You can see the details in the docs:

        f. Run rsync
       
        When using link mode, standby servers can be quickly upgraded using
        rsync. To accomplish this, from a directory on the primary server that
        is above the old and new database cluster directories, run this on the
        primary for each standby server:
       
        rsync --archive --delete --hard-links --size-only --no-inc-recursive
        old_cluster new_cluster remote_dir
       
        where old_cluster and new_cluster are relative to the current directory
        on the primary, and remote_dir is above the old and new cluster
        directories on the standby. The directory structure under the specified
        directories on the primary and standbys must match. Consult the rsync
        manual page for details on specifying the remote directory, e.g.
       
        rsync --archive --delete --hard-links --size-only --no-inc-recursive
        /opt/PostgreSQL/9.5 \
              /opt/PostgreSQL/9.6 standby.example.com:/opt/PostgreSQL
       
        You can verify what the command will do using rsync's --dry-run option.
        While rsync must be run on the primary for at least one standby, it is
        possible to run rsync on an upgraded standby to upgrade other standbys,
        as long as the upgraded standby has not been started.
       
        What this does is to record the links created by pg_upgrade's link mode
        that connect files in the old and new clusters on the primary server. It
        then finds matching files in the standby's old cluster and creates links
        for them in the standby's new cluster. Files that were not linked on the
        primary are copied from the primary to the standby. (They are usually
        small.) This provides rapid standby upgrades. Unfortunately, rsync
        needlessly copies files associated with temporary and unlogged tables
        because these files don't normally exist on standby servers.

--
  Bruce Momjian  <[hidden email]>        https://momjian.us
  EnterpriseDB                             https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee



Reply | Threaded
Open this post in threaded view
|

Re: Upgrade streaming replication and log-shipping standby servers

Greg Spiegelberg
On Tue, Jun 16, 2020 at 11:00 AM Bruce Momjian <[hidden email]> wrote:

Uh, pg_upgrade is already documented as being able to upgrade standby
servers:

        https://www.postgresql.org/docs/12/pgupgrade.html


I stand corrected.  Next time I shall RT-Whole-FM.
Thanks Bruce.

-Greg 
Reply | Threaded
Open this post in threaded view
|

Re: Upgrade streaming replication and log-shipping standby servers

Bruce Momjian
On Tue, Jun 16, 2020 at 11:28:27AM -0600, Greg Spiegelberg wrote:

> On Tue, Jun 16, 2020 at 11:00 AM Bruce Momjian <[hidden email]> wrote:
>
>
>     Uh, pg_upgrade is already documented as being able to upgrade standby
>     servers:
>
>             https://www.postgresql.org/docs/12/pgupgrade.html
>
>
>
> I stand corrected.  Next time I shall RT-Whole-FM.

It is a long, complex set of instructions, and people who try to
short-circuit it can end up not using pg_upgrade to its full potential.

--
  Bruce Momjian  <[hidden email]>        https://momjian.us
  EnterpriseDB                             https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee