Streaming replication upgrade sanity check

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Streaming replication upgrade sanity check

tsuraan
I have several streaming replication pairs running under postgres 10,
and I'm looking at upgrading them to postgres 13. Doing that on the
active side is pretty quick and easy: just run pg_upgrade once for
each of versions 11, 12, and 13, then start things up again. That's
tested (on non-replicated instances) and seems to be working great.
The standby side looks a little bit more interesting, so I wanted some
feedback about whether my approach looks safe.

It looks like the most correct approach would be to discard the
standby systems and just do a new pg_basebackup, but quite a few of
the pairs have a standby on AWS and limited upstream bandwidth, so the
basebackup can take several days. I'd like to avoid that if I can.

So, my thought is to just upgrade the active side as normal, and once
that's done and the master is back in use, stop the standby's
postgres, call "pg_start_backup(...)" on the master, rsync the changed
files from the running master's data dir to the standby's 10.x data
directory, call "pg_stop_backup()", migrate the recovery.conf into a
postgresql.auto.conf, touch the standby.signal file, and then start
the new postgres 13 on the standby. That seems to work, but I want to
be sure I'm not just having good luck due to the relatively low
activity on my test systems.

I use named replication slots with my streaming replication, so I
think that the rsync'd pg_wal directory should have everything I need
(or maybe this isn't guaranteed?), and the pg_start/stop_backup will
ensure that all the damage done by rsync'ing an active database can be
repaired from the pg_wal files that rsync fetches. Are those
assumptions right, or am I missing anything else? Is this idea
workable at all?

Thanks!


Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

Bruce Momjian
On Sat, Feb 13, 2021 at 10:02:20AM -0600, tsuraan wrote:

> I have several streaming replication pairs running under postgres 10,
> and I'm looking at upgrading them to postgres 13. Doing that on the
> active side is pretty quick and easy: just run pg_upgrade once for
> each of versions 11, 12, and 13, then start things up again. That's
> tested (on non-replicated instances) and seems to be working great.
> The standby side looks a little bit more interesting, so I wanted some
> feedback about whether my approach looks safe.
>
> It looks like the most correct approach would be to discard the
> standby systems and just do a new pg_basebackup, but quite a few of
> the pairs have a standby on AWS and limited upstream bandwidth, so the
> basebackup can take several days. I'd like to avoid that if I can.
>
> So, my thought is to just upgrade the active side as normal, and once
> that's done and the master is back in use, stop the standby's
> postgres, call "pg_start_backup(...)" on the master, rsync the changed
> files from the running master's data dir to the standby's 10.x data
> directory, call "pg_stop_backup()", migrate the recovery.conf into a
> postgresql.auto.conf, touch the standby.signal file, and then start
> the new postgres 13 on the standby. That seems to work, but I want to
> be sure I'm not just having good luck due to the relatively low
> activity on my test systems.
>
> I use named replication slots with my streaming replication, so I
> think that the rsync'd pg_wal directory should have everything I need
> (or maybe this isn't guaranteed?), and the pg_start/stop_backup will
> ensure that all the damage done by rsync'ing an active database can be
> repaired from the pg_wal files that rsync fetches. Are those
> assumptions right, or am I missing anything else? Is this idea
> workable at all?

Did you see the pg_upgrade instructions on upgrading standby servers?
Why are you not using that?

--
  Bruce Momjian  <[hidden email]>        https://momjian.us
  EDB                                      https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee



Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

tsuraan
> > It looks like the most correct approach would be to discard the
> > standby systems and just do a new pg_basebackup, but quite a few of
> > the pairs have a standby on AWS and limited upstream bandwidth, so the
> > basebackup can take several days. I'd like to avoid that if I can.
> >
> > So, my thought is to just upgrade the active side as normal, and once
> > that's done and the master is back in use, stop the standby's
> > postgres, call "pg_start_backup(...)" on the master, rsync the changed
> > files from the running master's data dir to the standby's 10.x data
> > directory, call "pg_stop_backup()", migrate the recovery.conf into a
> > postgresql.auto.conf, touch the standby.signal file, and then start
> > the new postgres 13 on the standby. That seems to work, but I want to
> > be sure I'm not just having good luck due to the relatively low
> > activity on my test systems.
> >
> > I use named replication slots with my streaming replication, so I
> > think that the rsync'd pg_wal directory should have everything I need
> > (or maybe this isn't guaranteed?), and the pg_start/stop_backup will
> > ensure that all the damage done by rsync'ing an active database can be
> > repaired from the pg_wal files that rsync fetches. Are those
> > assumptions right, or am I missing anything else? Is this idea
> > workable at all?
>
> Did you see the pg_upgrade instructions on upgrading standby servers?
> Why are you not using that?

Specifically referring to [0], the part I can't do (and that I
unfortunately left out of my initial email) is right at the beginning,
"Do not start any servers yet". The master needs to get running again
as soon as it's upgraded, and the standby systems can't stop the
master for their upgrades. I believe what I've described is
essentially the documented approach, but with a running master and
then wrapping the rsync command in pg_start_backup/pg_stop_backup
calls, but is it still a correct way to upgrade? From the
pg_start_backup description on the binary replication tutorial [1], it
looks like it should be, but I'd like to know if I'm missing anything.

0 - <https://www.postgresql.org/docs/current/pgupgrade.html#PGUPGRADE-STEP-REPLICAS>
1 - <https://wiki.postgresql.org/wiki/Binary_Replication_Tutorial#Cloning_a_Snapshot_of_the_Master>


Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

Bruce Momjian
On Thu, Mar 11, 2021 at 03:53:07PM -0600, tsuraan wrote:
> > Did you see the pg_upgrade instructions on upgrading standby servers?
> > Why are you not using that?
>
> Specifically referring to [0], the part I can't do (and that I

Yes, that is what I was referring to.

> unfortunately left out of my initial email) is right at the beginning,
> "Do not start any servers yet". The master needs to get running again
> as soon as it's upgraded, and the standby systems can't stop the
> master for their upgrades. I believe what I've described is
> essentially the documented approach, but with a running master and
> then wrapping the rsync command in pg_start_backup/pg_stop_backup
> calls, but is it still a correct way to upgrade? From the
> pg_start_backup description on the binary replication tutorial [1], it
> looks like it should be, but I'd like to know if I'm missing anything.

Those documented instructions are very specific about what is possible,
e.g.:

        What this does is to record the links created by pg_upgrade's link mode
        that connect files in the old and new clusters on the primary server. It
        then finds matching files in the standby's old cluster and creates links
        for them in the standby's new cluster. Files that were not linked on the
        primary are copied from the primary to the standby. (They are usually
        small.) This provides rapid standby upgrades. Unfortunately, rsync
        needlessly copies files associated with temporary and unlogged tables
        because these files don't normally exist on standby servers.

Anything that doesn't match this might lead to disaster.  Specifically,
you can't use the simplistic rsync arguments listed there since they
asssume we don't need to check if the files are the same.  When doing
this on a running primary, you effectively will be copying all of the
system tables over, which is fine, but for the user heap/index files, it
will not be able to use links because they can change on a running
system, so you really are doing a streaming backup, and you just happen
to have a cluster there.  I think will just end up copying all the
heap/index files into the empty new cluster directory, so you might as
well just copy the cluster directory in a more simple way.

You really need to backward and look at the file copy requirements for
pg_backup() and see what is possible.  We can discuss what is possible,
but I can't think of any shortcuts given your requirements.  I wonder if
you just need to use logical replication here.

--
  Bruce Momjian  <[hidden email]>        https://momjian.us
  EDB                                      https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee



Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

Stephen Frost
In reply to this post by tsuraan
Greetings,

* tsuraan ([hidden email]) wrote:
> I have several streaming replication pairs running under postgres 10,
> and I'm looking at upgrading them to postgres 13. Doing that on the
> active side is pretty quick and easy: just run pg_upgrade once for
> each of versions 11, 12, and 13, then start things up again. That's
> tested (on non-replicated instances) and seems to be working great.
> The standby side looks a little bit more interesting, so I wanted some
> feedback about whether my approach looks safe.

You don't need to go to 11 or 12 if you want to upgrade to 13- you can
just pg_upgrade directly to 13.

> So, my thought is to just upgrade the active side as normal, and once
> that's done and the master is back in use, stop the standby's
> postgres, call "pg_start_backup(...)" on the master, rsync the changed
> files from the running master's data dir to the standby's 10.x data
> directory, call "pg_stop_backup()", migrate the recovery.conf into a
> postgresql.auto.conf, touch the standby.signal file, and then start
> the new postgres 13 on the standby. That seems to work, but I want to
> be sure I'm not just having good luck due to the relatively low
> activity on my test systems.

Exactly how are you doing this rsync..?  A simplistic rsync would end up
just copying everything over and, as Bruce says later, if you're doing
that then you might as well just use pg_basebackup or a similar tool to
do it cleanly.  In other words, I don't think that you're actually
getting the benefit that you think you are with this..  Note that the
file names are not kept the same after the pg_upgrade...

Also- I don't think you realize how fast the rsync process to update the
standbys will be.  Done correctly, it should be faster than the
pg_upgrade..  Note that you'll want to make sure you TRUNCATE and
unlogged tables before the pg_upgrade, et al, otherwise you'll end up
rsync'ing those over.  Also make sure you haven't got any stray or
forgotten temp files or other things.  Again, done properly, the rsync
to upgrade the standbys should only be copying the catalog tables
themselves and it should be quite fast.

Thanks,

Stephen

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

Bruce Momjian
On Thu, Mar 11, 2021 at 07:50:16PM -0500, Stephen Frost wrote:

> > So, my thought is to just upgrade the active side as normal, and once
> > that's done and the master is back in use, stop the standby's
> > postgres, call "pg_start_backup(...)" on the master, rsync the changed
> > files from the running master's data dir to the standby's 10.x data
> > directory, call "pg_stop_backup()", migrate the recovery.conf into a
> > postgresql.auto.conf, touch the standby.signal file, and then start
> > the new postgres 13 on the standby. That seems to work, but I want to
> > be sure I'm not just having good luck due to the relatively low
> > activity on my test systems.
>
> Exactly how are you doing this rsync..?  A simplistic rsync would end up
> just copying everything over and, as Bruce says later, if you're doing
> that then you might as well just use pg_basebackup or a similar tool to
> do it cleanly.  In other words, I don't think that you're actually
> getting the benefit that you think you are with this..  Note that the
> file names are not kept the same after the pg_upgrade...
>
> Also- I don't think you realize how fast the rsync process to update the
> standbys will be.  Done correctly, it should be faster than the
> pg_upgrade..  Note that you'll want to make sure you TRUNCATE and

He/she wants to run the standby, I guess in read-only mode, while
pg_upgrade is running, which is why I didn't even bother to mention
that.  I think if downtime is the critical for this person, he/she
should be using logical replication for the upgrade.  pg_upgrade really
wants full control of the primary/standbys during its operation, and if
you can't do that, pg_upgrade is not the right tool to use.

> unlogged tables before the pg_upgrade, et al, otherwise you'll end up
> rsync'ing those over.  Also make sure you haven't got any stray or
> forgotten temp files or other things.  Again, done properly, the rsync
> to upgrade the standbys should only be copying the catalog tables
> themselves and it should be quite fast.

Yes, good points.

--
  Bruce Momjian  <[hidden email]>        https://momjian.us
  EDB                                      https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee



Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

Stephen Frost
Greetings,

* Bruce Momjian ([hidden email]) wrote:

> On Thu, Mar 11, 2021 at 07:50:16PM -0500, Stephen Frost wrote:
> > > So, my thought is to just upgrade the active side as normal, and once
> > > that's done and the master is back in use, stop the standby's
> > > postgres, call "pg_start_backup(...)" on the master, rsync the changed
> > > files from the running master's data dir to the standby's 10.x data
> > > directory, call "pg_stop_backup()", migrate the recovery.conf into a
> > > postgresql.auto.conf, touch the standby.signal file, and then start
> > > the new postgres 13 on the standby. That seems to work, but I want to
> > > be sure I'm not just having good luck due to the relatively low
> > > activity on my test systems.
> >
> > Exactly how are you doing this rsync..?  A simplistic rsync would end up
> > just copying everything over and, as Bruce says later, if you're doing
> > that then you might as well just use pg_basebackup or a similar tool to
> > do it cleanly.  In other words, I don't think that you're actually
> > getting the benefit that you think you are with this..  Note that the
> > file names are not kept the same after the pg_upgrade...
> >
> > Also- I don't think you realize how fast the rsync process to update the
> > standbys will be.  Done correctly, it should be faster than the
> > pg_upgrade..  Note that you'll want to make sure you TRUNCATE and
>
> He/she wants to run the standby, I guess in read-only mode, while
> pg_upgrade is running, which is why I didn't even bother to mention
> that.  I think if downtime is the critical for this person, he/she
> should be using logical replication for the upgrade.  pg_upgrade really
> wants full control of the primary/standbys during its operation, and if
> you can't do that, pg_upgrade is not the right tool to use.
What I typically do in that case is just spin up more replicas and let
those handle the read load while the upgrade happens, and then throw
them away after the upgrade of the primary plus the other standbys is
done.  I do agree that logical replication could be an alternative but
that takes a lot longer and puts a number of constraints on what you can
do while it's going on (DDL changes and such have to be done carefully,
etc).

> > unlogged tables before the pg_upgrade, et al, otherwise you'll end up
> > rsync'ing those over.  Also make sure you haven't got any stray or
> > forgotten temp files or other things.  Again, done properly, the rsync
> > to upgrade the standbys should only be copying the catalog tables
> > themselves and it should be quite fast.
>
> Yes, good points.

Yeah..  Would really be nice if we had a custom written tool which knew
how to detect unlogged tables and not try to copy them over and such.
Might be able to make something that's faster and less error-prone than
the rsync approach since we know a lot more about the PG data dir than
rsync does.

Thanks,

Stephen

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

Bruce Momjian
On Thu, Mar 11, 2021 at 08:01:24PM -0500, Stephen Frost wrote:

> > He/she wants to run the standby, I guess in read-only mode, while
> > pg_upgrade is running, which is why I didn't even bother to mention
> > that.  I think if downtime is the critical for this person, he/she
> > should be using logical replication for the upgrade.  pg_upgrade really
> > wants full control of the primary/standbys during its operation, and if
> > you can't do that, pg_upgrade is not the right tool to use.
>
> What I typically do in that case is just spin up more replicas and let
> those handle the read load while the upgrade happens, and then throw
> them away after the upgrade of the primary plus the other standbys is
> done.  I do agree that logical replication could be an alternative but
> that takes a lot longer and puts a number of constraints on what you can
> do while it's going on (DDL changes and such have to be done carefully,
> etc).

Agreed.  He/she could use some replicas to take read-only traffic while
the upgrade happens, and use others to do a quick rsync upgrade using
the intructions in the pg_upgrade docs.  I agree logical replication is
a lot more complex.

> > > unlogged tables before the pg_upgrade, et al, otherwise you'll end up
> > > rsync'ing those over.  Also make sure you haven't got any stray or
> > > forgotten temp files or other things.  Again, done properly, the rsync
> > > to upgrade the standbys should only be copying the catalog tables
> > > themselves and it should be quite fast.
> >
> > Yes, good points.
>
> Yeah..  Would really be nice if we had a custom written tool which knew
> how to detect unlogged tables and not try to copy them over and such.
> Might be able to make something that's faster and less error-prone than
> the rsync approach since we know a lot more about the PG data dir than
> rsync does.

True.

--
  Bruce Momjian  <[hidden email]>        https://momjian.us
  EDB                                      https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee



Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

tsuraan
In reply to this post by Stephen Frost
> You don't need to go to 11 or 12 if you want to upgrade to 13- you can
> just pg_upgrade directly to 13.

That's cool. In that case, I assume I could also only have the old
(10.x) and the new 13.2 postgres versions installed? The intermediate
steps of calling pg_upgrade aren't a problem, but if I could skip
shipping out and installing postgres 11 and 12, that would be pretty
nice.

> Exactly how are you doing this rsync..?  A simplistic rsync would end up
> just copying everything over and, as Bruce says later, if you're doing
> that then you might as well just use pg_basebackup or a similar tool to
> do it cleanly.  In other words, I don't think that you're actually
> getting the benefit that you think you are with this..  Note that the
> file names are not kept the same after the pg_upgrade...

The exact rsync flags are "-acz" "--delete" (so, --archive --checksum
--compress --delete). I'm now seeing that I left off --hardlink, which
was definitely a mistake, but I also don't know whether --checksum is
a valid thing to do when the master is active. From the general tone
of this discussion, it may be "fine", but it's not likely to be better
than just doing a pg_basebackup, and of course pg_basebackup is
definitely correct. I want to do things efficiently, but doing them
correctly comes first, so I'll probably just go with a basebackup and
deal with complaints as they come in :)

> Also- I don't think you realize how fast the rsync process to update the
> standbys will be.  Done correctly, it should be faster than the
> pg_upgrade..  Note that you'll want to make sure you TRUNCATE and
> unlogged tables before the pg_upgrade, et al, otherwise you'll end up
> rsync'ing those over.  Also make sure you haven't got any stray or
> forgotten temp files or other things.  Again, done properly, the rsync
> to upgrade the standbys should only be copying the catalog tables
> themselves and it should be quite fast.

Yeah, it's not as much a matter of how long a sync will take, but just
the way the systems fit together. The systems are all customer-owned,
running in customer locations, and we don't actually have a guarantee
that the standby targets are even powered up while the main database
is upgrading. They absolutely should be, but customer installations
can get pretty weird, so I'm going to stick with fully upgrading and
activating the master machines as quickly as possible, and then
dealing with getting the upgrades to the standby systems however I
have to. It's looking like a new pg_basebackup is just the best
approach given what I have to work with, so I'll switch to that unless
somebody stops me :)


Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

tsuraan
In reply to this post by Bruce Momjian
> He/she wants to run the standby, I guess in read-only mode, while
> pg_upgrade is running, which is why I didn't even bother to mention
> that.  I think if downtime is the critical for this person, he/she
> should be using logical replication for the upgrade.  pg_upgrade really
> wants full control of the primary/standbys during its operation, and if
> you can't do that, pg_upgrade is not the right tool to use.

This thread blew up a bit overnight, so I guess I might be getting a
bit repetitive, but it's less about super low downtime and more about
just not having all that much control over everything. The systems are
installed all over the world, and the upgrade process is very solid
and well understood (by my client's devs), but it's pretty reliant on
the main database being running, and also standby systems are fairly
independent, so doing a lot of coordination is somewhere between
complicated and maybe not entirely possible.

I do just want to clarify the bit about pg_upgrade wanting full
control over standbys. I'm looking over the page again, and it does
list some prereqs to make sure things will run smoothly on the
standby, but if I do just abandon the standby systems and do a new
pg_basebackup on them once the master is running again, that's not
going to cause any issues on the master, right? I haven't had any
explosions in testing, but none of my test systems are anywhere near
as large or busy as the larger customers, so I definitely could be
missing things.


Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

Bruce Momjian
On Fri, Mar 12, 2021 at 10:53:11AM -0600, tsuraan wrote:

> > He/she wants to run the standby, I guess in read-only mode, while
> > pg_upgrade is running, which is why I didn't even bother to mention
> > that.  I think if downtime is the critical for this person, he/she
> > should be using logical replication for the upgrade.  pg_upgrade really
> > wants full control of the primary/standbys during its operation, and if
> > you can't do that, pg_upgrade is not the right tool to use.
>
> This thread blew up a bit overnight, so I guess I might be getting a
> bit repetitive, but it's less about super low downtime and more about
> just not having all that much control over everything. The systems are
> installed all over the world, and the upgrade process is very solid
> and well understood (by my client's devs), but it's pretty reliant on
> the main database being running, and also standby systems are fairly
> independent, so doing a lot of coordination is somewhere between
> complicated and maybe not entirely possible.
>
> I do just want to clarify the bit about pg_upgrade wanting full
> control over standbys. I'm looking over the page again, and it does
> list some prereqs to make sure things will run smoothly on the
> standby, but if I do just abandon the standby systems and do a new
> pg_basebackup on them once the master is running again, that's not
> going to cause any issues on the master, right? I haven't had any
> explosions in testing, but none of my test systems are anywhere near
> as large or busy as the larger customers, so I definitely could be
> missing things.

Right, just making new standbys is simple.  The basic idea is that
pg_upgrade will complain about anything that isn't exactly as required,
to avoid invisible upgrade failures.  When you started asking about
exactly how we could stretch pg_upgrade, we started to have to discuss
exactly what we could relax with 100% reliability, and we didn't come up
with many options.  We are happy to continue that discussion but we
don't have any good ideas right now.

--
  Bruce Momjian  <[hidden email]>        https://momjian.us
  EDB                                      https://enterprisedb.com

  The usefulness of a cup is in its emptiness, Bruce Lee



Reply | Threaded
Open this post in threaded view
|

Re: Streaming replication upgrade sanity check

Stephen Frost
In reply to this post by tsuraan
Greetings,

* tsuraan ([hidden email]) wrote:
> > You don't need to go to 11 or 12 if you want to upgrade to 13- you can
> > just pg_upgrade directly to 13.
>
> That's cool. In that case, I assume I could also only have the old
> (10.x) and the new 13.2 postgres versions installed? The intermediate
> steps of calling pg_upgrade aren't a problem, but if I could skip
> shipping out and installing postgres 11 and 12, that would be pretty
> nice.

That's correct.

> > Exactly how are you doing this rsync..?  A simplistic rsync would end up
> > just copying everything over and, as Bruce says later, if you're doing
> > that then you might as well just use pg_basebackup or a similar tool to
> > do it cleanly.  In other words, I don't think that you're actually
> > getting the benefit that you think you are with this..  Note that the
> > file names are not kept the same after the pg_upgrade...
>
> The exact rsync flags are "-acz" "--delete" (so, --archive --checksum
> --compress --delete). I'm now seeing that I left off --hardlink, which
> was definitely a mistake, but I also don't know whether --checksum is
> a valid thing to do when the master is active. From the general tone
> of this discussion, it may be "fine", but it's not likely to be better
> than just doing a pg_basebackup, and of course pg_basebackup is
> definitely correct. I want to do things efficiently, but doing them
> correctly comes first, so I'll probably just go with a basebackup and
> deal with complaints as they come in :)
No, that wouldn't be more efficient than a pg_basebackup since it's just
going to end up copying over the entire cluster.  Using hard-link with
checksum and doing it properly *might* end up being alright and actually
reducing the data transfer but you'd want to make 100% sure that rsync
in that mode would check the files it's creating hard links for with
checksums and transfer any that are different.  Still doesn't address
the unlogged tables and temp files caveat that I previously mentioned.

> > Also- I don't think you realize how fast the rsync process to update the
> > standbys will be.  Done correctly, it should be faster than the
> > pg_upgrade..  Note that you'll want to make sure you TRUNCATE and
> > unlogged tables before the pg_upgrade, et al, otherwise you'll end up
> > rsync'ing those over.  Also make sure you haven't got any stray or
> > forgotten temp files or other things.  Again, done properly, the rsync
> > to upgrade the standbys should only be copying the catalog tables
> > themselves and it should be quite fast.
>
> Yeah, it's not as much a matter of how long a sync will take, but just
> the way the systems fit together. The systems are all customer-owned,
> running in customer locations, and we don't actually have a guarantee
> that the standby targets are even powered up while the main database
> is upgrading. They absolutely should be, but customer installations
> can get pretty weird, so I'm going to stick with fully upgrading and
> activating the master machines as quickly as possible, and then
> dealing with getting the upgrades to the standby systems however I
> have to. It's looking like a new pg_basebackup is just the best
> approach given what I have to work with, so I'll switch to that unless
> somebody stops me :)
That's certainly a simple and safe approach.  The biggest issue with
pg_basebackup is that it's single-threaded and therefore you could end
up CPU bound in rebuilding the replicas.  Maybe that's fine in your
case, but there are alternative solutions which can do parallel backup
and restore if you're looking for something that won't get throttled due
to only being able to use one CPU.

Thanks,

Stephen

signature.asc (836 bytes) Download Attachment