Remove Deprecated Exclusive Backup Mode

classic Classic list List threaded Threaded
69 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Remove Deprecated Exclusive Backup Mode

David Steele
Hackers,

I propose we remove the deprecated exclusive backup mode of
pg_start_backup()/pg_stop_backup() for Postgres 12.

The exclusive backup mode has a serious known issue.  If Postgres
terminates ungracefully during a backup (due to hardware, kernel,
Postgres issues, etc.) then Postgres may refuse to restart.

The reason is that the backup_label file will usually reference a
checkpoint LSN that is older than the WAL available in pg_wal.  Postgres
does emit a helpful error message while PANIC'ing but that's cold
comfort to an admin who must manually intervene to get their cluster
operational again.

The deprecated exclusive mode promises to make a difficult problem
simple but doesn't live up to that promise. That's why it was replaced
externally in 9.6 and why pg_basebackup has not used exclusive backups
since it was introduced in 9.2.

Non-exclusive backups have been available since 9.6 and several
third-party solutions support this mode, in addition to pg_basebackup.

The recently introduced recovery changes will break current automated
solutions so this seems like a good opportunity to make improvements on
the backup side as well.

I'll submit a patch for the 2019-01 commitfest.

Regards,
--
-David
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Michael Paquier-2
On Mon, Nov 26, 2018 at 10:13:34PM -0500, David Steele wrote:
> Non-exclusive backups have been available since 9.6 and several third-party
> solutions support this mode, in addition to pg_basebackup.

I think that two releases is not actually that much time to just nuke
the exclusive backup interface.  I would be fine if the docs show the
deprecation more aggressively using a warning section, and we could add
an explicit WARNING message about the matter directly when calling
pg_start_backup and pg_stop_backup.

My 2c.
--
Michael

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Andres Freund
Hi,

On 2018-11-27 12:20:13 +0900, Michael Paquier wrote:
> On Mon, Nov 26, 2018 at 10:13:34PM -0500, David Steele wrote:
> > Non-exclusive backups have been available since 9.6 and several third-party
> > solutions support this mode, in addition to pg_basebackup.
>
> I think that two releases is not actually that much time to just nuke
> the exclusive backup interface.  I would be fine if the docs show the
> deprecation more aggressively using a warning section, and we could add
> an explicit WARNING message about the matter directly when calling
> pg_start_backup and pg_stop_backup.

That was my gut reaction as well, but I think David's argument about
existing scripts / workflows being broken due the the recovery.conf
is a good reason to be more aggressive here.

Greetings,

Andres Freund

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Robert Haas
In reply to this post by David Steele
On Mon, Nov 26, 2018 at 10:13 PM David Steele <[hidden email]> wrote:
> I propose we remove the deprecated exclusive backup mode of
> pg_start_backup()/pg_stop_backup() for Postgres 12.

-1. I don't have a problem with deprecating exclusive backup mode
eventually, but I don't see any good reason to do it this soon.

It's not like the problems with exclusive backup are so serious that
you can't work around them.  If you know which machine is your master,
you can arrange to remove backup_label on reboot (only) on the master
(only). Sure, a lot of people probably do this wrong, but it's
infeasible to disallow all the things that people might use
incorrectly without making the system useless.

There must be hundreds or thousands of backup scripts written by
individual users that still use exclusive mode floating around out
there.  Forcing all of those to be updated or scrapped will annoy
users to no benefit.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Magnus Hagander-2
In reply to this post by Andres Freund
On Tue, Nov 27, 2018 at 4:46 AM Andres Freund <[hidden email]> wrote:
Hi,

On 2018-11-27 12:20:13 +0900, Michael Paquier wrote:
> On Mon, Nov 26, 2018 at 10:13:34PM -0500, David Steele wrote:
> > Non-exclusive backups have been available since 9.6 and several third-party
> > solutions support this mode, in addition to pg_basebackup.
>
> I think that two releases is not actually that much time to just nuke
> the exclusive backup interface.  I would be fine if the docs show the
> deprecation more aggressively using a warning section, and we could add
> an explicit WARNING message about the matter directly when calling
> pg_start_backup and pg_stop_backup.

That was my gut reaction as well, but I think David's argument about
existing scripts / workflows being broken due the the recovery.conf
is a good reason to be more aggressive here.


Yeah, I'm in the same boat here -- it feels like it's a bit too short, but since we're breaking things for people *anyway*, it's probably better to break both at once than to have all those people have their things broken multiple times.

--
Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

David Steele
In reply to this post by Robert Haas
On 11/26/18 11:04 PM, Robert Haas wrote:

> On Mon, Nov 26, 2018 at 10:13 PM David Steele <[hidden email]> wrote:
>> I propose we remove the deprecated exclusive backup mode of
>> pg_start_backup()/pg_stop_backup() for Postgres 12.
>
> -1. I don't have a problem with deprecating exclusive backup mode
> eventually, but I don't see any good reason to do it this soon.
>
> It's not like the problems with exclusive backup are so serious that
> you can't work around them.  If you know which machine is your master,
> you can arrange to remove backup_label on reboot (only) on the master
> (only).

I do not recommend that users write automated scripts to delete files
out of $PGDATA.  Accidents happen that way.

If this is our recommendation on how the mitigate the problem with
exclusive backup then I think it proves the point that it is broken and
should be removed.

> Sure, a lot of people probably do this wrong, but it's
> infeasible to disallow all the things that people might use
> incorrectly without making the system useless.

But we have already fixed this problem -- it only remains to remove the
old method.

> There must be hundreds or thousands of backup scripts written by
> individual users that still use exclusive mode floating around out
> there.  Forcing all of those to be updated or scrapped will annoy
> users to no benefit.

But we are OK with forcing them to scrap their recovery scripts?

And there is a benefit -- Postgres will be able to restart if it crashes
during a backup.

Regards,
--
-David
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Peter Eisentraut-6
In reply to this post by Andres Freund
On 27/11/2018 04:46, Andres Freund wrote:
> That was my gut reaction as well, but I think David's argument about
> existing scripts / workflows being broken due the the recovery.conf
> is a good reason to be more aggressive here.

But backup scripts are not affected by the recovery.conf changes.

--
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Michael Paquier-2
On Tue, Nov 27, 2018 at 02:21:49PM +0100, Peter Eisentraut wrote:
> On 27/11/2018 04:46, Andres Freund wrote:
>> That was my gut reaction as well, but I think David's argument about
>> existing scripts / workflows being broken due the the recovery.conf
>> is a good reason to be more aggressive here.
>
> But backup scripts are not affected by the recovery.conf changes.

In any of my own backup scripts (yeah!), I don't have any dependency to
that either.  Or perhaps pgBackRest has a dependency in this area?
--
Michael

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Simon Riggs
In reply to this post by David Steele
On Tue, 27 Nov 2018 at 03:13, David Steele <[hidden email]> wrote:
 
The deprecated exclusive mode promises to make a difficult problem
simple but doesn't live up to that promise. That's why it was replaced
externally in 9.6 and why pg_basebackup has not used exclusive backups
since it was introduced in 9.2.

Non-exclusive backups have been available since 9.6 and several
third-party solutions support this mode, in addition to pg_basebackup.

The recently introduced recovery changes will break current automated
solutions so this seems like a good opportunity to make improvements on
the backup side as well.

I'll submit a patch for the 2019-01 commitfest.

-1 for removal, in this release

It's not there because anyone likes it, it's there because removing it is a risk

Recent changes are the reason to keep it, not a reason to remove.

--
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Stephen Frost
In reply to this post by Michael Paquier-2
Greetings,

* Michael Paquier ([hidden email]) wrote:

> On Tue, Nov 27, 2018 at 02:21:49PM +0100, Peter Eisentraut wrote:
> > On 27/11/2018 04:46, Andres Freund wrote:
> >> That was my gut reaction as well, but I think David's argument about
> >> existing scripts / workflows being broken due the the recovery.conf
> >> is a good reason to be more aggressive here.
> >
> > But backup scripts are not affected by the recovery.conf changes.
>
> In any of my own backup scripts (yeah!), I don't have any dependency to
> that either.  Or perhaps pgBackRest has a dependency in this area?
If you don't consider your recovery scripts and your backup scripts to
be related then I've really got to wonder how you're regularly testing
your backups to make sure that they're actually valid.

If you aren't regularly testing your backups then I've got little
sympathy.

To be clear, pgbackrest doesn't have any dependency here- but it, like
all of the other 3rd party backup solutions and any restore solution
that a user has come up with, are going to have to be changed to deal
with the changes in how recovery works, so this is a good time to make
these changes.

Thanks!

Stephen

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Stephen Frost
In reply to this post by Robert Haas
Greetings,

* Robert Haas ([hidden email]) wrote:
> There must be hundreds or thousands of backup scripts written by
> individual users that still use exclusive mode floating around out
> there.  Forcing all of those to be updated or scrapped will annoy
> users to no benefit.

How about automated recovery scripts?

We've decided it's fine to break all of those which exist out there.

I'm concerned, seriously, that people don't have anywhere near the
concern about the recovery side of things as they do about the backup
side of things and that's really concerning.

Thanks!

Stephen

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Peter Eisentraut-6
In reply to this post by Stephen Frost
On 27/11/2018 15:45, Stephen Frost wrote:
>>> But backup scripts are not affected by the recovery.conf changes.
>> In any of my own backup scripts (yeah!), I don't have any dependency to
>> that either.  Or perhaps pgBackRest has a dependency in this area?
> If you don't consider your recovery scripts and your backup scripts to
> be related then I've really got to wonder how you're regularly testing
> your backups to make sure that they're actually valid.

The sort of installations that continue to use the exclusive backup mode
probably have the following tooling: a 20-line shell script to make the
backup and either a 10-line shell script or a similarly sized README or
wiki page to do the recovery.  Changing the latter for the recovery.conf
changes is probably a 3-line change.  Changing the former for the
removal of exclusive backups would require major changes.  (Try writing
a shell script that keeps a psql session open while it takes the backup
from the file system.  It's possible, but it requires significantly more
logic.)

--
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Stephen Frost
Greetings,

* Peter Eisentraut ([hidden email]) wrote:

> On 27/11/2018 15:45, Stephen Frost wrote:
> >>> But backup scripts are not affected by the recovery.conf changes.
> >> In any of my own backup scripts (yeah!), I don't have any dependency to
> >> that either.  Or perhaps pgBackRest has a dependency in this area?
> > If you don't consider your recovery scripts and your backup scripts to
> > be related then I've really got to wonder how you're regularly testing
> > your backups to make sure that they're actually valid.
>
> The sort of installations that continue to use the exclusive backup mode
> probably have the following tooling: a 20-line shell script to make the
> backup and either a 10-line shell script or a similarly sized README or
> wiki page to do the recovery.  Changing the latter for the recovery.conf
> changes is probably a 3-line change.  Changing the former for the
> removal of exclusive backups would require major changes.  (Try writing
> a shell script that keeps a psql session open while it takes the backup
> from the file system.  It's possible, but it requires significantly more
> logic.)
They're also the sort of installations which don't have reliable backups
and don't have any clue about the danger they are in due to the current
bug/issue/whatever we have with exclusive backups.

No, I don't agree that it's sensible to continue to march on as if
nothing is wrong.

Thanks!

Stephen

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Andreas Karlsson
In reply to this post by Stephen Frost
On 11/27/18 3:46 PM, Stephen Frost wrote:
> I'm concerned, seriously, that people don't have anywhere near the
> concern about the recovery side of things as they do about the backup
> side of things and that's really concerning.

I agree with your larger point, but in this case the two breakages do
not seem equal. As far as I gather the removal of recovery.conf will in
practice result in a longer downtime when your recovery scripts breaks
but not any data loss. While if we remove the exclusive backup mode then
some people's backup scripts will break and if they do not properly
monitor their backups they risk data loss.

I think we should use more caution when data loss is at stake rather
than "just" downtime. So personally I am in favor of updating the manual
with warnings (right now it does not even say if exclusive or
non-exclusive is the default) and adding a deprecation warning when
people use the exclusive mode.

Best regards,
Andreas

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Stephen Frost
Greetings,

* Andreas Karlsson ([hidden email]) wrote:

> On 11/27/18 3:46 PM, Stephen Frost wrote:
> >I'm concerned, seriously, that people don't have anywhere near the
> >concern about the recovery side of things as they do about the backup
> >side of things and that's really concerning.
>
> I agree with your larger point, but in this case the two breakages do not
> seem equal. As far as I gather the removal of recovery.conf will in practice
> result in a longer downtime when your recovery scripts breaks but not any
> data loss. While if we remove the exclusive backup mode then some people's
> backup scripts will break and if they do not properly monitor their backups
> they risk data loss.
Let's please try to remember that this is across a major version upgrade
and is removing a broken capability that's already been deprecated for a
couple years.

If they aren't monitoring their backup scripts today, and aren't
regularly performing restores of those backups, they're already risking
data loss.

> I think we should use more caution when data loss is at stake rather than
> "just" downtime. So personally I am in favor of updating the manual with
> warnings (right now it does not even say if exclusive or non-exclusive is
> the default) and adding a deprecation warning when people use the exclusive
> mode.

Waiting isn't going to change any of these factors, nor will throwing
warnings about the exclusive mode if people aren't monitoring their
scripts.

If we really want to keep the exclusive backup mode around, then, as
Magnus said on a nearby thread, it needs to be fixed.  If it's fixed and
just works and everyone's scripts work, then great, then we can
un-deprecate it and move on.

If we aren't going to fix it then let's remove it.

Thanks!

Stephen

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Andreas Karlsson
On 11/27/18 4:11 PM, Stephen Frost wrote:

>> I agree with your larger point, but in this case the two breakages do not
>> seem equal. As far as I gather the removal of recovery.conf will in practice
>> result in a longer downtime when your recovery scripts breaks but not any
>> data loss. While if we remove the exclusive backup mode then some people's
>> backup scripts will break and if they do not properly monitor their backups
>> they risk data loss.
>
> Let's please try to remember that this is across a major version upgrade
> and is removing a broken capability that's already been deprecated for a
> couple years.

I know that and you know that, but even our own manual use the exclusive
mode in some of the examples, e.g: "pg_start_backup('label_goes_here')"[1].

Your point about that people who do not monitor their backups wont
notice warnings is fair, but even so I think we should give people more
time to migrate when even our own documentation isn't always clear about
the exclusive mode being deprecated.

1.
https://www.postgresql.org/docs/11/functions-admin.html#FUNCTIONS-ADMIN-BACKUP

Andreas

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

Peter Eisentraut-6
In reply to this post by Stephen Frost
On 27/11/2018 16:02, Stephen Frost wrote:
> They're also the sort of installations which don't have reliable backups
> and don't have any clue about the danger they are in due to the current
> bug/issue/whatever we have with exclusive backups.
>
> No, I don't agree that it's sensible to continue to march on as if
> nothing is wrong.

It's fair to argue that this facility is broken and needs to be removed.
 But that needs to be its own argument.

The argument in this subthread is that since we're already making
changes in an adjacent area, removing this feature will have a low impact.

--
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

David Steele
In reply to this post by Peter Eisentraut-6
On 11/27/18 9:54 AM, Peter Eisentraut wrote:

> On 27/11/2018 15:45, Stephen Frost wrote:
>>>> But backup scripts are not affected by the recovery.conf changes.
>>> In any of my own backup scripts (yeah!), I don't have any dependency to
>>> that either.  Or perhaps pgBackRest has a dependency in this area?
>> If you don't consider your recovery scripts and your backup scripts to
>> be related then I've really got to wonder how you're regularly testing
>> your backups to make sure that they're actually valid.
>
> The sort of installations that continue to use the exclusive backup mode
> probably have the following tooling: a 20-line shell script to make the
> backup and either a 10-line shell script or a similarly sized README or
> wiki page to do the recovery.  Changing the latter for the recovery.conf
> changes is probably a 3-line change.  

Really?  We have gone from a file that can be safely overwritten
(recovery.conf) to a file that might need to be parsed to figure out if
the settings already exists (postgresql.auto.conf).  Of course, you can
just continue to append to the file but that seems pretty grotty to me.

This will also require changes to all HA solutions or backup solutions
that deal with recovery.  I think you underestimate how big a change
this is.  I've been thinking about how to write code for it and it is
significantly more complex -- also more flexible so I'm happy about that.

> Changing the former for the
> removal of exclusive backups would require major changes.  (Try writing
> a shell script that keeps a psql session open while it takes the backup
> from the file system.  It's possible, but it requires significantly more
> logic.)

We provide pg_basebackup with core and it is a solid tool for doing
backups.  Do we really want to encourage the use of bash scripts to do
what we already have a tool for?  If users want to do something more
complex than pg_basebackup then bash is certainly not the tool for that.

Regards,
--
-David
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

David Steele
In reply to this post by Peter Eisentraut-6
On 11/27/18 10:29 AM, Peter Eisentraut wrote:

> On 27/11/2018 16:02, Stephen Frost wrote:
>> They're also the sort of installations which don't have reliable backups
>> and don't have any clue about the danger they are in due to the current
>> bug/issue/whatever we have with exclusive backups.
>>
>> No, I don't agree that it's sensible to continue to march on as if
>> nothing is wrong.
>
> It's fair to argue that this facility is broken and needs to be removed.
>  But that needs to be its own argument.

I believe I made this argument in the OP.

> The argument in this subthread is that since we're already making
> changes in an adjacent area, removing this feature will have a low impact.

Fair enough, but I think the argument against exclusive backups stands
on its own.  Also, I don't see backup and restore as adjacent.  I see
them as thoroughly intertwined.

--
-David
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Remove Deprecated Exclusive Backup Mode

David Steele
In reply to this post by Simon Riggs
On 11/27/18 8:56 AM, Simon Riggs wrote:

> On Tue, 27 Nov 2018 at 03:13, David Steele <[hidden email]
> <mailto:[hidden email]>> wrote:
>  
>
>     The deprecated exclusive mode promises to make a difficult problem
>     simple but doesn't live up to that promise. That's why it was replaced
>     externally in 9.6 and why pg_basebackup has not used exclusive backups
>     since it was introduced in 9.2.
>
>     Non-exclusive backups have been available since 9.6 and several
>     third-party solutions support this mode, in addition to pg_basebackup.
>
>     The recently introduced recovery changes will break current automated
>     solutions so this seems like a good opportunity to make improvements on
>     the backup side as well.
>
>     I'll submit a patch for the 2019-01 commitfest.
>
>
> -1 for removal, in this release
>
> It's not there because anyone likes it, it's there because removing it
> is a risk
>
> Recent changes are the reason to keep it, not a reason to remove.

How so?

--
-David
[hidden email]

1234