Undo worker and transaction rollback

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Undo worker and transaction rollback

Dilip Kumar-2
Hello, hackers,

In previous threads[1], we proposed patches for generating and storing
undo records and now for undo-worker and the transaction rollback
stuff. The idea is that undo remains relevant as long as the
transaction is in progress and needs to be removed once it becomes
irrelevant. Specifically, for a  committed transaction, it remains
relevant until the transaction becomes all-visible; zheap will use
this for MVCC purposes.  However, for an aborted transaction, it
remains relevant until the “undo actions" described by the undo
records have been performed.  This patch introduces code to discard
undo when it is no longer needed and reuse the associated storage.
Additionally,  this patch adds code to execute undo actions. Let me
explain the finer details for each of the cases covered,


Rollback mechanism for undo-based storage

When a transaction is aborted/rolled-back, we need to apply the
corresponding undo-actions to complete the rollback. The undo actions
are applied either by the backend or by a dedicated undo worker. This
decision is based on the size of the transaction and a GUC —
rollback_overflow_size. The aborted transactions exceeding
rollback_overflow_size in size along with their database id are pushed
into the hash table, which is scanned by an undo launcher. Further,
undo launcher spawns undo worker per database. Now, it is the job of
undo worker(s) to connect to that database and perform the
undo-actions accordingly. Note that the undo-worker keeps applying the
undo-actions till it gets undo requests for that specific database,
exit otherwise.


Discarding irrelevant undo

As aforementioned, once a transaction becomes all visible or when it
is aborted the respective undo is considered irrelevant. To reuse the
space occupied by these stale undo we remove them and this process of
removing them is called discarding of undo. This is accomplished by a
dedicated background worker namely discard-worker. Additionally, we
maintain a variable named OldestXidHavingUndo which is the transaction
id of the oldest transaction whose undo is not yet discarded.  This
will be required by the zheap for visibility checks and for freezing
the slots of the older transactions but I will not get into its
details for now, as this part will be covered by the zheap storage
engine.

Additionally, we would like to highlight some of the details of the
modified transaction machinery,

- Normally, when rollback or rollback to savepoint requests is fired,
undo actions are applied under the current transaction. But, if there
is an error during the transaction and the user tries to rollback then
undo actions cannot be applied in the same transaction because that
erroneous transaction is aborted. In such a case, a new transaction is
started and undo actions are applied under that fresh transaction.

- For an efficient application of undo actions, we maintain start and
end undo record pointers for each transaction. When adding the
rollback requests to the rollback hash table we only need to push
these undo record pointers. Now, undo worker can directly get to the
undo records using these undo record pointers and apply the respective
undo actions.


This is still a WIP patch and shared to get some early feedback, so
please go ahead and try this out, we are eagerly waiting for your
response. Just an FYI, this work has been extracted from the zheap
branch [2].


This work could not have been in the shape as it is without
valuable inputs from Amit Kapila, be it the design of the feature,
code related to undo action, or his support throughout the project.
Additionally, I’d like to thank Rafia Sabih for working on rollbacks
and discard mechanism of prepared transactions and rollback
hash-table. Certainly, we could not have come this far without the
basic framework of undo-worker which was worked by Mithun. Last but in
no way least, I would like to thank Robert Haas, Thomas Munro and
Andres Freund for design inputs.

[1] https://www.postgresql.org/message-id/CAFiTN-syRxU3jTpkOxHQsEqKC95LGd86JTdZ2stozyXWDUSffg%40mail.gmail.com

[2] https://github.com/EnterpriseDB/zheap

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

0001-Undoworker-and-transaction-rollback.patch (166K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Undo worker and transaction rollback

Dilip Kumar-2
On Thu, Oct 11, 2018 at 11:30 AM Dilip Kumar <[hidden email]> wrote:

>
> Hello, hackers,
>
> In previous threads[1], we proposed patches for generating and storing
> undo records and now for undo-worker and the transaction rollback
> stuff. The idea is that undo remains relevant as long as the
> transaction is in progress and needs to be removed once it becomes
> irrelevant. Specifically, for a  committed transaction, it remains
> relevant until the transaction becomes all-visible; zheap will use
> this for MVCC purposes.  However, for an aborted transaction, it
> remains relevant until the “undo actions" described by the undo
> records have been performed.  This patch introduces code to discard
> undo when it is no longer needed and reuse the associated storage.
> Additionally,  this patch adds code to execute undo actions. Let me
> explain the finer details for each of the cases covered,
Latest rebased patch for undo worker and transaction rollback.  This
patch includes some bug fixes from the main zheap branch and also
include code for executing undo action using RMGR (RMGR related code
is merged from Thomas' patch set for  "POC: Cleaning up orphaned files
using undo logs"[1].  This patch can be applied on top of undolog and
undo-interface patches.

[1] https://www.postgresql.org/message-id/flat/CAEepm=0ULqYgM2aFeOnrx6YrtBg3xUdxALoyCG+XpssKqmezug@...

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

undoworker-transaction-rollback_v2.patch (183K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Undo worker and transaction rollback

Dilip Kumar-2
On Mon, Nov 5, 2018 at 5:10 PM Dilip Kumar <[hidden email]> wrote:

>
> On Thu, Oct 11, 2018 at 11:30 AM Dilip Kumar <[hidden email]> wrote:
> >
> > Hello, hackers,
> >
> > In previous threads[1], we proposed patches for generating and storing
> > undo records and now for undo-worker and the transaction rollback
> > stuff. The idea is that undo remains relevant as long as the
> > transaction is in progress and needs to be removed once it becomes
> > irrelevant. Specifically, for a  committed transaction, it remains
> > relevant until the transaction becomes all-visible; zheap will use
> > this for MVCC purposes.  However, for an aborted transaction, it
> > remains relevant until the “undo actions" described by the undo
> > records have been performed.  This patch introduces code to discard
> > undo when it is no longer needed and reuse the associated storage.
> > Additionally,  this patch adds code to execute undo actions. Let me
> > explain the finer details for each of the cases covered,
>
> Latest rebased patch for undo worker and transaction rollback.  This
> patch includes some bug fixes from the main zheap branch and also
> include code for executing undo action using RMGR (RMGR related code
> is merged from Thomas' patch set for  "POC: Cleaning up orphaned files
> using undo logs"[1].  This patch can be applied on top of undolog and
> undo-interface patches.
>
> [1] https://www.postgresql.org/message-id/flat/CAEepm=0ULqYgM2aFeOnrx6YrtBg3xUdxALoyCG+XpssKqmezug@...
Updated patch, include defect fix from Kuntal posted on [1].

[1] https://www.postgresql.org/message-id/CAGz5QCKpCG6egFAdazC%2BJgyk7YSE1OBN9h-QpwCkg-NnSWN5AQ%40mail.gmail.com

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

undoworker-transaction-rollback-v3.patch (183K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Undo worker and transaction rollback

Dmitry Dolgov
> On Mon, Nov 5, 2018 at 1:32 PM Dilip Kumar <[hidden email]> wrote:
>
> Updated patch, include defect fix from Kuntal posted on [1].
>
> [1] https://www.postgresql.org/message-id/CAGz5QCKpCG6egFAdazC%2BJgyk7YSE1OBN9h-QpwCkg-NnSWN5AQ%40mail.gmail.com

Thanks for working on this feature. Unfortunately, looks like the current
version of patch has some conflicts with the current master, could you please
post an updated version again?

Reply | Threaded
Open this post in threaded view
|

Re: Undo worker and transaction rollback

Dilip Kumar-2
On Fri, 30 Nov 2018, 20:42 Dmitry Dolgov <[hidden email] wrote:
> On Mon, Nov 5, 2018 at 1:32 PM Dilip Kumar <[hidden email]> wrote:
>
> Updated patch, include defect fix from Kuntal posted on [1].
>
> [1] https://www.postgresql.org/message-id/CAGz5QCKpCG6egFAdazC%2BJgyk7YSE1OBN9h-QpwCkg-NnSWN5AQ%40mail.gmail.com

Thanks for working on this feature. Unfortunately, looks like the current
version of patch has some conflicts with the current master, could you please
post an updated version again?

Currently, I am in process rebasing undolog patch set[1]. Post that I will rebase this patch on top of those patch set.

Reply | Threaded
Open this post in threaded view
|

Re: Undo worker and transaction rollback

Dilip Kumar-2
On Mon, Dec 3, 2018 at 6:14 PM Dilip Kumar <[hidden email]> wrote:

>
> On Fri, 30 Nov 2018, 20:42 Dmitry Dolgov <[hidden email] wrote:
>>
>> > On Mon, Nov 5, 2018 at 1:32 PM Dilip Kumar <[hidden email]> wrote:
>> >
>> > Updated patch, include defect fix from Kuntal posted on [1].
>> >
>> > [1] https://www.postgresql.org/message-id/CAGz5QCKpCG6egFAdazC%2BJgyk7YSE1OBN9h-QpwCkg-NnSWN5AQ%40mail.gmail.com
>>
>> Thanks for working on this feature. Unfortunately, looks like the current
>> version of patch has some conflicts with the current master, could you please
>> post an updated version again?
>
>
> Currently, I am in process rebasing undolog patch set[1]. Post that I will rebase this patch on top of those patch set.
>
> [1] https://www.postgresql.org/message-id/CAFiTN-syRxU3jTpkOxHQsEqKC95LGd86JTdZ2stozyXWDUSffg%40mail.gmail.com
Updated patch.  This will apply on top of undolog and undo interface
patch set[1].

[1] https://www.postgresql.org/message-id/CAFiTN-s112o6KLi_gVX%3Db-vO9t_2_ufxxRxG6Ff1iDx5Knvohw%40mail.gmail.com

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

0001-Undo-worker-and-transaction-rollback_v4.patch (184K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Undo worker and transaction rollback

Dilip Kumar-2
On Tue, Dec 4, 2018 at 3:13 PM Dilip Kumar <[hidden email]> wrote:

After the latest change in undo interface patch[1], this patch need to
be rebased.  Attaching the rebased version of the patch.

[1]https://www.postgresql.org/message-id/CAFiTN-tfquvm6tnWFaJNYYz-vSY%3D%2BSQTMv%2BFv1jPozTrW4mtKw%40mail.gmail.com


--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

0001-Undo-worker-and-transaction-rollback_v5.patch (184K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Undo worker and transaction rollback

Dilip Kumar-2
On Mon, Dec 31, 2018 at 11:04 AM Dilip Kumar <[hidden email]> wrote:
On Tue, Dec 4, 2018 at 3:13 PM Dilip Kumar <[hidden email]> wrote:

After the latest change in undo interface patch[1], this patch need to
be rebased.  Attaching the rebased version of the patch.

[1]https://www.postgresql.org/message-id/CAFiTN-tfquvm6tnWFaJNYYz-vSY%3D%2BSQTMv%2BFv1jPozTrW4mtKw%40mail.gmail.com


I have rebased this patch on top of latest version of undolog and undo-interface patch set [1]

These all patch set including uno-log patch applies on commit (be2e329f2e700032df087e08ac2103ba3d1fa811).



--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com

0001-Undo-worker-and-transaction-rollback_v6.patch (184K) Download Attachment