Write visibility map during CLUSTER/VACUUM FULL

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Write visibility map during CLUSTER/VACUUM FULL

Alexander Korotkov
Hi!

I found it weird that CLUSTER/VACUUM FULL don't write visibility map.
Attached patch implements writing visibility map in
heapam_relation_copy_for_cluster().

I've studied previous attempt to implement this [1].  The main problem
of that attempt was usage of existing heap_page_is_all_visible() and
visibilitymap_set() functions.  These functions works through buffer
manager, while heap rewriting is made bypass buffer manager.

In my patch visibility map pages are handled in the same way as heap
pages are.  RewriteStateData holds contents of single VM page.  Once
heap page is finished, corresponding bits are set to VM page etc.  VM
pages are flushed one-by-one to smgr.

This patch have to implement its own check if tuple is allvisible.
But it appears to be possible to simplify this check assuming that all
tuples already past HeapTupleSatisfiesVacuum(), which sets hint bits.

Links
1. https://www.postgresql.org/message-id/flat/20131203162556.GC27105%40momjian.us#90e4a6e77d92076650dcf1d96b32ba38

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

0001-write-vm-during-cluster-1.patch (12K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Write visibility map during CLUSTER/VACUUM FULL

Alexander Korotkov
On Sun, Sep 1, 2019 at 11:07 AM Alexander Korotkov
<[hidden email]> wrote:
> This patch have to implement its own check if tuple is allvisible.
> But it appears to be possible to simplify this check assuming that all
> tuples already past HeapTupleSatisfiesVacuum(), which sets hint bits.

Forgot to check tuple xmin against oldest xmin.  Fixed.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

0001-write-vm-during-cluster-2.patch (12K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Write visibility map during CLUSTER/VACUUM FULL

akapila
In reply to this post by Alexander Korotkov
On Sun, Sep 1, 2019 at 1:37 PM Alexander Korotkov
<[hidden email]> wrote:

>
> Hi!
>
> I found it weird that CLUSTER/VACUUM FULL don't write visibility map.
> Attached patch implements writing visibility map in
> heapam_relation_copy_for_cluster().
>
> I've studied previous attempt to implement this [1].  The main problem
> of that attempt was usage of existing heap_page_is_all_visible() and
> visibilitymap_set() functions.  These functions works through buffer
> manager, while heap rewriting is made bypass buffer manager.
>
> In my patch visibility map pages are handled in the same way as heap
> pages are.
>

I haven't studied this patch in detail, but while glancing I observed
that this doesn't try to sync the vm pages as we do for heap pages in
the end (during end_heap_rewrite).  Am I missing something?

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


Reply | Threaded
Open this post in threaded view
|

Re: Write visibility map during CLUSTER/VACUUM FULL

Alexander Korotkov
On Wed, Sep 11, 2019 at 3:30 PM Amit Kapila <[hidden email]> wrote:

> On Sun, Sep 1, 2019 at 1:37 PM Alexander Korotkov
> <[hidden email]> wrote:
> > I found it weird that CLUSTER/VACUUM FULL don't write visibility map.
> > Attached patch implements writing visibility map in
> > heapam_relation_copy_for_cluster().
> >
> > I've studied previous attempt to implement this [1].  The main problem
> > of that attempt was usage of existing heap_page_is_all_visible() and
> > visibilitymap_set() functions.  These functions works through buffer
> > manager, while heap rewriting is made bypass buffer manager.
> >
> > In my patch visibility map pages are handled in the same way as heap
> > pages are.
> >
>
> I haven't studied this patch in detail, but while glancing I observed
> that this doesn't try to sync the vm pages as we do for heap pages in
> the end (during end_heap_rewrite).  Am I missing something?

You're not missed anything.  Yes, VM need sync.  Will fix this.  And I
just noticed I need a closer look to what is going on with TOAST.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company


Reply | Threaded
Open this post in threaded view
|

Re: Write visibility map during CLUSTER/VACUUM FULL

Alexander Korotkov
On Thu, Sep 12, 2019 at 4:55 PM Alexander Korotkov
<[hidden email]> wrote:

> On Wed, Sep 11, 2019 at 3:30 PM Amit Kapila <[hidden email]> wrote:
> > On Sun, Sep 1, 2019 at 1:37 PM Alexander Korotkov
> > <[hidden email]> wrote:
> > > I found it weird that CLUSTER/VACUUM FULL don't write visibility map.
> > > Attached patch implements writing visibility map in
> > > heapam_relation_copy_for_cluster().
> > >
> > > I've studied previous attempt to implement this [1].  The main problem
> > > of that attempt was usage of existing heap_page_is_all_visible() and
> > > visibilitymap_set() functions.  These functions works through buffer
> > > manager, while heap rewriting is made bypass buffer manager.
> > >
> > > In my patch visibility map pages are handled in the same way as heap
> > > pages are.
> > >
> >
> > I haven't studied this patch in detail, but while glancing I observed
> > that this doesn't try to sync the vm pages as we do for heap pages in
> > the end (during end_heap_rewrite).  Am I missing something?
>
> You're not missed anything.  Yes, VM need sync.  Will fix this.  And I
> just noticed I need a closer look to what is going on with TOAST.
Attached patch syncs VM during end_heap_rewrite().

However, VM for TOAST still isn't read.  It appear to be much more
difficult to write VM for TOAST, because it's written by insertion
tuples one-by-one.  Despite it seems to fill TOAST heap pages
sequentially (assuming no FSM exists yet), it's quite hard to handle
page-switching event with reasonable level of abstraction.
Nevertheless, I find this patch useful in current shape.  Even if we
don't write VM for TOAST, it's still useful to do for regular heap.
Additionally, one of key advantages of having VM is index-only scan,
which don't work for TOAST anyway.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

0001-write-vm-during-cluster-3.patch (15K) Download Attachment