Extending amcheck to check toast size and compression

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Extending amcheck to check toast size and compression

Mark Dilger-5
Hackers,

During the version 14 development period, a few checks of toasted attributes were written but never committed.  For the version 15 development cycle, I'd like to consider extending the checks of toasted attributes.  First, no toasted attribute should ever have a rawsize larger than the 1GB varlena limit.  Second, no compressed toasted attribute should have an extsize indicating that the toast expanded during toasting.  Such a extsize could mean the compression code is malfunctioning, or that the extsize or rawsize fields are corrupt.  Third, any compressed attribute should have a valid compression method ID.

These checks are cheap.  Actually retrieving the compressed toasted data and checking that it uncompresses correctly would have very different performance implications, but that is not included in this patch.





Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company




v1-0001-Adding-more-toast-pointer-checks-to-amcheck.patch (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Extending amcheck to check toast size and compression

Justin Pryzby
+       /* Oversized toasted attributes should never be stored */
+       if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
+               report_corruption(ctx,
+                                                 psprintf("toast value %u rawsize %u exceeds limit %u",
+                                                                  toast_pointer.va_valueid,
+                                                                  toast_pointer.va_rawsize,
+                                                                  VARLENA_SIZE_LIMIT));
+

I think the comment sounds wrong since toast is precisely for storage of
"oversized" attributes.

https://www.postgresql.org/docs/current/storage-toast.html
| This section provides an overview of TOAST (The Oversized-Attribute Storage Technique).

--
Justin


Reply | Threaded
Open this post in threaded view
|

Re: Extending amcheck to check toast size and compression

Mark Dilger-5


> On May 4, 2021, at 9:43 AM, Justin Pryzby <[hidden email]> wrote:
>
> +       /* Oversized toasted attributes should never be stored */
> +       if (toast_pointer.va_rawsize > VARLENA_SIZE_LIMIT)
> +               report_corruption(ctx,
> +                                                 psprintf("toast value %u rawsize %u exceeds limit %u",
> +                                                                  toast_pointer.va_valueid,
> +                                                                  toast_pointer.va_rawsize,
> +                                                                  VARLENA_SIZE_LIMIT));
> +
>
> I think the comment sounds wrong since toast is precisely for storage of
> "oversized" attributes.
>
> https://www.postgresql.org/docs/current/storage-toast.html
> | This section provides an overview of TOAST (The Oversized-Attribute Storage Technique).
Thanks for reviewing!  Changed to:

+   /* Toasted attributes too large to be untoasted should never be stored */





Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company




v2-0001-Adding-more-toast-pointer-checks-to-amcheck.patch (3K) Download Attachment