Frequent HLL bitstream_unpack crashes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Frequent HLL bitstream_unpack crashes

Kirk, Steve

Folks,

We’ve been seeing nearly daily crashes from a PostgreSQL 9.6 application that is heavily
dependent on the HLL extension (v 2.10.2). All these crashes are from inside the HLL
bitstream_unpack function. Usually they’re from an INSERT VALUES statement, but
occasionally they are from an hll_cardinality call in a query.

I think I’ve identified the root cause, but I’d like someone who is familiar with the code
in the HLL library to confirm my hypothesis:

    In bitstream_unpack it pulls a full quadword of data out of the bitstream using the

    brc_curp pointer.  Usually this is not a problem.  However, if the brc_curp pointer is

    less than 8 bytes from the end of the bitstream data, then that quadword read is

    reading past the end of the actual bitstream data.  Because of the subsequent bit

    reordering, shifting, and masking this has no effect of the answers.  However, when

    the end of the bitstream is very close to the end of an OS page then the quadword

    read will attempt to read the next OS page, and if that next OS page does not exist

    in this process, then it will SEGV.

 

I posted this as a comment in the HLL GitHub, but have yet to get a response there:

    https://github.com/citusdata/postgresql-hll/issues/84

 

Thanks for any assistance!

 

Reply | Threaded
Open this post in threaded view
|

Re: Frequent HLL bitstream_unpack crashes

akapila
On Fri, Nov 8, 2019 at 8:30 PM Kirk, Steve <[hidden email]> wrote:
>
> I posted this as a comment in the HLL GitHub, but have yet to get a response there:
>
>     https://github.com/citusdata/postgresql-hll/issues/84
>

I don't think this is the right mailing list to expect an answer to
this problem.  This seems to be something related to citusdata's
extension.  Even, if this turns out to be a problem of core Postgres,
it is better to present a test or scenario describing the problem in
Postgres.

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com