High memory retention by QueryExecutorImpl.pgStream.encoding.decoderArray

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

High memory retention by QueryExecutorImpl.pgStream.encoding.decoderArray

Atul Deopujari-3
Hello
I am trying to figure out reasons for high memory utilization by PgConnection object. My application occasionally runs out of memory and the heap dump shows PgConnection as one of the largest consumers of memory.

The heap dump shows PgConnection having a retained size of 640 MB.
We use a connection pool of 8 connections. Each connection object is retaining about 80 MB. If I drill down into the PgConnection object, I see that QueryExecutorImpl taking almost all memory within which it is the pgStream.encoding.decoderArray taking all the memory.

Not all connection objects are 80 MB all the time. Sometimes the heap dump also shows that some connection objects are as low as 3 MB in size.

I would like to understand under what circumstances can pgStream.encoding.decoderArray retain 80MB. I have checked places in the code and have ensured that we are doing resultSet.close, stmt.close and conn.close wherever we are using them. We do have big objects in the database. I have tried to reproduce this problem using a simple program that queries tuples that are large text, some are even more than 500MB. I haven't been able to see such high memory usage of QueryExecutorImpl in my local setup.

We use the following JDBC driver and the server
  • JDBC driver build number: 42.2.2
  • Server version: 10.4 (Ubuntu 10.4-2.pgdg14.04+1)

Will appreciate if someone can throw light on the reasons for excessive object retention in  QueryExecutorImpl.pgStream.encoding.decoderArray which can help me dig this problem further.

Thanks,
Atul
Reply | Threaded
Open this post in threaded view
|

Re: High memory retention by QueryExecutorImpl.pgStream.encoding.decoderArray

Dave Cramer-8
Hello,

It would be useful if you could try the latest driver as that may have fixed the problem, alternatively it is also what we are developing against.

As you already have a heap dump can you shed any more light on what's going on?

Regards

On Mon, 4 Mar 2019 at 11:01, Atul Deopujari <[hidden email]> wrote:
Hello
I am trying to figure out reasons for high memory utilization by PgConnection object. My application occasionally runs out of memory and the heap dump shows PgConnection as one of the largest consumers of memory.

The heap dump shows PgConnection having a retained size of 640 MB.
We use a connection pool of 8 connections. Each connection object is retaining about 80 MB. If I drill down into the PgConnection object, I see that QueryExecutorImpl taking almost all memory within which it is the pgStream.encoding.decoderArray taking all the memory.

Not all connection objects are 80 MB all the time. Sometimes the heap dump also shows that some connection objects are as low as 3 MB in size.

I would like to understand under what circumstances can pgStream.encoding.decoderArray retain 80MB. I have checked places in the code and have ensured that we are doing resultSet.close, stmt.close and conn.close wherever we are using them. We do have big objects in the database. I have tried to reproduce this problem using a simple program that queries tuples that are large text, some are even more than 500MB. I haven't been able to see such high memory usage of QueryExecutorImpl in my local setup.

We use the following JDBC driver and the server
  • JDBC driver build number: 42.2.2
  • Server version: 10.4 (Ubuntu 10.4-2.pgdg14.04+1)

Will appreciate if someone can throw light on the reasons for excessive object retention in  QueryExecutorImpl.pgStream.encoding.decoderArray which can help me dig this problem further.

Thanks,
Atul
Reply | Threaded
Open this post in threaded view
|

Re: High memory retention by QueryExecutorImpl.pgStream.encoding.decoderArray

Atul Deopujari-3

--
Atul

On Tue, 5 Mar 2019 at 16:30, Dave Cramer <[hidden email]> wrote:
Hello,

It would be useful if you could try the latest driver as that may have fixed the problem, alternatively it is also what we are developing against.

As you already have a heap dump can you shed any more light on what's going on?

Regards

On Mon, 4 Mar 2019 at 11:01, Atul Deopujari <[hidden email]> wrote:
Hello
I am trying to figure out reasons for high memory utilization by PgConnection object. My application occasionally runs out of memory and the heap dump shows PgConnection as one of the largest consumers of memory.

The heap dump shows PgConnection having a retained size of 640 MB.
We use a connection pool of 8 connections. Each connection object is retaining about 80 MB. If I drill down into the PgConnection object, I see that QueryExecutorImpl taking almost all memory within which it is the pgStream.encoding.decoderArray taking all the memory.

Not all connection objects are 80 MB all the time. Sometimes the heap dump also shows that some connection objects are as low as 3 MB in size.

I would like to understand under what circumstances can pgStream.encoding.decoderArray retain 80MB. I have checked places in the code and have ensured that we are doing resultSet.close, stmt.close and conn.close wherever we are using them. We do have big objects in the database. I have tried to reproduce this problem using a simple program that queries tuples that are large text, some are even more than 500MB. I haven't been able to see such high memory usage of QueryExecutorImpl in my local setup.

We use the following JDBC driver and the server
  • JDBC driver build number: 42.2.2
  • Server version: 10.4 (Ubuntu 10.4-2.pgdg14.04+1)

Will appreciate if someone can throw light on the reasons for excessive object retention in  QueryExecutorImpl.pgStream.encoding.decoderArray which can help me dig this problem further.

Thanks,
Atul