Shared memory

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Shared memory

Natarajan R
Why postgres not providing freeing shared memory?
Reply | Threaded
Open this post in threaded view
|

Re: Shared memory

Robert Haas
On Fri, Oct 4, 2019 at 5:51 AM Natarajan R <[hidden email]> wrote:
> Why postgres not providing freeing shared memory?

Because it's intended to be used mostly for data structures that live
for the entire server lifetime.

There are some cases, such as various hash tables, where the number of
entries can grow and shrink over time. It might be useful to return
memory that is freed up when the hash table shrinks to the common
pool, but it would be complex, because then we'd have to keep track of
multiple chunks of freed memory and consolidate adjacent chunks and so
forth. I don't see that we'd be likely to get much benefit from such a
system, since a lot of cases memory fragmentation would prevent us
from getting any real benefit.

If you need a shared data structure that is temporary, you might want
to check out DSM and DSA.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


Reply | Threaded
Open this post in threaded view
|

Re: Shared memory

Craig Ringer-3
In reply to this post by Natarajan R
On Fri, 4 Oct 2019 at 17:51, Natarajan R <[hidden email]> wrote:
Why postgres not providing freeing shared memory?

It does.

You are presumably looking at static shared memory segments which are assigned at server start. Most extensions need to use one of these, of fixed size, for housekeeping and things like storing LWLocks .

But extensions can use DSM or DSA for most of their memory use, giving them flexible storage.

PostgreSQL doesn't have a fully dynamic shared heap with malloc()/free()-like semantics. AFAIK we don't support the use of AllocSet MemoryContext s on DSM-backed shared memory. That might be nice, but it's complicated.

Most extensions use coding patterns like variable length arrays instead.

I strongly advise you to go read the pglogical source code. It demonstrates a lot of the things you will need to understand to write complex extensions that touch multiple databases.

PostgreSQL itself could make this a LOT easier though. pglogical's basic worker and connection management would be considerably simpler if:

- Extensions could add shared catalog relations
- we could access, copy, and struct-embed BackgroundWorkerHandle
- registered bgworkers weren't relaunched after Pg crash-restart (as discussed on prior threads)
- the bgworker registration struct had more than uint32 worth of room for arguments
- extensions could request that Pg maintain extra shmem space for each bgworker for the extension's use (so we didn't have to maintain our own parallel array of shmem entries for each worker and coordinate that with the bgworker's own areas)

I guess I should probably pony up with a patch for some of this....

While I'm at it, there are so many other extension points I wish for, with the high points being:

- Extension-defined syscaches
- Extension-defined locktypes/methods/etc
- More extensible pg_depend
- Hooks in table rewrite

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 2ndQuadrant - PostgreSQL Solutions for the Enterprise