I’ve run into an issue that’s got me stumped
and I would be really grateful for any ideas. We’re deploying
Postgres 11.8 on Kubernetes 1.17.9. The nodes are running RedHat
EL 7.9 with the latest kernel for that distribution.
We create a Kubernetes pod that runs Postgres.
It’s setup with /pgdata going to a PersistentVolumeClaim that’s
thin provisioned with a size set at 500GB. Initially, the
allocated size of the PersistentVolumeClaim’s corresponding disk
file (VMware .VMDK) is around 12GB. However, the size of the .VMDK
keeps increasing at roughly 150-300MB/Hour even though the size of
the /pgdata folder as shown by df is stable around 230MB. To try
to troubleshoot this, I set up Postgres to put the pg_wal folder
on a different partition. That didn’t make any difference.
Replication is turned on for this Postgres
instance, and it was observed that the replica’s .VMDK file grew
at a rate of 25MB/Hour.
I’ve checked to make sure that the partition
is mounted with the DISCARD option. I created a test program that
would allocate disk space, and then free it. I confirmed that when
DISCARD is used for mount, VMware will reclaim the space as
expected. For example, if the .VMDK grows to 20GB during a test,
it shrinks back down to some size like 1GB when the test
To try to sort this out, I’ve done the
Written the test program and confirmed that on the
ESXI Host/VCenter space reclamation works.
Moved the pg_wal directory to another partition.
Changed Auto-Vacuum frequency to run less often (12
Executed “vacuum full”, followed by fstrim
</mount point> from the Kubernetes node. This freed
perhaps 12 MB from the /pgdata folder, and around 19MB from the
Checked the /proc/fd descriptors for the running
connections and confirmed no unusual temp files are opened.
Confirmed the replication is working as expected.
Set archive_mode off to eliminate that as a source
Does anyone have any ideas about what’s
causing this? Is there anything unusual about how Postgres
allocates temporary data files or frees them? I’m really just
grasping. Thanks for looking!