FATAL: terminating walreceiver process due to administrator command

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

FATAL: terminating walreceiver process due to administrator command

Maeldron T.
Hello,

Today, I received email notifications from a server telling me the replication was lagging.

The application is monitoring the delay. In the past, it happened a few times I received this notification during a high load but not from this server.

However, the replication was not lagging. It stopped.

There was a single line in the log of the standby server:

FATAL:  terminating walreceiver process due to administrator command

I did not find anything related in the master’s log. The rest of the log was all about the slow statements.

As far I can tell you, this never happened before. The standby server was running but the replication stopped. I restarted the server after 42 minutes of that line in the log. The replication caught up in 3-5 seconds.

Only I have access to the servers.

I did not stop the replication process. I don’t even know how to do it.

There is no cron task that would do such thing. Only one application access the database. I wrote it hence I know it didn’t do it either.

I have been running the servers for years with more or less the same configuration.

As far as I see, when I see the same line in earlier logs, the database was shut down as well. This was the only lonely line like that.

Recent changes on the servers:

* On 11 January, I upgraded from 10.5 to 10.6_2

* A few days ago, set up a new server that replicates one table from the same master. This is a huge table but it’s rarely written. The replication works. It’s the same time I used logical replication. The server where the replication stopped uses async stream replication.

* When I set up the logical replication, I increased the wal_sender_timeout 

I found nothing related in the logs (/var/log/messages, /var/log/all.log, dmesg). This slave is probably the least loaded server of the group.

FreeBSD xxx 11.2-RELEASE-p8 FreeBSD 11.2-RELEASE-p8 #0: Tue Jan  8 21:35:12 UTC 2019     [hidden email]:/usr/obj/usr/src/sys/GENERIC  amd64

/boot/loader.conf:
# PostgreSQL
kern.ipc.semmni=256
kern.ipc.semmns=512
kern.ipc.semmnu=256

Everything else is either FreeBSD default or unrelated.

There is a lot of free memory. I don’t mean usable but free. 3GB RAM was not even touched since the last boot.

M.



Reply | Threaded
Open this post in threaded view
|

Re: FATAL: terminating walreceiver process due to administrator command

Michael Paquier-2
On Fri, Feb 01, 2019 at 03:32:29PM +0100, Maeldron T. wrote:
> There was a single line in the log of the standby server:
>
> FATAL:  terminating walreceiver process due to administrator command

This message shows up when the WAL receiver receives SIGTERM for
termination.  Are you sure that you do not have some tooling which
could have sent that to the WAL receiver, stopping it?
--
Michael

signature.asc (849 bytes) Download Attachment