Postgres takes more than 6 minutes to come up during host/standby switch over

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Postgres takes more than 6 minutes to come up during host/standby switch over

RideNext

Hello!

 

We are using postgres version 9.2.4

 

Our system works in hot/standby mode. There were multiple table with multiple entries in the postgres sql database. During high load transaction, when hot/standby switchover happened. At times it took more than 6 minutes to come up in the standby.

 

Can anyone help to understand the reason why postgres took so much time. Is there anyway to optimise the switchover.

 

If required we can share Postgres logs!

 

Here is the sample logs:

Admin2:

 

postgresql-2019-11-14_000000.log:2019-11-14 01:55:48.456 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_015724.log:2019-11-14 01:57:24.703 GMT LOG:  streaming replication successfully connected to primary

00:02:36:247

 

postgresql-2019-11-14_024053.log:2019-11-14 04:54:03.771 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_050023.log:2019-11-14 05:00:23.346 GMT LOG:  streaming replication successfully connected to primary

00:06:19:575

postgresql-2019-11-14_050023.log:2019-11-14 05:16:41.574 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_052300.log:2019-11-14 05:23:00.681 GMT LOG:  streaming replication successfully connected to primary

00:06:19:107

postgresql-2019-11-14_052300.log:2019-11-14 05:39:10.682 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_054529.log:2019-11-14 05:45:29.030 GMT LOG:  streaming replication successfully connected to primary

00:06:18:348

postgresql-2019-11-14_054529.log:2019-11-14 06:01:37.178 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_060752.log:2019-11-14 06:07:52.763 GMT LOG:  streaming replication successfully connected to primary

00:06:15:585

 

postgresql-2019-11-14_060752.log:2019-11-14 06:24:03.038 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_063019.log:2019-11-14 06:30:19.919 GMT LOG:  streaming replication successfully connected to primary

 

00:06:16:881

 

Admin1:

postgresql-2019-11-14_023941.log:2019-11-14 04:44:33.041 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_044917.log:2019-11-14 04:49:17.444 GMT LOG:  streaming replication successfully connected to primary

00:04:33:403

postgresql-2019-11-14_044917.log:2019-11-14 05:05:22.645 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_051142.log:2019-11-14 05:11:42.154 GMT LOG:  streaming replication successfully connected to primary

00:06:19:509

postgresql-2019-11-14_051142.log:2019-11-14 05:27:57.913 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_053415.log:2019-11-14 05:34:15.305 GMT LOG:  streaming replication successfully connected to primary

00:06:18:392

postgresql-2019-11-14_053415.log:2019-11-14 05:50:24.466 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_055654.log:2019-11-14 05:56:54.619 GMT LOG:  streaming replication successfully connected to primary

00:06:30:153

postgresql-2019-11-14_055654.log:2019-11-14 06:12:49.944 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_061919.log:2019-11-14 06:19:19.795 GMT LOG:  streaming replication successfully connected to primary

00:06:30:851

postgresql-2019-11-14_061919.log:2019-11-14 06:35:15.341 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_064133.log:2019-11-14 06:41:33.872 GMT LOG:  streaming replication successfully connected to primary

00:06:18:531

 

 

Thanks!

 

Reply | Threaded
Open this post in threaded view
|

RE: Postgres takes more than 6 minutes to come up during host/standby switch over

RideNext

Hi Greg and all,

 

Thank you for the quick response and valuable suggestion!

 

We understand for the postgres version 9.2.4, there is no support and we would prefer to upgrade to the current version.

 

So we would like to know if we upgrade to latest version will it be smooth upgrade? Do you suspect any issues during upgrade and factors to be considered? For example any of API name or format would change in the latest version?

 

Kindly suggest and advice

 

Thanks!

 

 

From: Greg Clough <[hidden email]>
Sent: 28 November 2019 19:24
To: RideNext <[hidden email]>
Subject: RE: Postgres takes more than 6 minutes to come up during host/standby switch over

 

I presume you're aware that 9.2.4 is over 6 years out of date, and v9.2.x has been unsupported for 2 years:

https://www.postgresql.org/docs/9.2/release-9-2-4.html

 

Upgrading this to a current version that has security patches should be your first priority... then, after that you can consider reducing the checkpoint_timeout and increasing the checkpoint_completion_target = 0.9.

 

https://www.2ndquadrant.com/en/blog/basics-of-tuning-checkpoints/

 

It could be something else, so check in the postgresql.log to see what's causing the startup to be slow.

 

Regards,

Greg.

 

From: RideNext <[hidden email]>
Sent: 28 November 2019 06:53
To: [hidden email]
Subject: Postgres takes more than 6 minutes to come up during host/standby switch over

 

[CAUTION] EXTERNAL EMAIL ..

Hello!

 

We are using postgres version 9.2.4

 

Our system works in hot/standby mode. There were multiple table with multiple entries in the postgres sql database. During high load transaction, when hot/standby switchover happened. At times it took more than 6 minutes to come up in the standby.

 

Can anyone help to understand the reason why postgres took so much time. Is there anyway to optimise the switchover.

 

If required we can share Postgres logs!

 

Here is the sample logs:

Admin2:

 

postgresql-2019-11-14_000000.log:2019-11-14 01:55:48.456 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_015724.log:2019-11-14 01:57:24.703 GMT LOG:  streaming replication successfully connected to primary

00:02:36:247

 

postgresql-2019-11-14_024053.log:2019-11-14 04:54:03.771 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_050023.log:2019-11-14 05:00:23.346 GMT LOG:  streaming replication successfully connected to primary

00:06:19:575

postgresql-2019-11-14_050023.log:2019-11-14 05:16:41.574 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_052300.log:2019-11-14 05:23:00.681 GMT LOG:  streaming replication successfully connected to primary

00:06:19:107

postgresql-2019-11-14_052300.log:2019-11-14 05:39:10.682 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_054529.log:2019-11-14 05:45:29.030 GMT LOG:  streaming replication successfully connected to primary

00:06:18:348

postgresql-2019-11-14_054529.log:2019-11-14 06:01:37.178 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_060752.log:2019-11-14 06:07:52.763 GMT LOG:  streaming replication successfully connected to primary

00:06:15:585

 

postgresql-2019-11-14_060752.log:2019-11-14 06:24:03.038 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_063019.log:2019-11-14 06:30:19.919 GMT LOG:  streaming replication successfully connected to primary

 

00:06:16:881

 

Admin1:

postgresql-2019-11-14_023941.log:2019-11-14 04:44:33.041 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_044917.log:2019-11-14 04:49:17.444 GMT LOG:  streaming replication successfully connected to primary

00:04:33:403

postgresql-2019-11-14_044917.log:2019-11-14 05:05:22.645 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_051142.log:2019-11-14 05:11:42.154 GMT LOG:  streaming replication successfully connected to primary

00:06:19:509

postgresql-2019-11-14_051142.log:2019-11-14 05:27:57.913 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_053415.log:2019-11-14 05:34:15.305 GMT LOG:  streaming replication successfully connected to primary

00:06:18:392

postgresql-2019-11-14_053415.log:2019-11-14 05:50:24.466 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_055654.log:2019-11-14 05:56:54.619 GMT LOG:  streaming replication successfully connected to primary

00:06:30:153

postgresql-2019-11-14_055654.log:2019-11-14 06:12:49.944 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_061919.log:2019-11-14 06:19:19.795 GMT LOG:  streaming replication successfully connected to primary

00:06:30:851

postgresql-2019-11-14_061919.log:2019-11-14 06:35:15.341 GMT FATAL:  terminating connection due to administrator command

postgresql-2019-11-14_064133.log:2019-11-14 06:41:33.872 GMT LOG:  streaming replication successfully connected to primary

00:06:18:531

 

 

Thanks!

 

 

********* Confidential Disclaimer *********

This e-mail message and any attachments are confidential. Dissemination, distribution or copying of this e-mail or any attachments by anyone other than the intended recipient is prohibited. If you are not the intended recipient, please notify Ipreo immediately by replying to this e-mail, and destroy all copies of this e-mail and any attachments. If you have received this e-mail as part of a marketing communication and you would like to unsubscribe from future marketing communications, please review our privacy policy for more information.

 

 

             

 

 



This e-mail, including accompanying communications and attachments, is strictly confidential and only for the intended recipient. Any retention, use or disclosure not expressly authorised by IHSMarkit is prohibited. This email is subject to all waivers and other terms at the following link: https://ihsmarkit.com/Legal/EmailDisclaimer.html

Please visit www.ihsmarkit.com/about/contact-us.html for contact information on our offices worldwide.