Postgres server crash and not starting

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Postgres server crash and not starting

Ishan Joshi-2

Hi Team,

 

I am using Postgresql server v11.4 on RHEL 7. This is my load test environment and I have reach database 1TB of size.

 

Last night the server went down with errors “No Space left on Device” though there is about 2GB space available on FS.

 

2020-03-03 17:11:37.202 IST pocdb schema1 109 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 ERROR:  could not access status of transaction 0

2020-03-03 17:11:37.202 IST pocdb schema1 110 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 DETAIL:  Could not write to file "pg_commit_ts/5198" at offset 90112: No space left on device.

2020-03-03 17:11:37.202 IST pocdb schema1 111 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 STATEMENT:  UPDATE TEST SET STATUS = $1, LAST_UPDATE_DATE = CURRENT_TIMESTAMP WHERE ID = $2 AND STATUS = $4

2020-03-03 17:11:37.203 IST pocdb schema1 197 2020-02-27 15:23:51 IST pid:30398 xid:0 ip:10.236.113.36 ERROR:  could not access status of transaction 0

2020-03-03 17:11:37.203 IST pocdb schema1 198 2020-02-27 15:23:51 IST pi2020-03-03 17:14:29.937 IST   13 2020-02-27 15:15:14 IST pid:17705 xid:0 ip: LOG:  database system is shut down

 

I have tried to restart the postgres services many times but it is now failing with below message

 

2020-03-04 12:31:19.469 IST   1 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  database system shutdown was interrupted; last known up at 2020-03-04 11:49:08 IST

2020-03-04 12:31:19.704 IST   2 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  database system was not properly shut down; automatic recovery in progress

2020-03-04 12:31:19.706 IST   3 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  redo starts at 136E/9E042C78

2020-03-04 12:32:07.312 IST   4 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  invalid record length at 1371/6CF69FD8: wanted 24, got 0

2020-03-04 12:32:07.312 IST   5 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  redo done at 1371/6CF69F88

2020-03-04 12:32:07.312 IST   6 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  last completed transaction was at log time 2020-03-03 17:13:36.683476+02

2020-03-04 12:32:07.319 IST   7 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  checkpoint starting: end-of-recovery immediate

2020-03-04 12:32:07.964 IST   8 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: PANIC:  could not flush dirty data: Input/output error

2020-03-04 12:33:14.712 IST   7 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG:  startup process (PID 21703) was terminated by signal 6: Aborted

2020-03-04 12:33:14.712 IST   8 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG:  aborting startup due to startup process failure

2020-03-04 12:33:16.358 IST   9 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG:  database system is shut down

 

Can you please suggest way to startup the postgres server.

 

Thanks & Regards,

Ishan Joshi

 

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service

Reply | Threaded
Open this post in threaded view
|

Re: Postgres server crash and not starting

Amit jain
Most likely input / output error occurs when disk corruption error exist, check the disk health and make sure it's having available space too.

On Wed, Mar 4, 2020, 4:48 PM Ishan Joshi <[hidden email]> wrote:

Hi Team,

 

I am using Postgresql server v11.4 on RHEL 7. This is my load test environment and I have reach database 1TB of size.

 

Last night the server went down with errors “No Space left on Device” though there is about 2GB space available on FS.

 

2020-03-03 17:11:37.202 IST pocdb schema1 109 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 ERROR:  could not access status of transaction 0

2020-03-03 17:11:37.202 IST pocdb schema1 110 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 DETAIL:  Could not write to file "pg_commit_ts/5198" at offset 90112: No space left on device.

2020-03-03 17:11:37.202 IST pocdb schema1 111 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 STATEMENT:  UPDATE TEST SET STATUS = $1, LAST_UPDATE_DATE = CURRENT_TIMESTAMP WHERE ID = $2 AND STATUS = $4

2020-03-03 17:11:37.203 IST pocdb schema1 197 2020-02-27 15:23:51 IST pid:30398 xid:0 ip:10.236.113.36 ERROR:  could not access status of transaction 0

2020-03-03 17:11:37.203 IST pocdb schema1 198 2020-02-27 15:23:51 IST pi2020-03-03 17:14:29.937 IST   13 2020-02-27 15:15:14 IST pid:17705 xid:0 ip: LOG:  database system is shut down

 

I have tried to restart the postgres services many times but it is now failing with below message

 

2020-03-04 12:31:19.469 IST   1 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  database system shutdown was interrupted; last known up at 2020-03-04 11:49:08 IST

2020-03-04 12:31:19.704 IST   2 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  database system was not properly shut down; automatic recovery in progress

2020-03-04 12:31:19.706 IST   3 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  redo starts at 136E/9E042C78

2020-03-04 12:32:07.312 IST   4 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  invalid record length at 1371/6CF69FD8: wanted 24, got 0

2020-03-04 12:32:07.312 IST   5 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  redo done at 1371/6CF69F88

2020-03-04 12:32:07.312 IST   6 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  last completed transaction was at log time 2020-03-03 17:13:36.683476+02

2020-03-04 12:32:07.319 IST   7 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  checkpoint starting: end-of-recovery immediate

2020-03-04 12:32:07.964 IST   8 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: PANIC:  could not flush dirty data: Input/output error

2020-03-04 12:33:14.712 IST   7 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG:  startup process (PID 21703) was terminated by signal 6: Aborted

2020-03-04 12:33:14.712 IST   8 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG:  aborting startup due to startup process failure

2020-03-04 12:33:16.358 IST   9 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG:  database system is shut down

 

Can you please suggest way to startup the postgres server.

 

Thanks & Regards,

Ishan Joshi

 

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service

Reply | Threaded
Open this post in threaded view
|

RE: Postgres server crash and not starting

Ishan Joshi-2

Hi Amit,

 

Thanks for your response.

 

Space is already there on server as for Data 1 TB space available an for Postgres directries pg_commit, pg_stat,pg_xact etc have 2+ GB space available.

 

You  might be right on disk corruption, I have also have same suspect and checking the same.

 

Thanks & Regards,

Ishan Joshi

 

From: Amit jain <[hidden email]>
Sent: Wednesday, March 4, 2020 4:52 PM
To: Ishan Joshi <[hidden email]>
Cc: [hidden email]
Subject: Re: Postgres server crash and not starting

 

Most likely input / output error occurs when disk corruption error exist, check the disk health and make sure it's having available space too.

 

On Wed, Mar 4, 2020, 4:48 PM Ishan Joshi <[hidden email]> wrote:

Hi Team,

 

I am using Postgresql server v11.4 on RHEL 7. This is my load test environment and I have reach database 1TB of size.

 

Last night the server went down with errors “No Space left on Device” though there is about 2GB space available on FS.

 

2020-03-03 17:11:37.202 IST pocdb schema1 109 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 ERROR:  could not access status of transaction 0

2020-03-03 17:11:37.202 IST pocdb schema1 110 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 DETAIL:  Could not write to file "pg_commit_ts/5198" at offset 90112: No space left on device.

2020-03-03 17:11:37.202 IST pocdb schema1 111 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 STATEMENT:  UPDATE TEST SET STATUS = $1, LAST_UPDATE_DATE = CURRENT_TIMESTAMP WHERE ID = $2 AND STATUS = $4

2020-03-03 17:11:37.203 IST pocdb schema1 197 2020-02-27 15:23:51 IST pid:30398 xid:0 ip:10.236.113.36 ERROR:  could not access status of transaction 0

2020-03-03 17:11:37.203 IST pocdb schema1 198 2020-02-27 15:23:51 IST pi2020-03-03 17:14:29.937 IST   13 2020-02-27 15:15:14 IST pid:17705 xid:0 ip: LOG:  database system is shut down

 

I have tried to restart the postgres services many times but it is now failing with below message

 

2020-03-04 12:31:19.469 IST   1 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  database system shutdown was interrupted; last known up at 2020-03-04 11:49:08 IST

2020-03-04 12:31:19.704 IST   2 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  database system was not properly shut down; automatic recovery in progress

2020-03-04 12:31:19.706 IST   3 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  redo starts at 136E/9E042C78

2020-03-04 12:32:07.312 IST   4 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  invalid record length at 1371/6CF69FD8: wanted 24, got 0

2020-03-04 12:32:07.312 IST   5 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  redo done at 1371/6CF69F88

2020-03-04 12:32:07.312 IST   6 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  last completed transaction was at log time 2020-03-03 17:13:36.683476+02

2020-03-04 12:32:07.319 IST   7 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG:  checkpoint starting: end-of-recovery immediate

2020-03-04 12:32:07.964 IST   8 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: PANIC:  could not flush dirty data: Input/output error

2020-03-04 12:33:14.712 IST   7 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG:  startup process (PID 21703) was terminated by signal 6: Aborted

2020-03-04 12:33:14.712 IST   8 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG:  aborting startup due to startup process failure

2020-03-04 12:33:16.358 IST   9 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG:  database system is shut down

 

Can you please suggest way to startup the postgres server.

 

Thanks & Regards,

Ishan Joshi

 

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service