bdr replication breaks down

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

bdr replication breaks down

Daniel Fink (PDF)

Hi all,

 

I have a bdr replication setup that worked fine until recently.

But it seems like one command cannot be replicated, and it breaks down the whole process on all hosts where it is sent to.

I appended part of the log file, it gets repeated over and over.

Can I somehow omit this command and try if the next one works?

 

Best Regards,

Daniel Fink

 

< 2019-02-08 00:00:01.658 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:01.658 CET >CONTEXT:  apply QUEUED_DROP in commit 7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01 (action #2) from node (6208877715678412212,1,2942745)

< 2019-02-08 00:00:01.661 CET >LOG:  worker process: bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1, (PID 27640) exited with exit code 1

< 2019-02-08 00:00:06.666 CET >LOG:  starting background worker process "bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1,"

< 2019-02-08 00:00:07.641 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:07.641 CET >CONTEXT:  apply QUEUED_DROP in commit 7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01 (action #2) from node (6208877715678412212,1,2942745)

< 2019-02-08 00:00:07.644 CET >LOG:  worker process: bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1, (PID 27645) exited with exit code 1

< 2019-02-08 00:00:12.649 CET >LOG:  starting background worker process "bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1,"

< 2019-02-08 00:00:13.633 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:13.633 CET >CONTEXT:  apply QUEUED_DROP in commit 7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01 (action #2) from node (6208877715678412212,1,2942745)

 

DANIEL FINK
Senior Software Engineer

tel (+49) 89.767062.20
fax(+49) 89.767062.11
email [hidden email]

PDF Solutions GmbH 
(
a PDF Solutions Company)
Managing Director: Kimon Michaels
Schwanthalerstr.
10
D-80336 München, Germany  

München HRB 87307
DE 128214899

www.pdf.com

 


This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately permanently delete it and all attachments to it from your systems. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message or any attachments to it. The sender disclaims any liability for such unauthorized use. PLEASE NOTE that all incoming e-mails sent to PDF e-mail accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional e-mails (“spam”). If you have any concerns about this process, please contact us at [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: bdr replication breaks down

Alvaro Aguayo Garcia-Rada
What's the command you are running to trigger such behaviour?

Regards,

Alvaro Aguayo
Jefe de Operaciones
Open Comb Systems E.I.R.L.

(+51-1) 337-7813 Anexo 4002
www.ocs.pe

----- Original Message -----
From: "Daniel Fink (PDF)" <[hidden email]>
To: "pgsql-general" <[hidden email]>
Sent: Monday, 11 February, 2019 05:18:30
Subject: bdr replication breaks down

Hi all,



I have a bdr replication setup that worked fine until recently.

But it seems like one command cannot be replicated, and it breaks down the
whole process on all hosts where it is sent to.

I appended part of the log file, it gets repeated over and over.

Can I somehow omit this command and try if the next one works?



Best Regards,

Daniel Fink



< 2019-02-08 00:00:01.658 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:01.658 CET >CONTEXT:  apply QUEUED_DROP in commit
7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01
(action #2) from node (6208877715678412212,1,2942745)

< 2019-02-08 00:00:01.661 CET >LOG:  worker process: bdr
(6449651545875285115,1,16388,)->bdr (6208877715678412212,1, (PID 27640)
exited with exit code 1

< 2019-02-08 00:00:06.666 CET >LOG:  starting background worker process
"bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1,"

< 2019-02-08 00:00:07.641 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:07.641 CET >CONTEXT:  apply QUEUED_DROP in commit
7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01
(action #2) from node (6208877715678412212,1,2942745)

< 2019-02-08 00:00:07.644 CET >LOG:  worker process: bdr
(6449651545875285115,1,16388,)->bdr (6208877715678412212,1, (PID 27645)
exited with exit code 1

< 2019-02-08 00:00:12.649 CET >LOG:  starting background worker process
"bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1,"

< 2019-02-08 00:00:13.633 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:13.633 CET >CONTEXT:  apply QUEUED_DROP in commit
7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01
(action #2) from node (6208877715678412212,1,2942745)




*DANIEL FINK*

*Senior Software Engineer*

*tel* (+49) 89.767062.20
*fax*(+49) 89.767062.11
email [hidden email]

*PDF Solutions GmbH*
* (**a PDF Solutions Company)*
Managing Director: Kimon Michaels
Schwanthalerstr. 10
D-80336 München, Germany

München HRB 87307
DE 128214899

*www.pdf.com <http://www.pdf.com/>*

--
This message may contain confidential and privileged information. If it has
been sent to you in error, please reply to advise the sender of the error
and then immediately permanently delete it and all attachments to it from
your systems. If you are not the intended recipient, do not read, copy,
disclose or otherwise use this message or any attachments to it. The sender
disclaims any liability for such unauthorized use.  PLEASE NOTE that all
incoming e-mails sent to PDF e-mail accounts will be archived and may be
scanned by us and/or by external service providers to detect and prevent
threats to our systems, investigate illegal or inappropriate behavior,
and/or eliminate unsolicited promotional e-mails (“spam”).  If you have any
concerns about this process, please contact us at [hidden email]
<mailto:[hidden email]>.

Reply | Threaded
Open this post in threaded view
|

RE: bdr replication breaks down

Daniel Fink (PDF)
In reply to this post by Daniel Fink (PDF)

Hi again,

 

So I checked the x_logs and found the following lines.

It seems like it faild at commiting a long transaction…

I am still not sure what the pg_temp30 schema is used for.

But I was able to “fix” it with

select bdr.skip_changes_upto('6208877715678412212', 1, 2942745, '7D/2E314530');

 

I guess the listed transaction is now missing in some of the nodes.

 

rmgr: Standby     len (rec/tot):     28/    60, tx:          0, lsn: 7D/2E313238, prev 7D/2E313208, bkp: 0000, desc: running xacts: nextXid 106791716 latestCompletedXid 106791714 oldestRunningXid 106791715; 1 xacts: 106791715

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313278, prev 7D/2E313238, bkp: 0000, desc: new_cid: rel 1663/2942745/12834; tid 14/1; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E3132C0, prev 7D/2E313278, bkp: 0000, desc: delete: rel 1663/2942745/12834; tid 14/1 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313300, prev 7D/2E3132C0, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/82; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313348, prev 7D/2E313300, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/82 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313388, prev 7D/2E313348, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/81; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E3133D0, prev 7D/2E313388, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/81 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313410, prev 7D/2E3133D0, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/80; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313458, prev 7D/2E313410, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/80 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313498, prev 7D/2E313458, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/79; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E3134E0, prev 7D/2E313498, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/79 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313520, prev 7D/2E3134E0, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/78; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313568, prev 7D/2E313520, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/78 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E3135A8, prev 7D/2E313568, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/77; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E3135F0, prev 7D/2E3135A8, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/77 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313630, prev 7D/2E3135F0, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/76; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313678, prev 7D/2E313630, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/76 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E3136B8, prev 7D/2E313678, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/75; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313700, prev 7D/2E3136B8, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/75 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313740, prev 7D/2E313700, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/74; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313788, prev 7D/2E313740, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/74 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E3137C8, prev 7D/2E313788, bkp: 0000, desc: new_cid: rel 1663/2942745/12747; tid 12/10; cmin: 4294967295, cmax: 1, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313810, prev 7D/2E3137C8, bkp: 0000, desc: delete: rel 1663/2942745/12747; tid 12/10 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313850, prev 7D/2E313810, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/72; cmin: 4294967295, cmax: 1, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313898, prev 7D/2E313850, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/72 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E3138D8, prev 7D/2E313898, bkp: 0000, desc: new_cid: rel 1663/2942745/12747; tid 12/9; cmin: 4294967295, cmax: 2, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313920, prev 7D/2E3138D8, bkp: 0000, desc: delete: rel 1663/2942745/12747; tid 12/9 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313960, prev 7D/2E313920, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/71; cmin: 4294967295, cmax: 2, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E3139A8, prev 7D/2E313960, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/71 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E3139E8, prev 7D/2E3139A8, bkp: 0000, desc: new_cid: rel 1663/2942745/12759; tid 74/33; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313A30, prev 7D/2E3139E8, bkp: 0000, desc: delete: rel 1663/2942745/12759; tid 74/33 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313A70, prev 7D/2E313A30, bkp: 0000, desc: new_cid: rel 1663/2942745/12759; tid 74/34; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313AB8, prev 7D/2E313A70, bkp: 0000, desc: delete: rel 1663/2942745/12759; tid 74/34 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313AF8, prev 7D/2E313AB8, bkp: 0000, desc: new_cid: rel 1663/2942745/12759; tid 74/35; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313B40, prev 7D/2E313AF8, bkp: 0000, desc: delete: rel 1663/2942745/12759; tid 74/35 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313B80, prev 7D/2E313B40, bkp: 0000, desc: new_cid: rel 1663/2942745/12770; tid 14/33; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313BC8, prev 7D/2E313B80, bkp: 0000, desc: delete: rel 1663/2942745/12770; tid 14/33 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313C08, prev 7D/2E313BC8, bkp: 0000, desc: new_cid: rel 1663/2942745/12877; tid 65/73; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313C50, prev 7D/2E313C08, bkp: 0000, desc: delete: rel 1663/2942745/12877; tid 65/73 KEYS_UPDATED

rmgr: Heap2       len (rec/tot):     34/    66, tx:  106791715, lsn: 7D/2E313C90, prev 7D/2E313C50, bkp: 0000, desc: new_cid: rel 1664/0/12906; tid 1/49; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap        len (rec/tot):     26/    58, tx:  106791715, lsn: 7D/2E313CD8, prev 7D/2E313C90, bkp: 0000, desc: delete: rel 1664/0/12906; tid 1/49 KEYS_UPDATED

rmgr: Heap        len (rec/tot):    160/  1660, tx:  106791715, lsn: 7D/2E313D18, prev 7D/2E313CD8, bkp: 1000, desc: insert: rel 1663/2942745/2943410; tid 0/7

rmgr: Transaction len (rec/tot):    352/   384, tx:  106791715, lsn: 7D/2E3143B0, prev 7D/2E313D18, bkp: 0000, desc: commit: 2019-01-28 01:12:34.348577 PST; inval msgs: catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 46 catcache 45 catcache 64 catcache 63 catcache 64 catcache 63 catcache 49 relcache 23783862 snapshot 2608 snapshot 1214 snapshot 2608 snapshot 2608 relcache 23783862 snapshot 2608

rmgr: Standby     len (rec/tot):     44/    76, tx:          0, lsn: 7D/2E314530, prev 7D/2E3143B0, bkp: 0000, desc: nontransactional message with 35 bytes

 

From: Daniel Fink (PDF) [mailto:[hidden email]]
Sent: Monday, February 11, 2019 11:19 AM
To: [hidden email]
Subject: bdr replication breaks down

 

Hi all,

 

I have a bdr replication setup that worked fine until recently.

But it seems like one command cannot be replicated, and it breaks down the whole process on all hosts where it is sent to.

I appended part of the log file, it gets repeated over and over.

Can I somehow omit this command and try if the next one works?

 

Best Regards,

Daniel Fink

 

< 2019-02-08 00:00:01.658 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:01.658 CET >CONTEXT:  apply QUEUED_DROP in commit 7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01 (action #2) from node (6208877715678412212,1,2942745)

< 2019-02-08 00:00:01.661 CET >LOG:  worker process: bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1, (PID 27640) exited with exit code 1

< 2019-02-08 00:00:06.666 CET >LOG:  starting background worker process "bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1,"

< 2019-02-08 00:00:07.641 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:07.641 CET >CONTEXT:  apply QUEUED_DROP in commit 7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01 (action #2) from node (6208877715678412212,1,2942745)

< 2019-02-08 00:00:07.644 CET >LOG:  worker process: bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1, (PID 27645) exited with exit code 1

< 2019-02-08 00:00:12.649 CET >LOG:  starting background worker process "bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1,"

< 2019-02-08 00:00:13.633 CET >ERROR:  schema "pg_temp_30" does not exist

< 2019-02-08 00:00:13.633 CET >CONTEXT:  apply QUEUED_DROP in commit 7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01 (action #2) from node (6208877715678412212,1,2942745)

 

DANIEL FINK
Senior Software Engineer

tel (+49) 89.767062.20
fax(+49) 89.767062.11
email [hidden email]

PDF Solutions GmbH 
(
a PDF Solutions Company)
Managing Director: Kimon Michaels
Schwanthalerstr.
10
D-80336 München, Germany  

München HRB 87307
DE 128214899

www.pdf.com

 


This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately permanently delete it and all attachments to it from your systems. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message or any attachments to it. The sender disclaims any liability for such unauthorized use. PLEASE NOTE that all incoming e-mails sent to PDF e-mail accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional e-mails (“spam”). If you have any concerns about this process, please contact us at [hidden email].