Thread: BDR workers exiting?
I am loading up a 60G database into BDR database and these "ERRORS" are in my logs. Is not normal behavior or is somethinggoing bad. 2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12 09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr(6204748238611542317,1,16494,): apply" 2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"worker process: bdr (6204748238611542317,1,16494,)->bdr(6204748255428234532,1, (PID 30371) exited with exit code 1",,,,,,,,,"" 2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"starting background worker process""bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,"" Steve Pribyl Thanks ________________________________ [http://www.akunacapital.com/images/akuna.png] Steve Pribyl | Senior Systems Engineer Akuna Capital LLC 36 S Wabash, Suite 310 Chicago IL 60603 USA | www.akunacapital.com <http://www.akunacapital.com> p: +1 312 994 4646 | m: | f: +1 312 750 1667 | Steve.Pribyl@akunacapital.com Please consider the environment, before printing this email. This electronic message contains information from Akuna Capital LLC that may be confidential, legally privileged or otherwiseprotected from disclosure. This information is intended for the use of the addressee only and is not offered asinvestment advice to be relied upon for personal or professional use. Additionally, all electronic messages are recordedand stored in compliance pursuant to applicable SEC rules. If you are not the intended recipient, you are herebynotified that any disclosure, copying, distribution, printing or any other use of, or any action in reliance on, thecontents of this electronic message is strictly prohibited. If you have received this communication in error, please notifyus by telephone at (312)994-4640 and destroy the original message.
On 10/12/15 9:37 AM, Steve Pribyl wrote: > I am loading up a 60G database into BDR database and these "ERRORS" are in my logs. Is not normal behavior or is somethinggoing bad. > > 2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12 09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr(6204748238611542317,1,16494,): apply" > 2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"worker process: bdr (6204748238611542317,1,16494,)->bdr(6204748255428234532,1, (PID 30371) exited with exit code 1",,,,,,,,,"" > 2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"starting background workerprocess ""bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,"" Looks like something's going bad, but you need to ask on the BDR mailing list. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com
On 10/12/15 10:14 AM, Jim Nasby wrote: > On 10/12/15 9:37 AM, Steve Pribyl wrote: >> I am loading up a 60G database into BDR database and these "ERRORS" >> are in my logs. Is not normal behavior or is something going bad. >> >> 2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12 >> 09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr >> (6204748238611542317,1,16494,): apply" >> 2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12 >> 08:12:14 CDT,,0,LOG,00000,"worker process: bdr >> (6204748238611542317,1,16494,)->bdr (6204748255428234532,1, (PID >> 30371) exited with exit code 1",,,,,,,,,"" >> 2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12 >> 08:12:14 CDT,,0,LOG,00000,"starting background worker process ""bdr >> (6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,"" > > Looks like something's going bad, but you need to ask on the BDR mailing > list. Nevermind, just discovered there is no separate list. Sorry for the noise. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com
On 2015-10-12 14:37:07 +0000, Steve Pribyl wrote: > I am loading up a 60G database into BDR database and these "ERRORS" are in my logs. Is not normal behavior or is somethinggoing bad. > > 2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12 09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr(6204748238611542317,1,16494,): apply" > 2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"worker process: bdr (6204748238611542317,1,16494,)->bdr(6204748255428234532,1, (PID 30371) exited with exit code 1",,,,,,,,,"" > 2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"starting background workerprocess ""bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,"" There'll possibly be an error message on the other node about ending the connection. Do you use SSL? If so, try disabling renegotiation. Regards, Andres
Yup, there is a disconnect on other side. This disconnect is preceded by this. ERROR,XX000,"invalid memory alloc request size 1073741824",,,,,"slot ""bdr_16494_6204748238611542317_1_16494__"", outputplugin ""bdr"", in the change callback, associated LSN 2/FD250E48",,,,"bdr (6204748238611542317,1,16494,):receive" Steve Pribyl Sr. Systems Engineer steve.pribyl@akunacapital.com Desk: 312-994-4646 ________________________________________ From: Andres Freund <andres@anarazel.de> Sent: Monday, October 12, 2015 11:08 AM To: Steve Pribyl Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] BDR workers exiting? On 2015-10-12 14:37:07 +0000, Steve Pribyl wrote: > I am loading up a 60G database into BDR database and these "ERRORS" are in my logs. Is not normal behavior or is somethinggoing bad. > > 2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12 09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr(6204748238611542317,1,16494,): apply" > 2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"worker process: bdr (6204748238611542317,1,16494,)->bdr(6204748255428234532,1, (PID 30371) exited with exit code 1",,,,,,,,,"" > 2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"starting background workerprocess ""bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,"" There'll possibly be an error message on the other node about ending the connection. Do you use SSL? If so, try disabling renegotiation. Regards, Andres ________________________________ [http://www.akunacapital.com/images/akuna.png] Steve Pribyl | Senior Systems Engineer Akuna Capital LLC 36 S Wabash, Suite 310 Chicago IL 60603 USA | www.akunacapital.com <http://www.akunacapital.com> p: +1 312 994 4646 | m: | f: +1 312 750 1667 | Steve.Pribyl@akunacapital.com Please consider the environment, before printing this email. This electronic message contains information from Akuna Capital LLC that may be confidential, legally privileged or otherwiseprotected from disclosure. This information is intended for the use of the addressee only and is not offered asinvestment advice to be relied upon for personal or professional use. Additionally, all electronic messages are recordedand stored in compliance pursuant to applicable SEC rules. If you are not the intended recipient, you are herebynotified that any disclosure, copying, distribution, printing or any other use of, or any action in reliance on, thecontents of this electronic message is strictly prohibited. If you have received this communication in error, please notifyus by telephone at (312)994-4640 and destroy the original message.
The process used to created this Start with clean db Create host A database with bdr Join host B with dbr Load database using psql < file.sql I was able to get it work if I do the following. Start with clean db Create host A database Load data on host A Join host A to bdr. Join host b to bdr. Glad to have a work around but would like to get to understand the failure. Steve Pribyl ________________________________________ From: Steve Pribyl Sent: Monday, October 12, 2015 11:19 AM To: Andres Freund Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] BDR workers exiting? Yup, there is a disconnect on other side. This disconnect is preceded by this. ERROR,XX000,"invalid memory alloc request size 1073741824",,,,,"slot ""bdr_16494_6204748238611542317_1_16494__"", outputplugin ""bdr"", in the change callback, associated LSN 2/FD250E48",,,,"bdr (6204748238611542317,1,16494,):receive" Steve Pribyl ________________________________________ From: Andres Freund <andres@anarazel.de> Sent: Monday, October 12, 2015 11:08 AM To: Steve Pribyl Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] BDR workers exiting? On 2015-10-12 14:37:07 +0000, Steve Pribyl wrote: > I am loading up a 60G database into BDR database and these "ERRORS" are in my logs. Is not normal behavior or is somethinggoing bad. > > 2015-10-12 09:28:59.389 CDT,,,30371,,561bc17d.76a3,1,,2015-10-12 09:19:41 CDT,5/0,0,ERROR,XX000,"data stream ended",,,,,,,,,"bdr(6204748238611542317,1,16494,): apply" > 2015-10-12 09:28:59.390 CDT,,,12693,,561bb1ae.3195,20,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"worker process: bdr (6204748238611542317,1,16494,)->bdr(6204748255428234532,1, (PID 30371) exited with exit code 1",,,,,,,,,"" > 2015-10-12 09:29:04.395 CDT,,,12693,,561bb1ae.3195,21,,2015-10-12 08:12:14 CDT,,0,LOG,00000,"starting background workerprocess ""bdr (6204748238611542317,1,16494,)->bdr (6204748255428234532,1,""",,,,,,,,,"" There'll possibly be an error message on the other node about ending the connection. Do you use SSL? If so, try disabling renegotiation. Regards, Andres ________________________________ [http://www.akunacapital.com/images/akuna.png] Steve Pribyl | Senior Systems Engineer Akuna Capital LLC 36 S Wabash, Suite 310 Chicago IL 60603 USA | www.akunacapital.com <http://www.akunacapital.com> p: +1 312 994 4646 | m: | f: +1 312 750 1667 | Steve.Pribyl@akunacapital.com Please consider the environment, before printing this email. This electronic message contains information from Akuna Capital LLC that may be confidential, legally privileged or otherwiseprotected from disclosure. This information is intended for the use of the addressee only and is not offered asinvestment advice to be relied upon for personal or professional use. Additionally, all electronic messages are recordedand stored in compliance pursuant to applicable SEC rules. If you are not the intended recipient, you are herebynotified that any disclosure, copying, distribution, printing or any other use of, or any action in reliance on, thecontents of this electronic message is strictly prohibited. If you have received this communication in error, please notifyus by telephone at (312)994-4640 and destroy the original message.
BDR is currently memory-limited for extremely large transactions. At a guess, I'd say one of your big tables is large enough that the logical decoding facility BDR uses can't keep track of the transaction properly. There's no hard limit, it depends on details of the transaction and a number of other variables, but "many tens or hundreds of GB" is generally too much. If I was to load such a big DB, I'd probably do it with ETL tools that could split up the load and do it progressively.