Thread: "snapshot too large" error when initializing logical replication (9.4)
I sometimes get the error "snapshot too large" from my logical replication walsender process when in response to a CREATE_REPLICATION_SLOT. This is in SnapBuildExportSnapshot in snapbuild.c newxcnt is 212 at that point I have max_connections = 200 procArray->maxProcs=212 Should we be testing newxcnt > GetMaxSnapshotXidCount() instead of newxcnt >= GetMaxSnapshotXidCount()
Hi, On 2014-10-25 18:09:36 -0400, Steve Singer wrote: > I sometimes get the error "snapshot too large" from my logical replication > walsender process when in response to a CREATE_REPLICATION_SLOT. Yes. That's possible if 'too much' was going on until a consistent point was reached. I think we can just use a much larger size for the array if necessary. I've attached patch for this. Could you try whether that helps? I don't have a testcase handy that reproduces the problem. > This is in SnapBuildExportSnapshot in snapbuild.c > > newxcnt is 212 at that point > > I have max_connections = 200 > > procArray->maxProcs=212 > > Should we be testing > newxcnt > GetMaxSnapshotXidCount() > > instead of > newxcnt >= GetMaxSnapshotXidCount() It actually looks correct to me new - newxcnt is used as an offset into an array of size GetMaxSnapshotXidCount(). Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Attachment
On 10/28/2014 01:27 PM, Andres Freund wrote: > Hi, > > On 2014-10-25 18:09:36 -0400, Steve Singer wrote: >> I sometimes get the error "snapshot too large" from my logical replication >> walsender process when in response to a CREATE_REPLICATION_SLOT. > Yes. That's possible if 'too much' was going on until a consistent point > was reached. I think we can just use a much larger size for the array > if necessary. > > I've attached patch for this. Could you try whether that helps? I don't > have a testcase handy that reproduces the problem. This patch seems to fix things. I've done numerous runs of the test with I was doing before with your patch applied and don't seem to be having this issue anymore. >> This is in SnapBuildExportSnapshot in snapbuild.c >> >> newxcnt is 212 at that point >> >> I have max_connections = 200 >> >> procArray->maxProcs=212 >> >> Should we be testing >> newxcnt > GetMaxSnapshotXidCount() >> >> instead of >> newxcnt >= GetMaxSnapshotXidCount() > It actually looks correct to me new - newxcnt is used as an offset into > an array of size GetMaxSnapshotXidCount(). > > Greetings, > > Andres Freund > > >
Re: "snapshot too large" error when initializing logical replication (9.4)
From
Alvaro Herrera
Date:
Andres Freund wrote: > Hi, > > On 2014-10-25 18:09:36 -0400, Steve Singer wrote: > > I sometimes get the error "snapshot too large" from my logical replication > > walsender process when in response to a CREATE_REPLICATION_SLOT. > > Yes. That's possible if 'too much' was going on until a consistent point > was reached. I think we can just use a much larger size for the array > if necessary. > > I've attached patch for this. Could you try whether that helps? I don't > have a testcase handy that reproduces the problem. You haven't pushed this, have you? -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > Hi, > > > > On 2014-10-25 18:09:36 -0400, Steve Singer wrote: > > > I sometimes get the error "snapshot too large" from my logical replication > > > walsender process when in response to a CREATE_REPLICATION_SLOT. > > > > Yes. That's possible if 'too much' was going on until a consistent point > > was reached. I think we can just use a much larger size for the array > > if necessary. > > > > I've attached patch for this. Could you try whether that helps? I don't > > have a testcase handy that reproduces the problem. > > You haven't pushed this, have you? No, but it's on my todo list. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
On Mon, Nov 17, 2014 at 03:52:38PM +0100, Andres Freund wrote: > On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote: > > Andres Freund wrote: > > > Hi, > > > > > > On 2014-10-25 18:09:36 -0400, Steve Singer wrote: > > > > I sometimes get the error "snapshot too large" from my logical replication > > > > walsender process when in response to a CREATE_REPLICATION_SLOT. > > > > > > Yes. That's possible if 'too much' was going on until a consistent point > > > was reached. I think we can just use a much larger size for the array > > > if necessary. > > > > > > I've attached patch for this. Could you try whether that helps? I don't > > > have a testcase handy that reproduces the problem. > > > > You haven't pushed this, have you? > > No, but it's on my todo list. Uh, where are we on this? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + Everyone has their own god. +
On 2015-03-20 08:41:38 -0400, Bruce Momjian wrote: > On Mon, Nov 17, 2014 at 03:52:38PM +0100, Andres Freund wrote: > > On 2014-11-17 11:51:54 -0300, Alvaro Herrera wrote: > > > Andres Freund wrote: > > > > Hi, > > > > > > > > On 2014-10-25 18:09:36 -0400, Steve Singer wrote: > > > > > I sometimes get the error "snapshot too large" from my logical replication > > > > > walsender process when in response to a CREATE_REPLICATION_SLOT. > > > > > > > > Yes. That's possible if 'too much' was going on until a consistent point > > > > was reached. I think we can just use a much larger size for the array > > > > if necessary. > > > > > > > > I've attached patch for this. Could you try whether that helps? I don't > > > > have a testcase handy that reproduces the problem. > > > > > > You haven't pushed this, have you? > > > > No, but it's on my todo list. > > Uh, where are we on this? Nowhere, unfortunately. The fix I'd developed isn't actually correct. The problem is that we can only import a transaction with fewer than GetMaxSnapshotXidCount() transactions atm. The only reason it worked in Steve's case is because he probably wasn't importin gthe snapshot in the test case... Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services