Thread: Problem with ZPsycopgDA’s pool
Hi, I know that Zope support was dropped from psycopg2 but, as far as I can see on https://pypi.python.org/pypi/ZPsycopgDA/2.4.6 this list is still referenced for ZPsycopgDA. Please tell me if this list is not the right place for questions about ZPsycopgDA (and even better, please tell me if you know a better place for such questions). I’m in the process of switching some applications from very old Zope (version 2.6) with very old psycopg (version 1.1.10) and ZPsycopgDA to much newer Zope, psycopg2 and ZPsycopgDA. Unfortunately, I have experienced some failures which looked somewhat random. After much investigation, the problem seems to occur when 2 clients request a page (that issues SQL requests) simultaneously. Then, all (or most) clients get failures for a few hours. My understading is that the second client that requests the page before the first was served gets a connection from the pool that was not initialized. Because of that, the character encoding is not as expected, and data of type datetime is not converted to the right type, leading to errors in the application. Is that a known issue? I may try to patch this if needed. Cheers, -- Nicolas
On Wed, Apr 16, 2014 at 3:04 PM, Nicolas Boullis <postgresql@ilcode.fr> wrote: > Hi, > > I know that Zope support was dropped from psycopg2 but, as far as I can > see on > https://pypi.python.org/pypi/ZPsycopgDA/2.4.6 > this list is still referenced for ZPsycopgDA. > > Please tell me if this list is not the right place for questions about > ZPsycopgDA (and even better, please tell me if you know a better place > for such questions). Yes, this is the right place > I’m in the process of switching some applications from very old Zope > (version 2.6) with very old psycopg (version 1.1.10) and ZPsycopgDA to > much newer Zope, psycopg2 and ZPsycopgDA. > > Unfortunately, I have experienced some failures which looked somewhat > random. After much investigation, the problem seems to occur when 2 > clients request a page (that issues SQL requests) simultaneously. Then, > all (or most) clients get failures for a few hours. > > My understading is that the second client that requests the page before > the first was served gets a connection from the pool that was not > initialized. Because of that, the character encoding is not as expected, > and data of type datetime is not converted to the right type, leading to > errors in the application. > > Is that a known issue? I may try to patch this if needed. I think it is a known issue. Could be a manifestation of this bug: http://psycopg.lighthouseapp.com/projects/62710/tickets/155 Patches are very welcome: I don't get the Zope model at all and I can't assess if we have ever changed the behaviour on course or if this issue has ever existed. Tests are welcome too, as well as somebody who cares enough about zope to maintain the adapter. Thank you very much. -- Daniele
Hi Daniele, On Wed, Apr 16, 2014 at 03:20:20PM +0100, Daniele Varrazzo wrote: > > I think it is a known issue. Could be a manifestation of this bug: > http://psycopg.lighthouseapp.com/projects/62710/tickets/155 Thanks for pointing this. I had a look at the patch by zagy and it looks much like what I had in mind. I would also have suggested to have a pool per DB instance rather than per dsn, just as he suggested, because a Zope instance may contain several ZPsycopgDA connectors to the same dsn, but with different settings (especially different character encodings). That kind of defeats the point of the connection pool. Ideally, I think the connection should be initialized the first time it is used for a given page. That would be perfecty compatible with a per-dsn pool. But, with my very limited knowledge of Zope internals, I can’t find a way to get such a behaviour. > Patches are very welcome: I don't get the Zope model at all and I > can't assess if we have ever changed the behaviour on course or if > this issue has ever existed. > > Tests are welcome too, as well as somebody who cares enough about zope > to maintain the adapter. Hmmm… As I wrote previously, I have to keep running some old Zope2 applications that user ZPsycopgDA, so I am somewhat concerned. But, to be honnest, I don't really like Zope, and I’m willing to rewrite those applications with a different (Python) framework. So I’ll be happy to help, as long as I am concerned (probably for a few years, I am afraid). But I wouldn’t say that I care much about Zope. Cheers, -- Nicolas Boullis
Hi, On Wed, Apr 16, 2014 at 10:47:21PM +0200, Nicolas Boullis wrote: > > Ideally, I think the connection should be initialized the first time it > is used for a given page. That would be perfecty compatible with a > per-dsn pool. But, with my very limited knowledge of Zope internals, I > can’t find a way to get such a behaviour. I thought a little more about this, and I think I worked out a way to implement this. Here is a quick&dirty patch that does it. I gave it a try and it works for me; at least I cannot reproduce anymore the failures I experienced before. Please tell me what you think of this patch. Cheers, -- Nicolas Boullis
Attachment
Hi, Sorry for asking, but I really need to have this problem solved, so I’d love it if I could receive some feedback about my patch, before I begin using it on a production server. Christian Zagrodnick (zagy) and Wolfgang Eibner, do you read this list? Do you have opinions about that patch? Best regards, -- Nicolas Boullis
On Mon, Apr 28, 2014 at 9:47 AM, Nicolas Boullis <postgresql@ilcode.fr> wrote: > Hi, > > Sorry for asking, but I really need to have this problem solved, so I’d > love it if I could receive some feedback about my patch, before I begin > using it on a production server. > > Christian Zagrodnick (zagy) and Wolfgang Eibner, do you read this list? > Do you have opinions about that patch? I'm not sure they are subscribed to this ML (if they were it would be nice). You may want to poke them via github. -- Daniele