I am using the PostgreSQL PubSub feature with psycopg2 and gevent in the following manner (dbconn is, of course, a psycopg2 connection object on which LISTEN has been called):
while True:
if gevent.select.select([dbconn], [], []) != ([], [], []):
dbconn.poll()
gevent.spawn(process_result,dbconn.notifies)
Which works fine. Now my understanding of how this all works is that when dbconn.poll() is called, it should pull in *all* pending NOTIFYs and append them to the dbconn.notifies object. So lets say I have NOTIFYs coming in at a rate of 1 per second. I would think that if I added a "gevent.sleep(5)" to the end of the above while loop, then each time through I should have 5 notifies in the dbconn.notifies list, since it has been 5 seconds since I last checked. However, that doesn't appear to be the case - rather, no mater how long a sleep I put in, I still only get *one* item in the notifies list, making me think that I am missing data.
Can someone explain why this is? Why am I not getting 5 at a time under that scenario?
In case someone was wondering, the reasoning behind adding the sleep() lies in that process_result function. Due to the looping it contains, it can process, say, 5 data points in one call much more efficiently than it can process those same 5 data points in 5 calls of 1 data point each. So rather than run it every time a data point comes in, I would like to let the data "pile up" as it were for a short period of time before processing. I was thinking I could easily accomplish this by simply waiting some appropriate period of time between poll() ing the database for new NOTIFYs, but apparently that's not working. The select() is there because the data isn't coming in at a regular speed like in my example, but rather it could come in much faster, or not at all for a while, depending on the time of day.
-----------------------------------------------
5245 Airport Industrial Rd
-----------------------------------------------
