Thread: Using threads in FDW for read-ahead
I have created a FDW for a storage backend and it is working well, and now I would like to overlap processing with I/O by performing read-ahead. I started by using a thread to do background I/O and this worked, but problems started to arise when I tried to do predicate filtering in the thread.
In particular, it seems as though `check_stack_depth` is built to assume a single threaded environment (`stack_base_ptr` is global).
I'm wondering if there is a solution to this problem, or if there are examples of overlapping tuple I/O and predicate filtering using non-multithreading techniques?
Thanks,
Noah
Noah Watkins <noahwatkins@gmail.com> writes: > I have created a FDW for a storage backend and it is working well, and now > I would like to overlap processing with I/O by performing read-ahead. I > started by using a thread to do background I/O and this worked, but > problems started to arise when I tried to do predicate filtering in the > thread. > In particular, it seems as though `check_stack_depth` is built to assume a > single threaded environment (`stack_base_ptr` is global). That is not even the tip of the iceberg of what will break if you try to use multiple threads in a Postgres backend. It's not supported. You might possibly manage to not break things if you keep the extra threads sufficiently narrowly scoped --- which for starters would include no use of palloc nor elog. Executing query predicates is right out. regards, tom lane
I've done this, but I had to resort to using middleware outside of the FWD.
On Tue, Jun 10, 2014 at 12:10 PM, Noah Watkins <noahwatkins@gmail.com> wrote:
I have created a FDW for a storage backend and it is working well, and now I would like to overlap processing with I/O by performing read-ahead. I started by using a thread to do background I/O and this worked, but problems started to arise when I tried to do predicate filtering in the thread.In particular, it seems as though `check_stack_depth` is built to assume a single threaded environment (`stack_base_ptr` is global).I'm wondering if there is a solution to this problem, or if there are examples of overlapping tuple I/O and predicate filtering using non-multithreading techniques?Thanks,Noah