Hi all,
I have gone through the code a bit more this time and I think that what
I originally thought of as a task of executor in the beginning (as I
read the documentation) seems to be split at multiple points. So i am
asking a question based on what I have understood so far. Keeping it a
bit organized, I have to say this:
==> What I am planning to do?
I want to parallelize the search / scan (specifically hash) by using
multiple cores in the processors today.
==> What do I know so far?
Till now all that I know seems to be a problem for this design. I know
that
* PG backend is not thread safe. Even we can't create a thread in there.
* For anything to be integrated into the PG as well as be crash safe, it
needs to comply with the rules of the WAL and buffer manager subsystem.
* Almost everything in PG is divided into tables and every table is
organized into pages, all of which have the same structure described in
the "database page layout" Section in the documentation.
* Each page (in case of a table created by the user) will contain
multiple number of entries (at least one). In most cases it will contain
the full data. If it does not, then the storage will be done using
TOAST. [I am considering the aspect which does not involve toast].
* The buffer manager loads pages into the disk and any search is done
using the buffer.
* Any modification done to the buffer is first written to the WAL and
only after WAL is synced to disk will the change take place on the disk
page and the buffer will then be discarded.
==> IS it compatible?
Well, I think that there is a way at hand which can be used to create
some sort of (say) 'magic' processes which can be run in parallel over
the data to search the data. (can't be used for modification, only
search).
Now there are some requirements of these processes.
* First, they cannot be treated as 'processes' as we talk in everyday
terms. We cannot call them 'threads' either. (Hence I call them
'magic'). [Please do not ask how it can be done. All I can say is that
there is a way out] So the OS still is running only one process.
* Second, say if I have 4 processes, then none of them will start unless
each 4 of them have something to do. This means that we need to give
them a batch of work together.
* Third, the sequence of actions in PG backend have to be linear. These
magic processes (all 4 of them) can be launched as if it was a function
call. Something like
launch_magic_search(4, buffers_array_of_4_buffers_in_memory,
additional_parameters);
So this method can/may be used in the backend. These processes still
remain part of that one single backend which executed the function call
and pg backend will not bothered about creation of a child process or a
separate worker thread. It will always appear to be linear.
==> The point of concern:
Since work has to be done in parallel by the processes and in sync, it
is best if we give them equal amount of work which follows similar
pattern for work (one magic process cannot be searching inside a hash
buffer while other is on a seqscan, they are part of same function
call).
Since at a relatively finer level of organization of data, the buffer /
page is same for each type of table, we can ask the processes to work on
similar page buffers.
==> The final question:
Since I want to parallalize at the buffer level, I basically want to
load multiple pages at once and then release them at once. So the final
question is:
Can I ask the buffer manager to load 4 (in this case) pages at one time
and use them for my algorithm to search inside them (no modification is
done here, so no problem for the WAL)?
==================================
Thanks for reading the email. I would have asked this one single
question but as it happened in the IRC channel, many questions came up
and I was suggested to ask on the list. I know that the same questions
will come up here and rather than eating up time and head of pg hackers,
I thought of writing it in a comprehensive style and detail what is
possible.
I would like to know the suggestions.
Regards,
Vaibhav (*_*)
