Thread: Re: Threads vs Processes (was: NuSphere and PostgreSQL for window

> Claudio Natoli <claudio.natoli@memetrics.com> writes:
> > FWIW, I've got a threaded version of the WIN32_DEV branch more or less
> > "running" (it is a terrible hack job, so NO, no patches... yet :-), as a
> > proof of concept. Still a work in progress (ok, I've qualified it
enough),
> > but it is showing enough promise to convince me that threading is the
way to
> > go for the Win32 port.
>
> How are you dealing with the issue of wanting some static variables to
> be per-thread and others not?
>
>             regards, tom lane

To be perfectly honest, I'm still trying to familiarize myself with the code
sufficiently well so that I can tell which variables need to be per-thread
and which are shared (and, in turn, which of these need to be protected from
concurrent access). So, in short, I'm not dealing with the issue (and,
hence, it is only "running" in the very loosest sense of the word).

Unfortunately, I'm not yet even in a position to propose a reasonable model,
let alone one that'd play well with the existing code base. Perhaps some
time soon, hopefully... (I really want to get involved in this.)

Cheers,
Claudio

---
WE HAVE MOVED - PLEASE NOTE OUR NEW CONTACT DETAILS:
THE BASEMENT, 33 EWELL STREET, BALMAIN NSW 2041
TEL: +61 2 9555 1544 FAX: +61 2 9555 6911
Certain disclaimers and policies apply to all email sent from Memetrics.
For the full text of these disclaimers and policies see
http://www.memetrics.com/emailpolicy.html

Claudio Natoli <claudio.natoli@memetrics.com> writes:
>> How are you dealing with the issue of wanting some static variables to
>> be per-thread and others not?

> To be perfectly honest, I'm still trying to familiarize myself with the code
> sufficiently well so that I can tell which variables need to be per-thread
> and which are shared (and, in turn, which of these need to be protected from
> concurrent access).

Well, the first-order approximation would be to duplicate the current
fork semantics: *all* static variables are per-thread, and should be
copied from the parent thread at thread creation.  If there is some
reasonably non-invasive way to do that, we'd have a long leg up on the
problem.

            regards, tom lane

Tom Lane wrote:

> Claudio Natoli <claudio.natoli@memetrics.com> writes:
>
>>>How are you dealing with the issue of wanting some static variables to
>>>be per-thread and others not?
>
>
>>To be perfectly honest, I'm still trying to familiarize myself with the code
>>sufficiently well so that I can tell which variables need to be per-thread
>>and which are shared (and, in turn, which of these need to be protected from
>>concurrent access).
>
>
> Well, the first-order approximation would be to duplicate the current
> fork semantics: *all* static variables are per-thread, and should be
> copied from the parent thread at thread creation.  If there is some
> reasonably non-invasive way to do that, we'd have a long leg up on the
> problem.

Hmm.. I was looking for some fast tutorials on thread local storage. I found
this one..
http://publib16.boulder.ibm.com/pseries/en_US/aixprggd/genprogc/thread_specific_data.htm

Basically, in a process we are free to declare as many globals as we can.
Converting them to thread local is not an easy job because each variable would
need it's own key and there is limit on how many keys can be allocated.

One thing that can be done is to arrange all globals/statics in a structure and
make that structure thread local. We need to change all invocations of any of
those variables to use a pointer. We just need only one global variable. And
some macro trickery possibly so that we can extend that structure easily and
automatically.

Upshot is duplicating environment is easy. We need to do a huge memcpy and any
specific depp copy of strings on thread creation. Besides even in process model,
  this kind of initialization will allow to put all variables on heap instead of
stack. But then we need to add initialization code explicitly.

Something like int a=10; can not be added just like that.

If globals are less than 100 in numbers, I think it should be reasonably blind
job of converting them to a structure type stuff. Don't know really though. My
estimations are always 10% of what it takes..:-)

I hope I got it correct..

  Shridhar

Tom Lane wrote:

>Claudio Natoli <claudio.natoli@memetrics.com> writes:
>
>
>>>How are you dealing with the issue of wanting some static variables to
>>>be per-thread and others not?
>>>
>>>
>
>
>
>>To be perfectly honest, I'm still trying to familiarize myself with the code
>>sufficiently well so that I can tell which variables need to be per-thread
>>and which are shared (and, in turn, which of these need to be protected from
>>concurrent access).
>>
No. Not protected from concurrent access. Each thread must have it's own
copy.

>>
>>
>
>Well, the first-order approximation would be to duplicate the current
>fork semantics: *all* static variables are per-thread, and should be
>copied from the parent thread at thread creation.  If there is some
>reasonably non-invasive way to do that, we'd have a long leg up on the
>problem.
>
There is a declspec(thread) that makes a global variable per-thread.
AFAIK it uses linker magic to replace the actual memory accesses with
calls to TlsAlloc() etc. Note that declspec(thread) doesn't work from
within dynamic link libraries, but that shouldn't be a big problem.

--
    Manfred

Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
> One thing that can be done is to arrange all globals/statics in a
> structure and make that structure thread local.

That's about as far from "non-invasive" as I can imagine :-(

I really, really want to avoid doing anything like the above, because it
would force us to expose to the whole backend many data structures and
state variables that are currently local to individual .c files.  That
complicates understanding and debugging tremendously, not to mention
slowing the edit/compile/debug cycle when you are changing such
structures.

            regards, tom lane

On Thursday, September 25, 2003, at 10:03 AM, Tom Lane wrote:

> Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
>> One thing that can be done is to arrange all globals/statics in a
>> structure and make that structure thread local.
>
> That's about as far from "non-invasive" as I can imagine :-(
>
> I really, really want to avoid doing anything like the above, because 
> it
> would force us to expose to the whole backend many data structures and
> state variables that are currently local to individual .c files.  That
> complicates understanding and debugging tremendously, not to mention
> slowing the edit/compile/debug cycle when you are changing such
> structures.


Another option would be to create thread local hashtable or other lookup
structure to which you would register a structure for a particular .c 
file
or group of files.

You could then define the structures you need locally without affecting other parts of the codebase.



Myron Scott

Tom Lane wrote:

> Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
>
>>One thing that can be done is to arrange all globals/statics in a
>>structure and make that structure thread local.
>
>
> That's about as far from "non-invasive" as I can imagine :-(

I know.

I have following half baked solution. Obviously it won't be as good and
automatic as __declspec(thread). But still... Just for starters

1. Put all static/global variables in .h file. No more in .c files
2. Encapsulate them in a #define.
3. Create a new header files which includes lot of macro invocations. That way
we expand a macro definition, the structure definition changes on recompile.

Like in a.h
a.h
---
#define module1 int a;\
        int b;
---

b.h
---
typedef struct
{
  module1
  module2
} globalDataStructure

---

Whoever includes a.h earlier, needs to include b.h now. That way things will
flow smoothly. This could be optional but I would rather not include a.h anymore
as a good practice.

4. Add a macro definition for initialization function. The global initialization
function will again invoke macro implementations which are free to redirect it
to any function of choice.

That way we can maintain variables in modules that they are but still put them
in a single compile unit when they get compiled. Only one file b.h, need to be
maintained for addition of new files. The variables would be taken care of in
the respective modules. ( This could nicely be done in C++ using classes but
thats a
altogether different subject.)

The changes to actual C code obviously need to follow.

I agree that's messy and non transparent. but it still retains distributed
nature of variable declarations and maintenance thereof..

But I feel we are missing the point. Last time threads for unix builds were
discussed, one of the good suggestion was to use them to distribute load across
CPU's rather than using thread per connection. Personally I don't think
postgresql is missing lot of performance by using process/connection rather than
thread/connection(Except for slowaris and windows perhaps).

Even on windows, the time taken for a new connection is unlikely to be an issue,
I feel.

We really don't need threads to replace existing functionality. That would be
dog work.

If we are writing new functionality, we can do it clean and portable way rather
than fiddling with existing code. Changing working code is always a messy
business irrespective of technicalities involved. As a support engineer, I can
attest that.

  Shridhar

Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
> We really don't need threads to replace existing functionality. That
> would be dog work.

No, that's not the point at all.  The problem we are facing at the
moment with the Windows port is lack of fork(), which means there's
no way for separate-subprocess backends to inherit variable values
from the postmaster.  Bruce has been trying to fix that by having the
subprocesses somehow reload or re-deduce all those variables; which
is messy, bug-prone, and probably race-condition-prone too.  In a
threaded implementation it would maybe be relatively easy to initialize
a new thread's TLS by copying the postmaster thread's TLS, in which case
a whole pile of as-yet-unwritten Windows-only code won't be needed.

            regards, tom lane

On Friday 26 September 2003 20:19, Tom Lane wrote:
> Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
> > We really don't need threads to replace existing functionality. That
> > would be dog work.
>
> No, that's not the point at all.  The problem we are facing at the
> moment with the Windows port is lack of fork(), which means there's
> no way for separate-subprocess backends to inherit variable values
> from the postmaster.  Bruce has been trying to fix that by having the
> subprocesses somehow reload or re-deduce all those variables; which
> is messy, bug-prone, and probably race-condition-prone too.  In a
> threaded implementation it would maybe be relatively easy to initialize
> a new thread's TLS by copying the postmaster thread's TLS, in which case
> a whole pile of as-yet-unwritten Windows-only code won't be needed.

Umm.. I understand child process created by createProcess does not inherit
variable values from parent process. That's where problem originates..

We can simply create a registry key that would contain shared memory id from
where a child process should get the variable values.

And that would need initialization function I talked about earlier.  And since
anyways TLS->TLS copy is still needed anyways, I think this approach can
still save us dealing with threads.

God.. it doesn't get any less messy..I hope this is of some value..

 Shridhar

Tom Lane wrote:
> Shridhar Daithankar <shridhar_daithankar@persistent.co.in> writes:
> > We really don't need threads to replace existing functionality. That
> > would be dog work.
>
> No, that's not the point at all.  The problem we are facing at the
> moment with the Windows port is lack of fork(), which means there's
> no way for separate-subprocess backends to inherit variable values
> from the postmaster.  Bruce has been trying to fix that by having the
> subprocesses somehow reload or re-deduce all those variables; which
> is messy, bug-prone, and probably race-condition-prone too.  In a
> threaded implementation it would maybe be relatively easy to initialize
> a new thread's TLS by copying the postmaster thread's TLS, in which case
> a whole pile of as-yet-unwritten Windows-only code won't be needed.

I haven't said much in this thread yet because I wasn't sure what to
say.

I worked 2-3 weeks on Win32 back in March/April, and a few days last
month getting it to compile again, and for the interfaces/clients to run
under Win32.

I haven't had any time to work on the fork/exec, though I now have a few
weeks to spend on it, so it isn't that I have been trying to get
fork/exec working, and failing, it is that I haven't even tried lately.

My plan is to pass a few values to the child via the command line:

    #ifdef EXEC_BACKEND
        Assert(UsedShmemSegID != 0 && UsedShmemSegAddr != NULL);
        /* database name at the end because it might contain commas */
        snprintf(pbuf, NAMEDATALEN + 256, "%d,%d,%d,%p,%s", port->sock, canAcceptConnections(),
                 UsedShmemSegID, UsedShmemSegAddr, port->database_name);
        av[ac++] = pbuf;
    #else

and pass the GUC settings via a special binary file.  (Those are already
in main CVS.)  The other values I plan to regenerate in the child the
same way the postmaster does it at initialization time.  The easy part
of that is that I only have to worry about postmaster variables.  All my
current fork/exec work is marked by #ifdef EXEC_BACKEND in current CVS,
so it can be easily ripped out.

One solution is for me to continue with this in the Win32 CVS version
until I have fork/exec() working on Unix, then test on Win32.  I think
that could be done in a few weeks, if not less.

Another solution, already mentioned, is to use threads and TLS.  This is
what SRA's code uses.  I know SRA wants to contribute that code back to
the community, so I can ask them to see if they are ready to release it.
That would show us all the changes needed to do threading.  Their code
is based on 7.3.X, rather than PeerDirect's which is based on 7.2.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:
> One solution is for me to continue with this in the Win32 CVS version
> until I have fork/exec() working on Unix, then test on Win32.  I think
> that could be done in a few weeks, if not less.

> Another solution, already mentioned, is to use threads and TLS.  This is
> what SRA's code uses.  I know SRA wants to contribute that code back to
> the community, so I can ask them to see if they are ready to release it.

If you are willing to expend the effort, I think it would be worth the
time to pursue both approaches.  We don't yet have enough info to decide
which one will be cleaner, so we need to push forward on both until we
can make a realistic comparison.

            regards, tom lane

Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > One solution is for me to continue with this in the Win32 CVS version
> > until I have fork/exec() working on Unix, then test on Win32.  I think
> > that could be done in a few weeks, if not less.
>
> > Another solution, already mentioned, is to use threads and TLS.  This is
> > what SRA's code uses.  I know SRA wants to contribute that code back to
> > the community, so I can ask them to see if they are ready to release it.
>
> If you are willing to expend the effort, I think it would be worth the
> time to pursue both approaches.  We don't yet have enough info to decide
> which one will be cleaner, so we need to push forward on both until we
> can make a realistic comparison.

I think I know enough to get the fork/exec working, particularly because
most of the work can be tested under Unix.  I don't know enough to get
the threads working, and I have had no offers of help since I started.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

> > We really don't need threads to replace existing functionality. That
> > would be dog work.
>
> No, that's not the point at all.  The problem we are facing at the
> moment with the Windows port is lack of fork(), which means there's
> no way for separate-subprocess backends to inherit variable values
> from the postmaster.  Bruce has been trying to fix that by having the
> subprocesses somehow reload or re-deduce all those variables; which
> is messy, bug-prone, and probably race-condition-prone too.  In a
> threaded implementation it would maybe be relatively easy to initialize
> a new thread's TLS by copying the postmaster thread's TLS, in which case
> a whole pile of as-yet-unwritten Windows-only code won't be needed.

Kepp in mind though all the cool things that could be done if we had
thread capabilities.  eg. evaluating different subexpressings on
fdifferent cpuis for the one query, etc.

Chris