Thread: Signals on Win32 (yet again)
Considering the input we've received lately, it looks like the option of making signal handlers thread safe is going to be really difficult. Likewise, finding "good places" to tuck in SleepEx calls is probaly not going to be easy. (I still think SleepEx and User APCs have to be a much faster and cleaner solutions than a hidden window - while rqeuiring the exact same thing which is a set of polling points) Which puts us back at the solution that uses a kernel driver to deliver signals/APCs. While it would be very nice to be able to do things in userspace only, that might just not be possible to do cleanly. And using the driver model will defintly make the modifications *of postgresql* smallest. Considering earlier comments of undocumented features etc, I tried to work through all the background material to that post (including the info where you had to register to get it). I've come up with the following points: 1) There are actually *two* implementations. a) Is the one references, that's on http://www.codeproject.com/threads/queueuserapcex.asp b) Is the one on http://www.windevnet.com/wdm/articles/2001/0108/ [free reg required] The main difference between these is that (a) uses the undocumented (see later point) ETHREAD structure, whereas (b) appears to only use documented kernel functions. (a) also supports "generic user APCs", whereas (b) only supports signal functions (or rather, anything which looks like it). 2) The ETHREAD structure is only semi-undocumented. See http://www.microsoft.com/mspress/books/sampchap/4354b.asp [this is from *the* guide to Win32 internals]. You can dump this structure from the kernel debugger. While not documented, you can at least get at it. The variable changed is clearly named "byte UserApcPending", so it should be easy to keep track of. It sholdn't be too hard to check if it's present at exactly that address in NT4, XP and 2003. The articles both list Inside Windows NT as the soruce, and it's still there in Windows 2000 according to the link above. This makes if feel quite a bit safer than I thought before. 3) Solution (a) is more generic, while still using less code in the kernel driver. In (a) the driver just sets the process to alertable. In (b) it actually calls the function. It should be possible to load and unload the driver "on the fly" from the postmaster assuming the user has permissions (for example, when running from commandline). As well as installing it in the system for automatic starting. It will completely remove the ability to run postgresql on a win32 box if you don't have *access* to an admin account (to install the driver). We'd probably need to ship the driver in binary format even for those that get the source distribution - a lot more people have the SDK to build programs than the DDK used to build drivers. Thoughts? //Magnus
Magnus Hagander wrote: >Considering the input we've received lately, it looks like the option of >making signal handlers thread safe is going to be really difficult. > > :-( If we ever want to get to a fully threaded Postgres that will surely have to be tackled. I agree it might mean major surgery, and we should not hold up W32 for it. >Likewise, finding "good places" to tuck in SleepEx calls is probaly not >going to be easy. > Maybe. I'm not quite convinced of that yet - we can SleepEx with a very small timeout, no? There must be a few critical places the call could be made, which would in effect just delay delivery of the signal for a very short time to some convenient sequence point. >(I still think SleepEx and User APCs have to be a much >faster and cleaner solutions than a hidden window - while rqeuiring the >exact same thing which is a set of polling points) > > I agree. >[snip] discussion of kernel driver solution > > Now you're over my head ;-) Thanks for all the good research. cheers andrew
>Maybe. I'm not quite convinced of that yet - we can SleepEx with >a very small timeout, no? There must be a few critical places the >call could be made, which would in effect just delay delivery of >the signal for a very short time to some convenient sequence point. FWIW that method gets my vote - calling SleepEx(0) in some critical places; I believe that will yield the CPU but not wait any time (so if nothing else wants the CPU and there aren't any procedures that need calling then it amounts to a no-op). Using a driver to do this is killing an ant with a hammer, no matter that we find the ant somewhat irritating.. :) - Steve
Andrew Dunstan wrote: > Maybe. I'm not quite convinced of that yet - we can SleepEx with a very > small timeout, no? There must be a few critical places the call could be > made, which would in effect just delay delivery of the signal for a very > short time to some convenient sequence point. Actually, you don't need any timeout at all. WaitForSingleObject(INFINITE) keeps on running if the event object is kept signaled, so performance is not an issue. We can use 'manual' events to keep the Event object open all the time unless explicitly turned off via a signal thread. Implementation difficulties aside, what is more attractive from an aesthetic standpoint? Releasing (and supporting) a 100 line binary kernel driver for win32 or adding polling to the source in all the key loops? Merlin
Steve Tibbett wrote: >FWIW that method gets my vote - calling SleepEx(0) in some critical >places; I believe that will yield the CPU but not wait any time (so if >nothing else wants the CPU and there aren't any procedures that need >calling then it amounts to a no-op). > > The semantics are: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/sleepex.asp "A value of zero causes the thread to relinquish the remainder of its time slice to any other thread of equal priority that is ready to run. If there are no other threads of equal priority ready to run, the function returns immediately, and the thread continues execution." - Joe Barrera Former Microsoft Employee Currently 3PP at Sun
>>Maybe. I'm not quite convinced of that yet - we can SleepEx with >>a very small timeout, no? There must be a few critical places the >>call could be made, which would in effect just delay delivery of >>the signal for a very short time to some convenient sequence point. > >FWIW that method gets my vote - calling SleepEx(0) in some critical >places; I believe that will yield the CPU but not wait any time (so if >nothing else wants the CPU and there aren't any procedures that need >calling then it amounts to a no-op). Correct, SleepEx(0) with nothing to do will only incur the cost of a switch to kernel mode and back. If other threads are runnable at the same priority level, they will get scheduled, but no actual sleeping is done. I agree that that one is probably the best one, if it can be done at a limited number of places. If we need to put hundreds of SleepEx()es int ehre, that's just too much. I was under the impression that this was the case, but perhaps we just need to research that line a bit more. I guess you'd also have to modify the lock manager in some way to make it enter alertable state when waiting for a lock from a different process, but that sholdn't be too hard. Haven't looked at all on that code. //Magnus
Magnus Hagander wrote: > Correct, SleepEx(0) with nothing to do will only incur the cost of a > switch to kernel mode and back. If other threads are runnable at the > same priority level, they will get scheduled, but no actual sleeping is > done. Is that 100% correct? The way I read MSDN docs for SleepEx() is that if there is at least one other thread running at any priority in the same process, the thread that calls SleepEx() gives up the remainder of its timeslice to that thread. This could have performance implications if other threads have to be created for some reason. I think WaitForSingleObject(INFINITE) (in the backend thread) is more appropriate. This will return immediately if the object is signaled but unlike SleepEx() does not give up the remainder of its timeslice. We can turn on the event when the backend starts up and leave it on until a signal thread turns it off (meaning, the backend must stop and take another action). Also, the signal processing will occur in the main thread, not the worker thread. As soon as the backend enters the signal handling routine, it resets the event status to 'on' meaning it can accept more signal interrupts. In the rare case that signals stack up, a FIFO stack could hold them (is this allowed in unix? I honestly don't know). Merlin
Merlin Moncure wrote: >Magnus Hagander wrote: > > >>Correct, SleepEx(0) with nothing to do will only incur the cost of a >>switch to kernel mode and back. If other threads are runnable at the >>same priority level, they will get scheduled, but no actual sleeping >> >> >is > > >>done. >> >> > >Is that 100% correct? The way I read MSDN docs for SleepEx() is that if >there is at least one other thread running at any priority in the same >process, the thread that calls SleepEx() gives up the remainder of its >timeslice to that thread. This could have performance implications if >other threads have to be created for some reason. > > If we catch the events in the main thread then there is nothing else to yield to. If we use a separate thread to catch them then it should call WaitForSingleObject instead of SleepEx, ISTM. In any case, this couldn't be handled by appropriate setting of thread priorities? >I think WaitForSingleObject(INFINITE) (in the backend thread) is more >appropriate. This will return immediately if the object is signaled but >unlike SleepEx() does not give up the remainder of its timeslice. We >can turn on the event when the backend starts up and leave it on until a >signal thread turns it off (meaning, the backend must stop and take >another action). Also, the signal processing will occur in the main >thread, not the worker thread. As soon as the backend enters the signal >handling routine, it resets the event status to 'on' meaning it can >accept more signal interrupts. In the rare case that signals stack up, >a FIFO stack could hold them (is this allowed in unix? I honestly don't >know). > > > On Unix IIRC you should not assume that you will get a signal more than once. If 2 signals are generated before the receiving process catches one, it might only see the signal once. That's why it is common to see a loop in a SIGCHLD handler, to catch all the children that might have died. cheers andrew
>>Is that 100% correct? The way I read MSDN docs for SleepEx() >is that if >>there is at least one other thread running at any priority in the same >>process, the thread that calls SleepEx() gives up the remainder of its >>timeslice to that thread. This could have performance implications if >>other threads have to be created for some reason. >> >> > >If we catch the events in the main thread then there is >nothing else to >yield to. If we use a separate thread to catch them then it >should call >WaitForSingleObject instead of SleepEx, ISTM. In any case, >this couldn't >be handled by appropriate setting of thread priorities? The system will schedule a thread of equal priority *in any process*. Which means another backend, or even Solitaire (I beleive it doesn't change thread priorities) can be scheduled. (meaning we give up our timeslice, I may have been a bit unclear about that in my first mail) If we go with WFSO() in the main thread for the event sent from the other process, then the sending backends end up blocking until we actually pick up the signal. I think we're much better off with a separate thread that picks it up and queues a user APC for the main thread. >>I think WaitForSingleObject(INFINITE) (in the backend thread) is more >>appropriate. This will return immediately if the object is >signaled but >>unlike SleepEx() does not give up the remainder of its timeslice. We >>can turn on the event when the backend starts up and leave it >on until a >>signal thread turns it off (meaning, the backend must stop and take >>another action). Also, the signal processing will occur in the main >>thread, not the worker thread. As soon as the backend enters >the signal >>handling routine, it resets the event status to 'on' meaning it can >>accept more signal interrupts. In the rare case that signals >stack up, >>a FIFO stack could hold them (is this allowed in unix? I >honestly don't >>know). I'm not quite following this part. You mean use the event "backwards", in the way that it is almost always signalled? Why not use it in the normal way and use sleep time of 0 instead? That will return immidiatly (after going into kernel mode and back, of course), without giving up the quantum. Then there is always WaitForSingleObjectEx() which will also schedule APCs. I definitly think we need to pick up the signals on a different thread, so the sending backend can get out fast. But then we can use several different methods to signal the main thread, assuming we do polling: 1) UserAPCs with either SleepEx() or WaitForSingleObjectEx() 2) An anonymous event that is checked with WaitForSingleObject() 3) Simply using an interlocked DWORD variable. The downside of this is that it's not Wait..():able, so we can *only* check this one. Both (1) and (2) lends us to the possibility to wake up from a lock or similar if the lock is implemented using a waitable object (mutex, semaphore, event etc). //Magnus
Magnus Hagander said: > >>>Is that 100% correct? The way I read MSDN docs for SleepEx() >>is that if >>>there is at least one other thread running at any priority in the same >>>process, the thread that calls SleepEx() gives up the remainder of its >>>timeslice to that thread. This could have performance implications if >>>other threads have to be created for some reason. >>> >>> >> >>If we catch the events in the main thread then there is >>nothing else to >>yield to. If we use a separate thread to catch them then it >>should call >>WaitForSingleObject instead of SleepEx, ISTM. In any case, >>this couldn't >>be handled by appropriate setting of thread priorities? > > The system will schedule a thread of equal priority *in any process*. > Which means another backend, or even Solitaire (I beleive it doesn't > change thread priorities) can be scheduled. (meaning we give up our > timeslice, I may have been a bit unclear about that in my first mail) > Ahh. OK, I understand now. > If we go with WFSO() in the main thread for the event sent from the > other process, then the sending backends end up blocking until we > actually pick up the signal. I think we're much better off with a > separate thread that picks it up and queues a user APC for the main > thread. > Given the above I agree at least that the event catching should be in a separate thread. If the handlers have thread safety issues one way could be have the event- catching thread just set a flag and then check for it in the main thread, which is a very common model for signal processing. cheers andrew
> I definitly think we need to pick up the signals on a > different thread, so the sending backend can get out fast. FWIW, I concur. Cheers, Claudio --- Certain disclaimers and policies apply to all email sent from Memetrics. For the full text of these disclaimers and policies see <a href="http://www.memetrics.com/emailpolicy.html">http://www.memetrics.com/em ailpolicy.html</a>
> The system will schedule a thread of equal priority *in any process*. > Which means another backend, or even Solitaire (I beleive it doesn't [...] OK, that is helpful. SleepEx is only appropriate when you really want your thread to sleep, not when you want to signaling. SleepEx is used mostly for Asynch I/O while waiting for File I/O to complete. > >>I think WaitForSingleObject(INFINITE) (in the backend thread) is more > >>appropriate. [...] > I'm not quite following this part. You mean use the event "backwards", > in the way that it is almost always signalled? Why not use it in the > normal way [...] My thought was to execute the signal in the worker thread while suspending the main thread. Then I realized an APC is simply a more direct way to do this. If we go the polling approach instead of the kernel driver approach, then IMO QueueUserAPC is the best route, with a timeout of 0 as you specified. > I definitly think we need to pick up the signals on a different thread, > so the sending backend can get out fast. Yes. > different methods to signal the main thread, assuming we do polling: > 1) UserAPCs with either SleepEx() or WaitForSingleObjectEx() Yes, but with WFSO, not SleepEx(). > 2) An anonymous event that is checked with WaitForSingleObject() I don't think there is a truly elegant solution with this approach. At minimum, after every 'poll' there has to be if (0 != ret) which should be avoided, if possible. There are some other ways to work it out, none AFAICT as elegant as QueueUserAPC. > Simply using an interlocked DWORD variable I Don't think this is a good idea. Has a final decision been made wrt the kernel driver approach? ISTM this should be fully discounted before going with a regular synchronization model. I've never seen one in action, though. Merlin
Hello, Just got back from vacation and the talk about signals implementation seems to have dried up. Is anybody working on this? If not, maybe I could help out. Merlin
I'm working on an implementation right now. NOt complete, but I hope to have a sketch out later today or tomorrow. //Magnus >-----Original Message----- >From: Merlin Moncure [mailto:merlin.moncure@rcsonline.com] >Sent: den 8 januari 2004 20:15 >To: pgsql-hackers-win32 >Subject: Re: [pgsql-hackers-win32] Signals on Win32 (yet again) > > >Hello, > >Just got back from vacation and the talk about signals implementation >seems to have dried up. Is anybody working on this? If not, maybe I >could help out. > >Merlin > >---------------------------(end of >broadcast)--------------------------- >TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/docs/faqs/FAQ.html >
I believe Claudio is working on it ... cheers andrew Merlin Moncure wrote: >Hello, > >Just got back from vacation and the talk about signals implementation >seems to have dried up. Is anybody working on this? If not, maybe I >could help out. > >Merlin > > > >
I was, but there's nothing like having some one beat you to the punch :-) -----Original Message----- From: Andrew Dunstan To: pgsql-hackers-win32 Sent: 1/9/04 6:50 AM Subject: Re: [pgsql-hackers-win32] Signals on Win32 (yet again) I believe Claudio is working on it ... cheers andrew Merlin Moncure wrote: >Hello, > >Just got back from vacation and the talk about signals implementation >seems to have dried up. Is anybody working on this? If not, maybe I >could help out. > >Merlin > > > > ---------------------------(end of broadcast)--------------------------- TIP 7: don't forget to increase your free space map settings --- Certain disclaimers and policies apply to all email sent from Memetrics. For the full text of these disclaimers and policies see <a href="http://www.memetrics.com/emailpolicy.html">http://www.memetrics.com/em ailpolicy.html</a>