Thread: loading libraries on Postmaster startup

While using PL/R in a web based application, I noticed that the library
load and initialization time is significant enough to be annoying. So I
wrote a quick hack to load and initialize the library on postmaster
startup. This way, the backends get a fully initialized copy of the
interpreter when they are forked. The hack was to postmaster.c just
after the SSL initialization code at about line 650 (just remembered
this is 7.3.2 though):
   if (true) /* later use startup GUC var */   {     char *fullname = "$libdir/plr.so";     char *funcname =
"start_interp";    func_ptr initfunc;
 
     initfunc = (func_ptr)                 load_external_function(fullname, funcname, true, NULL);     (*initfunc)();
}

(I also had to add a #define for func_ptr)

This brings me to a couple questions:

1. Is there anything inherently dangerous with this approach?    My light testing seems to show that it works quite
wellfor    my purpose.
 

2. It seems to me that other libraries such as those for PL/Tcl,    PL/Perl, etc may have the same issue. Is there any
meritin    a GUC variable to allow libraries such as this to be loaded    and initialized at postmaster start? I'll
generalizethis and    send in a patch if there is interest.
 

Joe

Joe Conway <mail@joeconway.com> writes:

> 2. It seems to me that other libraries such as those for PL/Tcl,
>      PL/Perl, etc may have the same issue. Is there any merit in
>      a GUC variable to allow libraries such as this to be loaded
>      and initialized at postmaster start? I'll generalize this and
>      send in a patch if there is interest.

A similar situation arises with mod_perl. Because perl is quite heavy-weight
and systems often need lots of packages with static data it's common to load a
startup.pl script that just loads lots of packages before the Apache server
forks. This reduces memory usage drastically.

The main gotcha is that you have to be careful about resources that you don't
want shared. The typical case is database handles which are sockets that
wouldn't be happy having two processes writing and reading on them.

At first blush it seemed unlikely you would have a database connection in an
embedded perl script. But then, hm, that would be a sly way of doing
interdatabase connections. In any case there are other situations where you
might want to have open file descriptors or sockets lying around.

So in short, not only is it useful, but it would be valuable to allow
mechanism to cause the language to load modules before forking. But there have
to be prominent caveats that no such shared packages should create resources
that can't be safely shared.

--
greg

Joe Conway <mail@joeconway.com> writes:
> [ what about autoloading libraries into the postmaster? ]

I can see a couple possible downsides: (a) the library might have some
weird behavior across fork boundaries; (b) the additional memory space
that has to be duplicated into child processes will cost something per
child launch, even if the child never uses it.  But these are only
arguments that it might not *always* be a prudent thing to do, not that
we shouldn't give the DBA the tool to do it if he wants.  So fire away.

(I seem to recall Peter muttering about linking plperl, pltcl, etc
statically into the backend; which would reduce the need for this.
But it would not eliminate it ... and he hasn't done it anyway...)
        regards, tom lane

Joe Conway writes:

> So I wrote a quick hack to load and initialize the library on postmaster
> startup.

On glibc systems you can probably do this using the environment variable
LD_PRELOAD.  I guess others have a similar mechanism.

-- 
Peter Eisentraut   peter_e@gmx.net

Peter Eisentraut wrote:
> Joe Conway writes:
> 
>>So I wrote a quick hack to load and initialize the library on postmaster
>>startup.
> 
> On glibc systems you can probably do this using the environment variable
> LD_PRELOAD.  I guess others have a similar mechanism.
> 

Hmmm. I could try that. But I found during testing that the loading was 
actually not the slow part, it was running the initialization function 
for the interpreter that was. I wonder if there is there any way to get 
an initialization function to automatically execute?

Joe

On Thursday 13 February 2003 22:24, Joe Conway wrote:
> Peter Eisentraut wrote:
> > Joe Conway writes:
> >>So I wrote a quick hack to load and initialize the library on postmaster
> >>startup.
> >
> > On glibc systems you can probably do this using the environment variable
> > LD_PRELOAD.  I guess others have a similar mechanism.
>
> Hmmm. I could try that. But I found during testing that the loading was
> actually not the slow part, it was running the initialization function
> for the interpreter that was. I wonder if there is there any way to get
> an initialization function to automatically execute?
>
Joe, did You got the answer to this question ?
I would like to acomplish something like this:execute some stored procedure on backend(not postmaster) start and exit.
So, it is not the same reason, but it is still the same question.

Regards !