Thread: DTrace build dependency rules

Hi,

There seems to be a bug in the make rules when DTrace is enabled. It
causes dtrace -G to be invoked twice when building PostgreSQL as a
FreeBSD port: once during the build itself, and once during
installation. For a long time this has been worked around on FreeBSD
with a change to libdtrace itself, but this workaround is proving
problematic and I'd like to fix the problem properly. I'm not sure
whether the problem has been observed on other operating systems that
support DTrace.

The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated
object file, depends on the objfiles.txt for each of the backend
subdirs. These files depend in turn on the object files themselves; if
objfiles.txt is out of date with respect to one of its object files, the
mtime of objfiles.txt is updated with "touch" (see backend/common.mk).
The problem is that dtrace -G, which runs at the end of the build,
modifies a number of object files (it overwrites their probe sites with
NOPs), thus making their corresponding objfiles.txt out of date. Then,
when "make install" traverses the backend subdirs, it updates
objfiles.txt, which causes probes.o to be rebuilt, resulting in an error
from dtrace(1).

The attached patch fixes the problem by having probes.o depend on object
files directly, rather than on objfiles.txt. I've tested it with
PostgreSQL 9.0-9.4 on FreeBSD CURRENT.

Thanks!
-Mark

On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston <markj@freebsd.org> wrote:
> There seems to be a bug in the make rules when DTrace is enabled. It
> causes dtrace -G to be invoked twice when building PostgreSQL as a
> FreeBSD port: once during the build itself, and once during
> installation. For a long time this has been worked around on FreeBSD
> with a change to libdtrace itself, but this workaround is proving
> problematic and I'd like to fix the problem properly. I'm not sure
> whether the problem has been observed on other operating systems that
> support DTrace.
>
> The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated
> object file, depends on the objfiles.txt for each of the backend
> subdirs. These files depend in turn on the object files themselves; if
> objfiles.txt is out of date with respect to one of its object files, the
> mtime of objfiles.txt is updated with "touch" (see backend/common.mk).
> The problem is that dtrace -G, which runs at the end of the build,
> modifies a number of object files (it overwrites their probe sites with
> NOPs), thus making their corresponding objfiles.txt out of date. Then,
> when "make install" traverses the backend subdirs, it updates
> objfiles.txt, which causes probes.o to be rebuilt, resulting in an error
> from dtrace(1).

Gosh, that's pretty ugly.  I would have thought it would be a real
no-no to update the .o file once it got generated.  If nothing else, a
modification to the .c file concurrent with a make invocation might
lead to the .o not getting rebuilt the next time make is run.

> The attached patch fixes the problem by having probes.o depend on object
> files directly, rather than on objfiles.txt. I've tested it with
> PostgreSQL 9.0-9.4 on FreeBSD CURRENT.

I don't see a particular reason not to make this change, though.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas wrote:
> On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston <markj@freebsd.org> wrote:

> > The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated
> > object file, depends on the objfiles.txt for each of the backend
> > subdirs. These files depend in turn on the object files themselves; if
> > objfiles.txt is out of date with respect to one of its object files, the
> > mtime of objfiles.txt is updated with "touch" (see backend/common.mk).
> > The problem is that dtrace -G, which runs at the end of the build,
> > modifies a number of object files (it overwrites their probe sites with
> > NOPs), thus making their corresponding objfiles.txt out of date. Then,
> > when "make install" traverses the backend subdirs, it updates
> > objfiles.txt, which causes probes.o to be rebuilt, resulting in an error
> > from dtrace(1).
> 
> Gosh, that's pretty ugly.  I would have thought it would be a real
> no-no to update the .o file once it got generated.  If nothing else, a
> modification to the .c file concurrent with a make invocation might
> lead to the .o not getting rebuilt the next time make is run.

I had the same thought, and wondered for a bit whether we should instead
have the compilation rules produce some intermediate file (prior to
dtrace fumbling), then emit the .o from dtrace -G.  OTOH this might be
more trouble than is worth for a feature that doesn't see a lot of use.

> > The attached patch fixes the problem by having probes.o depend on object
> > files directly, rather than on objfiles.txt. I've tested it with
> > PostgreSQL 9.0-9.4 on FreeBSD CURRENT.
> 
> I don't see a particular reason not to make this change, though.

I wanted to test it on Linux yesterday but didn't get any further than
installing a couple of packages.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Aug 18, 2015 at 12:08 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> Robert Haas wrote:
>> On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston <markj@freebsd.org> wrote:
>
>> > The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated
>> > object file, depends on the objfiles.txt for each of the backend
>> > subdirs. These files depend in turn on the object files themselves; if
>> > objfiles.txt is out of date with respect to one of its object files, the
>> > mtime of objfiles.txt is updated with "touch" (see backend/common.mk).
>> > The problem is that dtrace -G, which runs at the end of the build,
>> > modifies a number of object files (it overwrites their probe sites with
>> > NOPs), thus making their corresponding objfiles.txt out of date. Then,
>> > when "make install" traverses the backend subdirs, it updates
>> > objfiles.txt, which causes probes.o to be rebuilt, resulting in an error
>> > from dtrace(1).
>>
>> Gosh, that's pretty ugly.  I would have thought it would be a real
>> no-no to update the .o file once it got generated.  If nothing else, a
>> modification to the .c file concurrent with a make invocation might
>> lead to the .o not getting rebuilt the next time make is run.
>
> I had the same thought, and wondered for a bit whether we should instead
> have the compilation rules produce some intermediate file (prior to
> dtrace fumbling), then emit the .o from dtrace -G.  OTOH this might be
> more trouble than is worth for a feature that doesn't see a lot of use.

Given the lack of further interest from the PostgreSQL community,
that's my guess.  I've pushed this patch to master; let's see if we
get any complaints.  If it makes life better for FreeBSD without
making life worse for anyone else, I suppose we might as well do it
until something better comes along.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hola<br /> Tengo una duda, que tan pesado es poner ssl en una base. (me refiero si es mas pesado para el equipo usar
estaconexión o es igual a una con ip en hba?<br /> Mil Gracias<br /><br /><br /><div class="moz-signature">-- <br />
SaludosEnrique</div> 

On Tue, Oct 13, 2015 at 3:47 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Tue, Aug 18, 2015 at 12:08 PM, Alvaro Herrera
> <alvherre@2ndquadrant.com> wrote:
>> Robert Haas wrote:
>>> On Sat, Aug 15, 2015 at 6:45 PM, Mark Johnston <markj@freebsd.org> wrote:
>>
>>> > The bug is in src/backend/Makefile. probes.o, the dtrace(1)-generated
>>> > object file, depends on the objfiles.txt for each of the backend
>>> > subdirs. These files depend in turn on the object files themselves; if
>>> > objfiles.txt is out of date with respect to one of its object files, the
>>> > mtime of objfiles.txt is updated with "touch" (see backend/common.mk).
>>> > The problem is that dtrace -G, which runs at the end of the build,
>>> > modifies a number of object files (it overwrites their probe sites with
>>> > NOPs), thus making their corresponding objfiles.txt out of date. Then,
>>> > when "make install" traverses the backend subdirs, it updates
>>> > objfiles.txt, which causes probes.o to be rebuilt, resulting in an error
>>> > from dtrace(1).
>>>
>>> Gosh, that's pretty ugly.  I would have thought it would be a real
>>> no-no to update the .o file once it got generated.  If nothing else, a
>>> modification to the .c file concurrent with a make invocation might
>>> lead to the .o not getting rebuilt the next time make is run.
>>
>> I had the same thought, and wondered for a bit whether we should instead
>> have the compilation rules produce some intermediate file (prior to
>> dtrace fumbling), then emit the .o from dtrace -G.  OTOH this might be
>> more trouble than is worth for a feature that doesn't see a lot of use.
>
> Given the lack of further interest from the PostgreSQL community,
> that's my guess.  I've pushed this patch to master; let's see if we
> get any complaints.  If it makes life better for FreeBSD without
> making life worse for anyone else, I suppose we might as well do it
> until something better comes along.

Per http://www.postgresql.org/message-id/24541.1444928519@sss.pgh.pa.us
I have had to revert this patch, because it's breaking parallel builds
even for non-dtrace users.

Sorry for the inconvenience.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

2015-10-13 17:22 GMT-03:00 Rodolfo Campero <rodolfo.campero@anachronics.com>:
> La lista de correo apropiada para tu consulta es pgsql-es-ayuda
> (pgsql-es-ayuda(at)postgresql(dot)org).
> Saludos,
>
> El 13 de octubre de 2015, 16:51, Enrique Escobar <ekurth@live.com.mx>
> escribió:
>>
>> Hola
>> Tengo una duda, que tan pesado es poner ssl en una base. (me refiero si es
>> mas pesado para el equipo usar esta conexión o es igual a una con ip en hba?
>> Mil Gracias


A tu pregunta, sí, es más pesado. Considerablemente, pero el impacto
depende mucho de tu carga.

Lo más pesado de SSL es establecer la conexión. Si son conexiones con
larga vida, no hay tanto impacto. Menos cuanto menos transferencia de
datos tengas (ej: resultados pequeños). Pero si van a ser muchas
conexiones de corta duración, ahí sí el costo de establecer la
conexión te va a impactar mucho.

Como siempre, bencharkeá.