Thread: SPI Headers -- RPM distribution

SPI Headers -- RPM distribution

From
Lamar Owen
Date:
Ok, thanks to Oliver and others, I think I may have a handle on which
header files to include in the postgresql-devel RPM for the ability to
compile backend modules.

So, I'm going to put these headers under /usr/include/pgsql/backend.

Any objections or comments?

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Re: [HACKERS] SPI Headers -- RPM distribution

From
wieck@debis.com (Jan Wieck)
Date:
>
> Ok, thanks to Oliver and others, I think I may have a handle on which
> header files to include in the postgresql-devel RPM for the ability to
> compile backend modules.
>
> So, I'm going to put these headers under /usr/include/pgsql/backend.
>
> Any objections or comments?

    I'm  not  totally  sure, but IIRC the dependency list was the
    one of spi.c, no?

    If so, it's IMHO wrong. A user written  module  doesn't  need
    anything,  spi.c  needs  (especially  the  parts  included by
    spi_priv.h). SPI hides many internals by making the  prepared
    plan an opaque object (void *).


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #

Re: [HACKERS] SPI Headers -- RPM distribution

From
Brook Milligan
Date:
       I'm  not  totally  sure, but IIRC the dependency list was the
       one of spi.c, no?

I've had problems just including spi.h and trigger.h, actually.

Cheers,
Brook

Re: [HACKERS] SPI Headers -- RPM distribution

From
wieck@debis.com (Jan Wieck)
Date:
>
>        I'm  not  totally  sure, but IIRC the dependency list was the
>        one of spi.c, no?
>
> I've had problems just including spi.h and trigger.h, actually.

    This is from gcc -M for contrib/spi/autoinc.c:

        src/backend/fmgr.h
        src/include/access/attnum.h
        src/include/access/funcindex.h
        src/include/access/heapam.h
        src/include/access/htup.h
        src/include/access/relscan.h
        src/include/access/sdir.h
        src/include/access/skey.h
        src/include/access/strat.h
        src/include/access/transam.h
        src/include/access/tupdesc.h
        src/include/access/tupmacs.h
        src/include/access/xact.h
        src/include/c.h
        src/include/catalog/pg_am.h
        src/include/catalog/pg_attribute.h
        src/include/catalog/pg_class.h
        src/include/catalog/pg_language.h
        src/include/catalog/pg_proc.h
        src/include/catalog/pg_type.h
        src/include/commands/trigger.h
        src/include/config.h
        src/include/executor/execdefs.h
        src/include/executor/execdesc.h
        src/include/executor/executor.h
        src/include/executor/hashjoin.h
        src/include/executor/spi.h
        src/include/executor/tuptable.h
        src/include/lib/fstack.h
        src/include/nodes/execnodes.h
        src/include/nodes/memnodes.h
        src/include/nodes/nodes.h
        src/include/nodes/params.h
        src/include/nodes/parsenodes.h
        src/include/nodes/pg_list.h
        src/include/nodes/plannodes.h
        src/include/nodes/primnodes.h
        src/include/nodes/relation.h
        src/include/os.h
        src/include/parser/parse_node.h
        src/include/postgres.h
        src/include/postgres_ext.h
        src/include/rewrite/prs2lock.h
        src/include/storage/block.h
        src/include/storage/buf.h
        src/include/storage/buf_internals.h
        src/include/storage/buffile.h
        src/include/storage/bufmgr.h
        src/include/storage/bufpage.h
        src/include/storage/fd.h
        src/include/storage/ipc.h
        src/include/storage/item.h
        src/include/storage/itemid.h
        src/include/storage/itemptr.h
        src/include/storage/lmgr.h
        src/include/storage/lock.h
        src/include/storage/off.h
        src/include/storage/page.h
        src/include/storage/shmem.h
        src/include/storage/spin.h
        src/include/tcop/dest.h
        src/include/tcop/pquery.h
        src/include/tcop/tcopprot.h
        src/include/tcop/utility.h
        src/include/utils/array.h
        src/include/utils/builtins.h
        src/include/utils/datetime.h
        src/include/utils/datum.h
        src/include/utils/dt.h
        src/include/utils/elog.h
        src/include/utils/fcache.h
        src/include/utils/geo_decls.h
        src/include/utils/hsearch.h
        src/include/utils/inet.h
        src/include/utils/int8.h
        src/include/utils/lztext.h
        src/include/utils/mcxt.h
        src/include/utils/memutils.h
        src/include/utils/nabstime.h
        src/include/utils/numeric.h
        src/include/utils/palloc.h
        src/include/utils/pg_lzcompress.h
        src/include/utils/portal.h
        src/include/utils/rel.h
        src/include/utils/syscache.h
        src/include/utils/tqual.h


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #

Re: [HACKERS] SPI Headers -- RPM distribution

From
Lamar Owen
Date:
Jan Wieck wrote:
> > Ok, thanks to Oliver and others, I think I may have a handle on which
> > header files to include in the postgresql-devel RPM for the ability to
> > compile backend modules.

>     I'm  not  totally  sure, but IIRC the dependency list was the
>     one of spi.c, no?

>     If so, it's IMHO wrong. A user written  module  doesn't  need
>     anything,  spi.c  needs  (especially  the  parts  included by
>     spi_priv.h). SPI hides many internals by making the  prepared
>     plan an opaque object (void *).

As is probably obvious from my posting, I have never built an SPI module
-- however, I do want to make the RPM's SPI-friendly.  Oliver posted the
list of includes that he ships in the Debian packages -- I am digesting
that list now, comparing with a list of the files in the distribution
tarball.

While including the entire source tree is a last resort, it should be
possible to allow SPI programming without the whole 28MB unbuilt source
tree. According to the documentation, all an SPI module needs is to
#include executor/spi.h.

Well, spi.h needs (all includes after full expansion via /usr/lib/cpp
-I../../include -I../../backend -M spi.h, stripping the system header
files, stripping the ../../, reformatting to one header per line, and
sorting):
(paths relative to PG-SOURCE-TREE/src/)

 backend/fmgr.h
 include/access/attnum.h
 include/access/funcindex.h
 include/access/heapam.h
 include/access/htup.h
 include/access/ibit.h
 include/access/itup.h
 include/access/relscan.h
 include/access/sdir.h
 include/access/skey.h
 include/access/strat.h
 include/access/transam.h
 include/access/tupdesc.h
 include/access/tupmacs.h
 include/access/xact.h
 include/c.h
 include/catalog/catname.h
 include/catalog/pg_am.h
 include/catalog/pg_attribute.h
 include/catalog/pg_class.h
 include/catalog/pg_index.h
 include/catalog/pg_language.h
 include/catalog/pg_proc.h
 include/catalog/pg_type.h
 include/config.h
 include/executor/execdefs.h
 include/executor/execdesc.h
 include/executor/executor.h
 include/executor/hashjoin.h
 include/executor/tuptable.h
 include/lib/fstack.h
 include/nodes/execnodes.h
 include/nodes/memnodes.h
 include/nodes/nodes.h
 include/nodes/params.h
 include/nodes/parsenodes.h
 include/nodes/pg_list.h
 include/nodes/plannodes.h
 include/nodes/primnodes.h
 include/nodes/relation.h
 include/os.h
 include/parser/parse_node.h
 include/parser/parse_type.h
 include/postgres.h
 include/postgres_ext.h
 include/rewrite/prs2lock.h
 include/storage/block.h
 include/storage/buf.h
 include/storage/buf_internals.h
 include/storage/bufmgr.h
 include/storage/bufpage.h
 include/storage/fd.h
 include/storage/ipc.h
 include/storage/item.h
 include/storage/itemid.h
 include/storage/itemptr.h
 include/storage/lmgr.h
 include/storage/lock.h
 include/storage/off.h
 include/storage/page.h
 include/storage/shmem.h
 include/storage/sinvaladt.h
 include/storage/spin.h
 include/tcop/dest.h
 include/tcop/pquery.h
 include/tcop/tcopprot.h
 include/tcop/utility.h
 include/utils/array.h
 include/utils/builtins.h
 include/utils/datetime.h
 include/utils/datum.h
 include/utils/dt.h
 include/utils/elog.h
 include/utils/fcache.h
 include/utils/geo_decls.h
 include/utils/hsearch.h
 include/utils/inet.h
 include/utils/int8.h
 include/utils/mcxt.h
 include/utils/memutils.h
 include/utils/nabstime.h
 include/utils/numeric.h
 include/utils/palloc.h
 include/utils/portal.h
 include/utils/rel.h
 include/utils/syscache.h
 include/utils/tqual.h
(87 header files)

Well, analyzing spi.c the same way, it needs only 90 headers -- diff
tells me that the ONLY additional headers needed by spi.c that are not
needed by spi.h are spi.h (duh), spi_priv.h, and printtup.h.

To reiterate my thought -- a full source tree should not be necessary to
do meaningful work on PostgreSQL -- that's the whole reason for
existence of the binary distributions.  I am going to incorporate the
above listed headers in the postgresql-devel RPM -- I am not at all sure
where I need to put them, however.  The PostgreSQL include directory
under the RPM installation is /usr/include/pgsql (not my choice, but
would be hard to retroactively change) -- maybe put these files under
that in either backend, or SPI??
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Re: [HACKERS] SPI Headers -- RPM distribution

From
Tom Lane
Date:
Lamar Owen <lamar.owen@wgcr.org> writes:
> So, I'm going to put these headers under /usr/include/pgsql/backend.

Some of these headers already are exported.  The right thing is to
export the rest in parallel, not throw in an extra level of directory.
The path from /usr/include/pgsql should be the same as the path from
.../src/include in the source tree.

The only real problem with doing that is with fmgr.h, which for reasons
that I don't fathom isn't in src/include at all.  Seems to me it would
be a good idea to have fmgr.h (and I guess parse.h too) put into
src/include, and then we could get rid of -I../backend from the switches
used for backend compilations.

A good longer-term project would be to reduce the number of headers that
we need to export.  80 include files to compile spi.h?  Ugh.

            regards, tom lane

Re: [HACKERS] SPI Headers -- RPM distribution

From
Lamar Owen
Date:
Tom Lane wrote:

> Lamar Owen <lamar.owen@wgcr.org> writes:
> > So, I'm going to put these headers under /usr/include/pgsql/backend.
>
> Some of these headers already are exported.  The right thing is to
> export the rest in parallel, not throw in an extra level of directory.
> The path from /usr/include/pgsql should be the same as the path from
> .../src/include in the source tree.

Good point.

> The only real problem with doing that is with fmgr.h, which for reasons
> that I don't fathom isn't in src/include at all.  Seems to me it would
> be a good idea to have fmgr.h (and I guess parse.h too) put into
> src/include, and then we could get rid of -I../backend from the switches
> used for backend compilations.

Also, I have noticed that some headers are #included with <>, while
others are included with "".  Now, I understand that system headers
should be included with <>, but I'm seeing PostgreSQL headers wrapped
with <>.  This makes the cpp -M variants a little confusing -- while
tracing down the header dependencies, I started using cpp -MM, which
only follows user headers and not system headers -- but it doesn't get
all of the headers.

And that fmgr.h thingy -- ewww.  I'm sure there is a reason somewhere
that it is in backend, but it escapes me as to why.

> A good longer-term project would be to reduce the number of headers that
> we need to export.  80 include files to compile spi.h?  Ugh.

After reading Jan's message about the list of dependencies for spi.c
containing more than the dependencies for a user SPI module, I was
prepared to see alot less headers -- and was quite disappointed to see
87 headers show up.

My goal is to provide the minimum required to build things for the
backend for those who want to do so, without requiring them to install a
source tarball and relearn where all of the files are located.  Like I
said, if I have to provide a complete source tarball to do some things,
I will -- I just would rather not.  If someone wants the source to
study, they can install the src.rpm and play to their hearts' content --
and they can then rebuild the whole package in a manner consistent with
the standard RPM installation with a single command, and not have to
relearn anything.

So, taking your suggestion, I'm going to add all these required headers
to the RPM package as part of the postgresql-devel RPM.  Since RPM's
database handles the aspect of what files belong to what package, it is
redundant to place these SPI includes in a separate place -- and then
developers only need to properly set the location of the PostgreSQL
include tree once (/usr/include/pgsql).

I will try to fully document these changes -- and am building this RPM
as a beta RPM until such time as it is verified to work for those
developing SPI modules under the RPM installation.

Thanks Tom.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Re: [HACKERS] SPI Headers -- RPM distribution

From
Tom Lane
Date:
Lamar Owen <lamar.owen@wgcr.org> writes:
> Also, I have noticed that some headers are #included with <>, while
> others are included with "".  Now, I understand that system headers
> should be included with <>, but I'm seeing PostgreSQL headers wrapped
> with <>.  This makes the cpp -M variants a little confusing -- while
> tracing down the header dependencies, I started using cpp -MM, which
> only follows user headers and not system headers -- but it doesn't get
> all of the headers.

Bruce has done a nice job of cleaning that up in current sources, but
yes, 6.5.* and older releases were mighty inconsistent about header
inclusions.

            regards, tom lane