Thread: PG_MODULE_MAGIC issue with small extension
Hello all,
I'm not sure if this is the correct list for extensions, if not please let me know where this should be sent too.
I have a mid sized append only table that is read fairly infrequently and contains 2 columns that contain on average about 0.5Kb and 2Kb of xml per row. In an attempt to same some space I am planning to deflate them with a predefined dictionary (we have a set schema for the xml) as unfortunately changing the table design to a more traditional structure is off the table. In general the compression and decompression will be done in the application layer but because we sometimes need to look at the content I am writing some PG functions to inflate and deflate them with this dictionary.
What I have found however is that when I go to load my extensions I get this error:
test=# create extension pg_zlib;
ERROR: incompatible library "/usr/local/lib/postgresql/pg_zlib.so": missing magic block
HINT: Extension libraries are required to use the PG_MODULE_MAGIC macro.
make install installcheck
[~/work/pg_zlib (master)]$ make install installcheck || cat regression.diffs
clang -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -O2 -std=c11 -I. -I./ -I/usr/local/Cellar/postgresql/9.5.3/include/server -I/usr/local/Cellar/postgresql/9.5.3/include/internal -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2 -c -o pg_zlib.o pg_zlib.c
pg_zlib.c:24:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
PG_MODULE_MAGIC;
^
1 warning generated.
clang -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -O2 -bundle -multiply_defined suppress -Wl,-undefined,dynamic_lookup -o pg_zlib.so pg_zlib.o -L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib -Wl,-dead_strip_dylibs -bundle_loader /usr/local/Cellar/postgresql/9.5.3/bin/postgres
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/lib/postgresql'
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/share/postgresql/extension'
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/share/postgresql/extension'
/usr/bin/install -c -m 755 pg_zlib.so '/usr/local/lib/postgresql/pg_zlib.so'
/usr/bin/install -c -m 644 .//pg_zlib.control '/usr/local/share/postgresql/extension/'
/usr/bin/install -c -m 644 .//pg_zlib--1.0.sql '/usr/local/share/postgresql/extension/'
/usr/local/lib/postgresql/pgxs/src/makefiles/../../src/test/regress/pg_regress --inputdir=./ --bindir='/usr/local/Cellar/postgresql/9.5.3/bin' --dbname=contrib_regression pg_zlib
(using postmaster on Unix socket, default port)
============== dropping database "contrib_regression" ==============
DROP DATABASE
============== creating database "contrib_regression" ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries ==============
test pg_zlib ... FAILED
======================
1 of 1 tests failed.
======================
The differences that caused some tests to fail can be viewed in the
file "/Users/bdouglas/work/pg_zlib/regression.diffs". A copy of the test summary that you see
above is saved in the file "/Users/bdouglas/work/pg_zlib/regression.out".
make: *** [installcheck] Error 1
*** /Users/bdouglas/work/pg_zlib/expected/pg_zlib.out Thu Sep 1 23:59:01 2016
--- /Users/bdouglas/work/pg_zlib/results/pg_zlib.out Fri Sep 2 10:26:14 2016
***************
*** 0 ****
--- 1,3 ----
+ create extension pg_zlib;
+ ERROR: incompatible library "/usr/local/lib/postgresql/pg_zlib.so": missing magic block
+ HINT: Extension libraries are required to use the PG_MODULE_MAGIC macro.
======================================================================
MODULE_big = pg_zlib
OBJS = pg_zlib.o
EXTENSION = pg_zlib
DATA = pg_zlib--1.0.sql
REGRESS = pg_zlib
pg_zlib.o: override CFLAGS += -std=c11
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
pg_zlib--1.0.sql
sql/pg_zlib.sql
I'm not sure if this is the correct list for extensions, if not please let me know where this should be sent too.
I have a mid sized append only table that is read fairly infrequently and contains 2 columns that contain on average about 0.5Kb and 2Kb of xml per row. In an attempt to same some space I am planning to deflate them with a predefined dictionary (we have a set schema for the xml) as unfortunately changing the table design to a more traditional structure is off the table. In general the compression and decompression will be done in the application layer but because we sometimes need to look at the content I am writing some PG functions to inflate and deflate them with this dictionary.
What I have found however is that when I go to load my extensions I get this error:
test=# create extension pg_zlib;
ERROR: incompatible library "/usr/local/lib/postgresql/pg_zlib.so": missing magic block
HINT: Extension libraries are required to use the PG_MODULE_MAGIC macro.
And I can't for the life of me work out why. As far as I understand them, the docs indicate that this should only really happen if I'm compiling against an 8.* headers. I'm on OSX el capitan, PG 9.5.3 (from brew), clang is Apple LLVM version 7.3.0 (clang-703.0.31). I have manually checked all the lib and include dirs and other than some suspicious looking ones in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/ (which I removed) everything looks like it is correctly linked to the 9.5.3 version.
I don't regularly work in C and I've never worked on a PG extension before so most of what I have is cobbled together by looking at what the mysql_fdw extension does. If I'm missing something fundamental to either C or extensions or if you can think of anything else I should check please let me know.
I have inlined the output from `make install installcheck` and the relevant parts of the files below. Thanks for you time and sorry about the formatting.
Brent
make install installcheck
[~/work/pg_zlib (master)]$ make install installcheck || cat regression.diffs
clang -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -O2 -std=c11 -I. -I./ -I/usr/local/Cellar/postgresql/9.5.3/include/server -I/usr/local/Cellar/postgresql/9.5.3/include/internal -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/libxml2 -c -o pg_zlib.o pg_zlib.c
pg_zlib.c:24:1: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
PG_MODULE_MAGIC;
^
1 warning generated.
clang -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -O2 -bundle -multiply_defined suppress -Wl,-undefined,dynamic_lookup -o pg_zlib.so pg_zlib.o -L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib -Wl,-dead_strip_dylibs -bundle_loader /usr/local/Cellar/postgresql/9.5.3/bin/postgres
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/lib/postgresql'
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/share/postgresql/extension'
/bin/sh /usr/local/lib/postgresql/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/share/postgresql/extension'
/usr/bin/install -c -m 755 pg_zlib.so '/usr/local/lib/postgresql/pg_zlib.so'
/usr/bin/install -c -m 644 .//pg_zlib.control '/usr/local/share/postgresql/extension/'
/usr/bin/install -c -m 644 .//pg_zlib--1.0.sql '/usr/local/share/postgresql/extension/'
/usr/local/lib/postgresql/pgxs/src/makefiles/../../src/test/regress/pg_regress --inputdir=./ --bindir='/usr/local/Cellar/postgresql/9.5.3/bin' --dbname=contrib_regression pg_zlib
(using postmaster on Unix socket, default port)
============== dropping database "contrib_regression" ==============
DROP DATABASE
============== creating database "contrib_regression" ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries ==============
test pg_zlib ... FAILED
======================
1 of 1 tests failed.
======================
The differences that caused some tests to fail can be viewed in the
file "/Users/bdouglas/work/pg_zlib/regression.diffs". A copy of the test summary that you see
above is saved in the file "/Users/bdouglas/work/pg_zlib/regression.out".
make: *** [installcheck] Error 1
*** /Users/bdouglas/work/pg_zlib/expected/pg_zlib.out Thu Sep 1 23:59:01 2016
--- /Users/bdouglas/work/pg_zlib/results/pg_zlib.out Fri Sep 2 10:26:14 2016
***************
*** 0 ****
--- 1,3 ----
+ create extension pg_zlib;
+ ERROR: incompatible library "/usr/local/lib/postgresql/pg_zlib.so": missing magic block
+ HINT: Extension libraries are required to use the PG_MODULE_MAGIC macro.
======================================================================
Makefile
MODULE_big = pg_zlib
OBJS = pg_zlib.o
EXTENSION = pg_zlib
DATA = pg_zlib--1.0.sql
REGRESS = pg_zlib
pg_zlib.o: override CFLAGS += -std=c11
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
create function inf(bytea)
returns text
as 'MODULE_PATHNAME'
language C strict;
create function def(text)
returns bytea
as 'MODULE_PATHNAME'
language C strict;
comment = 'zlib extension for compressing and decompressing text'
default_version = '1.0'
module_pathname = '$libdir/pg_zlib'
relocatable = true
returns text
as 'MODULE_PATHNAME'
language C strict;
create function def(text)
returns bytea
as 'MODULE_PATHNAME'
language C strict;
pg_zlib.control
default_version = '1.0'
module_pathname = '$libdir/pg_zlib'
relocatable = true
pg_zlib.c
#include "postgres.h"
#include "zlib.h"
#define CHUNK 16384
#define DICT_SIZE 32768
PG_MODULE_MAGIC;
text *
def(const bytea *in);
bytea *
inf(const text *in);
extern PGDLLEXPORT void _PG_init(void);
char *DICT;
size_t DICT_LEN;
void
_PG_init(void)
{
#include "postgres.h"
#include "zlib.h"
#define CHUNK 16384
#define DICT_SIZE 32768
PG_MODULE_MAGIC;
text *
def(const bytea *in);
bytea *
inf(const text *in);
extern PGDLLEXPORT void _PG_init(void);
char *DICT;
size_t DICT_LEN;
void
_PG_init(void)
{
//Load dict
}
bytea *
inf(const text *in)
{
}
bytea *
inf(const text *in)
{
//inflate
}
text *
def(const bytea *in)
{
}
text *
def(const bytea *in)
{
//deflate
}
}
sql/pg_zlib.sql
create extension pg_zlib;
Brent Douglas <brent.n.douglas@gmail.com> writes:
> What I have found however is that when I go to load my extensions I get
> this error:
> test=# create extension pg_zlib;
> ERROR: incompatible library "/usr/local/lib/postgresql/pg_zlib.so":
> missing magic block
> HINT: Extension libraries are required to use the PG_MODULE_MAGIC macro.
Yeah, that would be expected if you don't include such a macro.
> And I can't for the life of me work out why. As far as I understand them,
> the docs indicate that this should only really happen if I'm compiling
> against an 8.* headers.
Hm, where are you reading that? I forget when the requirement was added,
but it's certainly never been dropped.
regards, tom lane
Hi Tom,
Ok, I think I miss-understood the docs. I have the macro in the extension anyway, so basically the cause it that somewhere I'm picking up non 9.5.3 libs when I'm compiling? I upgraded from 9.4.5 not too long ago but I can't find any traces of it left on my system to be interfering.
Brent
On Fri, Sep 2, 2016 at 11:07 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Brent Douglas <brent.n.douglas@gmail.com> writes:
> What I have found however is that when I go to load my extensions I get
> this error:
> test=# create extension pg_zlib;
> ERROR: incompatible library "/usr/local/lib/postgresql/pg_zlib.so":
> missing magic block
> HINT: Extension libraries are required to use the PG_MODULE_MAGIC macro.
Yeah, that would be expected if you don't include such a macro.
> And I can't for the life of me work out why. As far as I understand them,
> the docs indicate that this should only really happen if I'm compiling
> against an 8.* headers.
Hm, where are you reading that? I forget when the requirement was added,
but it's certainly never been dropped.
regards, tom lane
Brent Douglas <brent.n.douglas@gmail.com> writes:
> pg_zlib.c:24:1: warning: type specifier missing, defaults to 'int'
> [-Wimplicit-int]
> PG_MODULE_MAGIC;
> ^
> 1 warning generated.
Ah, I'd thought you'd just misread the documentation, but looking closer
I realize that you're probably not understanding the significance of this
warning. I think what's wrong is you have not #include'd "fmgr.h" which
is where PG_MODULE_MAGIC is defined. It's not exactly clear to me why
that's resulting in a warning rather than an error, but certainly this
is indicating a problem.
regards, tom lane
Thanks a lot Tom, it works as expected now. When reading the docs I must have skipped to the code block without properly reading the sentence above.
Brent
On Fri, Sep 2, 2016 at 11:19 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Brent Douglas <brent.n.douglas@gmail.com> writes:
> pg_zlib.c:24:1: warning: type specifier missing, defaults to 'int'
> [-Wimplicit-int]
> PG_MODULE_MAGIC;
> ^
> 1 warning generated.
Ah, I'd thought you'd just misread the documentation, but looking closer
I realize that you're probably not understanding the significance of this
warning. I think what's wrong is you have not #include'd "fmgr.h" which
is where PG_MODULE_MAGIC is defined. It's not exactly clear to me why
that's resulting in a warning rather than an error, but certainly this
is indicating a problem.
regards, tom lane