Hello,
First some background:
For my application I need to collect and retrieve data at a very fast rate,
faster than postgres can produce, but I also have the need to store and
access a large range of metadata about the large sets of data I am storing.
I am therefore attempting to use postgres for the metadata in combination
with the HDF5 filesystem for the large sets of data to store data fast and
have all of the power of a relational database over the metadata.
Now onto the problem:
To combine the two sets of data I have created a C function which can be
dynamically loaded into postgres and takes a primary key index into the data
and returns a large table (around 30000 elements) of the data (value and
timestamp). When I access the data directly it takes me around 30ms to just
read the data out of my test case. As the postgres C function is called
many times and I must convert the raw data to tuples it introduces more
overhead bringing the data to take around 2000ms to run the testcase, this
is still acceptable. My problem is when I perform an actual query over my
test case postgres takes around 22000 ms to retrieve the sets of data. My
concern is: what is postgres doing in these extra 20 seconds, and how can I
make this run faster?
In summary:
I have a large testcase.
I am running a C function from postgres which takes a primary key and
returns a setof (int4, int4) with around 30000 tuples returned per call
my c function, when timed over the duration of the test case, takes 2
seconds to run.
the postgres query "select * from function(args);" takes around 11 times
this amount of time to execute
my function is defined as
create or replace function(IN args, IN..., OUT name1 int4, OUT name2 int4)
returns setof record as 'libname.so', 'fcnname' LANGUAGE C IMMUTABLE STRICT;
I'm sorry if this is the wrong forum, I was torn between hackers and
performance.
Thank you for any assistance.
--
View this message in context:
http://www.nabble.com/Postgres-delays-function-returning-large-set-of-data-tp23853886p23853886.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
