Re: Postgres delays function returning large set of data - Mailing list pgsql-hackers

Hello

can you send source code? There are two types of C SRF functions. One
returns row ro by row, second store returned rows and returns block.
What did you use?

ignore comments (in czech please) and look on code -
http://www.postgres.cz/index.php/Iter%C3%A1tor_pole

regards
Pavel Stehule

2009/6/3 flippo00110001 <ddevec@umich.edu>:
>
> Hello,
>
> First some background:
> For my application I need to collect and retrieve data at a very fast rate,
> faster than postgres can produce, but I also have the need to store and
> access a large range of metadata about the large sets of data I am storing.
> I am therefore attempting to use postgres for the metadata in combination
> with the HDF5 filesystem for the large sets of data to store data fast and
> have all of the power of a relational database over the metadata.
>
> Now onto the problem:
> To combine the two sets of data I have created a C function which can be
> dynamically loaded into postgres and takes a primary key index into the data
> and returns a large table (around 30000 elements) of the data (value and
> timestamp).  When I access the data directly it takes me around 30ms to just
> read the data out of my test case.  As the postgres C function is called
> many times and I must convert the raw data to tuples it introduces more
> overhead bringing the data to take around 2000ms to run the testcase, this
> is still acceptable.  My problem is when I perform an actual query over my
> test case postgres takes around 22000 ms to retrieve the sets of data.  My
> concern is: what is postgres doing in these extra 20 seconds, and how can I
> make this run faster?
>
> In summary:
> I have a large testcase.
> I am running a C function from postgres which takes a primary key and
> returns a setof (int4, int4) with around 30000 tuples returned per call
> my c function, when timed over the duration of the test case, takes 2
> seconds to run.
> the postgres query "select * from function(args);" takes around 11 times
> this amount of time to execute
>
> my function is defined as
> create or replace function(IN args, IN..., OUT name1 int4, OUT name2 int4)
> returns setof record as 'libname.so', 'fcnname' LANGUAGE C IMMUTABLE STRICT;
>
> I'm sorry if this is the wrong forum, I was torn between hackers and
> performance.
> Thank you for any assistance.
>
> --
> View this message in context:
http://www.nabble.com/Postgres-delays-function-returning-large-set-of-data-tp23853886p23853886.html
> Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>