Request a Demo
Home   >   mailing lists

Re: In-Memory Columnar Store - Mailing list pgsql-hackers

From desmodemone
Subject Re: In-Memory Columnar Store
Date
Msg-id CAEs9oFn920CSw_0k+TTa79cdF6zQC+TdHz3xwvbdXmXa_iEMZQ@mail.gmail.com
Whole thread Raw
In response to In-Memory Columnar Store  (knizhnik <knizhnik@garret.ru>)
Responses Re: In-Memory Columnar Store
List pgsql-hackers
Tree view



2013/12/9 knizhnik <knizhnik@garret.ru>
Hello!

I want to annouce my implementation of In-Memory Columnar Store extension for PostgreSQL:

     Documentation: http://www.garret.ru/imcs/user_guide.html
     Sources: http://www.garret.ru/imcs-1.01.tar.gz

Any feedbacks, bug reports and suggestions are welcome.

Vertical representation of data is stored in PostgreSQL shared memory.
This is why it is important to be able to utilize all available physical memory.
Now servers with Tb or more RAM are not something exotic, especially in financial world.
But there is limitation in Linux with standard 4kb pages  for maximal size of mapped memory segment: 256Gb.
It is possible to overcome this limitation either by creating multiple segments - but it requires too much changes in PostgreSQL memory manager.
Or just set MAP_HUGETLB flag (assuming that huge pages were allocated in the system).

I found several messages related with MAP_HUGETLB flag, the most recent one was from 21 of November:
http://www.postgresql.org/message-id/20131125032920.GA23793@toroid.org

I wonder what is the current status of this patch?






--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Hello,
           excellent work! I begin to do testing and it's very fast, by the way I found a strange case of "endless" query with CPU a 100%  when the value used as filter does not exists:

I am testing with postgres 9.3.1 on debian  and I used default value for the extension except memory ( 512mb )

how to recreate the test case :

## create a table :

create table endless ( col1 int , col2 char(30) , col3 int ) ;

## insert some values:

insert into endless values ( 1, 'ahahahaha', 3);

insert into endless values ( 2, 'ghghghghg', 4);

## create the column store objects:

select cs_create('endless','col1','col2');
 cs_create
-----------
 
(1 row)

## try and test column store :

select cs_avg(col3) from  endless_get('ahahahaha');
 cs_avg
--------
      3
(1 row)

select cs_avg(col3) from  endless_get('ghghghghg');
 cs_avg
--------
      4
(1 row)

## now select with a value that does not exist :

select cs_avg(col3) from  endless_get('testing');

# and now  start to loop on cpu and seems to never ends , I had to terminate backend

Bye

Mat

pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: Reference to parent query from ANY sublink
Next
From: Andres Freund
Date:
Subject: Re: logical changeset generation v6.8