Re: Zedstore - compressed in-core columnar storage - Mailing list pgsql-hackers

From Ashutosh Sharma
Subject Re: Zedstore - compressed in-core columnar storage
Date
Msg-id CAE9k0PnDrbmW7MCUCrgszO1ncjUgu3TqcBg-bwScL+rmNvt5+A@mail.gmail.com
Whole thread Raw
In response to Re: Zedstore - compressed in-core columnar storage  (Ashutosh Sharma <ashu.coek88@gmail.com>)
Responses Re: Zedstore - compressed in-core columnar storage
List pgsql-hackers
On Thu, Sep 19, 2019 at 11:35 AM Ashutosh Sharma <ashu.coek88@gmail.com> wrote:
>
> On Thu, Sep 19, 2019 at 8:10 AM Alexandra Wang <lewang@pivotal.io> wrote:
> >
> > On Tue, Sep 17, 2019 at 4:15 AM Ashutosh Sharma <ashu.coek88@gmail.com> wrote:
> >>
> >> create table t1(a int, b int) using zedstore;
> >> insert into t1 select i, i+10 from generate_series(1, 1000000) i;
> >> postgres=# update t1 set b = 200;
> >> server closed the connection unexpectedly
> >> This probably means the server terminated abnormally
> >> before or while processing the request.
> >> The connection to the server was lost. Attempting reset: Failed.
> >>
> >> Above update statement crashed due to some extensive memory leak.
> >
> >
> > Thank you for reporting! We have located the memory leak and also
> > noticed some other memory related bugs. We are working on the fixes
> > please stay tuned!
> >
>
> Cool. As I suspected earlier, it's basically "ZedstoreAMTupleBuffers"
> context that is completely exhausting the memory and it is being used
> to spool the tuples.
>

Some more updates on top of this:

When doing update operation, for each tuple being modified,
*tuplebuffers_insert()* says that there is no entry for the relation
being modified in the hash table although it was already added when
the first tuple in the table was updated. Why is it so? I mean if I
have added an entry in the hash table *tuplebuffers* for let's say
table t1 then should the subsequent call to tuplebuffers_insert() say
that there is no entry for table t1 in the *tuplebuffers*. Shouldn't
that only happen once you have flushed all the tuples in the
tupbuffer->attbuffers. Because of this reason, for each tuple,
tupbuffer->attbuffers is allocated resulting into a lot of memory
consumption. OTOH if the insert is performed on the same table only
for the first tuple tuplebuffers_insert() says that is no entry for
the the table t1 in hash but from the second time onwards that doesn;t
happen. I think because of this reason the memory leak is happening in
case of update operation. Please let me know if I'm missing something
here just because I didn't get chance to spent
much time on this. Thank you.

-- 
With Regards,
Ashutosh Sharma
EnterpriseDB:http://www.enterprisedb.com



pgsql-hackers by date:

Previous
From: Nikolay Shaplov
Date:
Subject: Re: [PATCH] src/test/modules/dummy_index -- way to test reloptions from inside of access method
Next
From: Alexey Kondratov
Date:
Subject: Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace onthe fly