Re: On columnar storage - Mailing list pgsql-hackers

Tom Lane wrote:
> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > Amit Kapila wrote:
> >> Will the column store obey snapshot model similar to current heap tuples,
> >> if so will it derive the transaction information from heap tuple?
> 
> > Yes, visibility will be tied to the heap tuple -- a value is accessed
> > only when its corresponding heap row has already been determined to be
> > visible.  One interesting point that raises from this is about vacuum:
> > when are we able to remove a value from the store?  I have some
> > not-completely-formed ideas about this.
> 
> Hm.  This seems not terribly ambitious --- mightn't a column store
> extension wish to store tables *entirely* in the column store, rather
> than tying them to a perhaps-vestigial heap table?

Well, yes, it might, but that opens a huge can of worms.  What heapam
offers is not just tuple storage, but a lot of functionality on top of
that -- in particular, tuple locking and visibility.  I am certainly not
considering re-implementing any of that.  We might eventually go there,
but we will *additionally* need different implementations of those
things, and I'm pretty sure that will be painful, so I'm trying to stay
away from that.

> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > Tom Lane wrote:
> >> I can't help thinking that this could tie in with the storage level API
> >> that I was waving my arms about last year.  Or maybe not --- the goals
> >> are substantially different --- but I think we ought to reflect on that
> >> rather than just doing a narrow hack for column stores used in the
> >> particular way you're describing here.
> 
> > I can't seem to remember this proposal you mention.  Care to be more
> > specific?  Perhaps a link to archives is enough.
> 
> I never got to the point of having a concrete proposal, but there was a
> discussion about it at last year's PGCon unconference; were you there?

No, regrettably I wasn't there.

> Anyway the idea was to try to cut a clearer line between heap storage
> and the upper levels of the system, particularly the catalog/DDL code
> that we have so much of.  Based on Salesforce's experience so far,
> it's near impossible to get rid of HeapTuple as the lingua franca for
> representing rows in the upper system levels, so we've not really tried;
> but it would be nice if the DDL code weren't so much in bed with
> heap-specific knowledge, like the wired-into-many-places assumption that
> row insert and update actions require index updates but deletions don't.

Agreed on both counts.  As far as catalog code goes, removing direct
mapping from HeapTuple to C structs would require a huge rewrite of tons
of code.  Unless we're considering rewriting small pieces of specific
catalog handling at a time, it seems unlikely that we will have columns
from system catalogs in column stores.  (It seems reasonable that as
soon as we have column stores, we can have particular catalog code to
work with columnar storage, but I don't think there's much need for that
currently.)

I agree with your second point also --- it might be good to have a layer
in between, and it seems not completely unreasonable.  It would require
touching lots of places but not hugely transforming things.  (I think
it's not in the scope of the things I'm currently after, though.)

> We're also not very happy with the general assumption that a TID is an
> adequate row identifier (as our storage engine does not have TIDs), so
> I'm a bit disappointed to see you doubling down on that restriction
> rather than trying to lift it.

Well, in the general design, there is room for different tuple
identifiers.  I'm just not implementing it for the first version.

> Now much of this pain only comes into play if one is trying to change
> the underlying storage format for system catalogs, which I gather is
> not considered in your proposal.  But if you want one format for catalogs
> and another for user tables then you have issues like how do you guarantee
> atomic commit and crash safety across multiple storage engines.  That way
> lies a mess, especially if you're trying to keep the engines at arms'
> length which is what a pluggable architecture implies.  MySQL is a
> cautionary example we should be keeping in mind while thinking about this.

Right.  I don't want a separate "storage engine" that needs to
reimplement transactions (as is the case in MySQL), or visibility rules.
I don't want to have a different format for tables or catalogs; both
would still be based on the current heapam API.  I simply want to extend
the API so that I can have some columns in a separate place.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services