Re: Variable length varlena headers redux - Mailing list pgsql-hackers

Greg Stark wrote:
> 
> Bruce Momjian <bruce@momjian.us> writes:
> 
> > Bruce Momjian wrote:
> > > 
> > > Uh, I thought the approach was to create type-specific in/out functions,
> > > and add casting so every time there were referenced, they would expand
> > > to a varlena structure in memory.
> 
> Are you talking about actual casts? Because that would lead to all kinds of
> strange places with indexes and function lookups and so on. Or are you just
> talking about code in the macro api to datum?

I am thinking of auto-casts, sort of like how varchar, char, and text
are all internally treated as interchangable.

> > Oh, one more thing.  You are going to need to teach the code that walks
> > through a tuple attributes about the short header types.  I think you
> > should set pg_type.typlen = -3 (vs -1 for varlena) and put your macro
> > code there too.  (As an example, see the macro att_addlength().)
> 
> I thought of doing this. It would let us, for example, treat text/varchar,
> bpchar, and numeric but leave other data types unchanged.
> 
> That does help somewhat but unfortunately text is the problem case. There's
> tons of code that generates text without using textin. All of pgcrypto for
> example.

Well, TEXT can't use short headers.

> > I know it is kind of odd to have a data type that is only used on disk,
> > and not in memory, but I see this as a baby varlena type, used only to
> > store and get varlena values using less disk space.
> 
> I was leaning toward generating the short varlena headers primarily in
> heap_form*tuple and just having the datatype specific code generate 4-byte
> headers much as you describe.

Yep.

> However that doesn't get us away from having VARDATA/VARSIZE aware of the new
> headers. Since heap_deform*tuple and the other entry points which extract
> individual attributes return pointers to the datum in the tuple. They can't
> expand the header to a 4-byte header on the fly.

Yep, you are going to have to special-case those call points to test
typlen and use your short macros there if == -3.

> I thought of doing it in DETOAST_DATUM on the theory that everyone's going to
> be calling it on their arguments. However there are other cases than just
> arguments. Other functions might call, say, text_concat() and then call
> VARDATA() on the result. 

Right, I think all the in-memory stuff has to be varlena.

> Even if we only ever generate short headers on heap_form*tuple and always
> expand them on DETOAST we could have code that passes around tuples that it
> "knows" are entirely in memory and therefore not toasted. I'm thinking of
> plpgsql here primarily. Perhaps it would be enough to outlaw this behaviour
> but it still seems sort of fragile to me.

Yea, we might need some cleanup, but the cleanup is just to do things
properly.  I am unclear on the case you are describing.

--  Bruce Momjian  <bruce@momjian.us>          http://momjian.us EnterpriseDB
http://www.enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +