Re: update on TOAST status' - Mailing list pgsql-hackers

Tom Lane wrote:
> JanWieck@t-online.de (Jan Wieck) writes:
> >     After debugging something I thought was a bug in the toaster,
> >     I've found something really causing headaches.
> >     TOAST AS IS IS NOT CAPABLE OF HOLDING INDEXED VALUES!
> >     It appears that brtee indices (at least) can keep  references
> >     to  old  toast  values  that  survive  a  VACUUM! Seems these
> >     references live in nodes actually not  referring  to  a  heap
> >     tuple   any   more,   but   used  during  tree  traversal  in
> >     comparisions.  As if an index tuple delete from a  btree  not
> >     necessarily  causes  the  index  value  to disappear from the
> >     btree completely.  It'll never be returned by an index  scan,
> >     but the value is still there somewhere.
>
> Oooh, nasty.  Probably the keys you are looking at are in upper-
> level btree pages and indicate the ranges of keys found in lower
> pages, rather than being pointers to real tuples.
   So  our  btree  implementation  is  closer  to  an  ISAM file   organization than to a real tree? Anyway, either one
or  the   other  is  the  reason  that an attempt to insert a new value   results in an lztext_cmp() call that cannot
be resolved  due   to a missing toast value.
 
   I  added  some  checks  to  the  detoaster  just  to throw an   elog(ERROR) instead of a coredump  in  such  a  case
earlier   today.
 

> One answer is to rebuild indexes from scratch during VACUUM,
> before we vacuum the TOAST relation.  We've been talking about
> doing that for a long time.  Maybe it's time to bite the bullet
> and do it.  (Of course that means fixing the relation-versioning
> problem, which it seems we don't have a consensus on yet...)
   Doesn't  matter if we do it before or after, because the main   heap shouldn't contain any more toast references  to
deleted   (later to be vacuumed) toast entries at that time.
 
   Anyway,  it's  a nice idea that should solve the problem. For   indices, which can allways be rebuilt from the heap
data, I   don't  see  such  a  big need for the versioning. Only that a   partially rebuilt index (rebuild crashed in
themiddle) needs   another  vacuum  before  the  the DB is accessible again. How   often does that happen?
 
   So why not having vacuum truncating the index  file  to  zero   and rebuilding it from scratch in place? Can anyone
accessan   index while vacuum has a lock on it's heap?
 

>
> >     Seems  the  designs of btree and toast are colliding. As soon
> >     as "text" is toastable, this'll hurt - be warned.
>
> Text *is* marked toastable in current CVS...
   Whow - haven't noticed.
   Will run my tests against text ... parallel. Does it have any   impact  on the regression test execution time? Does
anytoast   table (that should now be there in the regression  DB)  loose   it's zero size during the tests?
 

>
> >     AFAICS, we need to detoast values for index  inserts  allways
> >     and  have  another toaster inside the index access methods at
> >     some day.  In the meantime we should decide  a  safe  maximum
> >     index tuple size and emit an explanative error message on the
> >     attempt to insert oversized index entries instead of possibly
> >     corrupting the index.
>
> I don't like that --- seems it would put a definite crimp in the
> whole point of TOAST, which is not to have arbitrary limits on field
> sizes.
   If we can solve it, let's do so. If we cannot, let's restrict   it for 7.1.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #