Thread: Patch queue triage

Bruce Momjian <bruce@momjian.us> writes:
> FYI, Tom, Heikki, I need one of you to post the list of patches and
> where we think we are on each one, even if the list is imperfect.

This message is an attempt to sort out which patch queue entries have
no hope of getting into 8.3 (and so we shouldn't spend more time on them
right now), which ones can get in but are waiting on their authors for
more work, and which ones are ready for immediate review.

You'll notice that numerically quite a lot of the patches fall into the
"dead" category.  This isn't actually all that surprising because if
they were apply-able they'd have gotten in.  (It's not like we've
completely neglected applying patches for the last six months...)
However, many of the remaining live patches are huge ones, like HOT and
delayed commit, and are going to consume considerable review effort
(again, if they didn't, they might have been in already).

The bottom line is that we are desperately in need of more review
manpower.  I'm pleased to report that Heikki Linnakangas has promised to
devote most of the next few weeks to helping review, and has already
picked out some patches he feels qualified to work on, as you'll see
below.  I have also been coordinating reviewing assignments off-list with
Neil and Magnus to avoid duplication of effort.  But I'd like to encourage
everyone who's done any backend hacking to look at the live patches not
shown as assigned to someone specific.  The more eyeballs the better;
anything you catch is something someone else might miss.  Also, several
of the open patches are in need of more performance testing, so if you
can help out in that fashion please do so.

With that, the patches:


* Re: [pgsql-patches] [PATCHES] [HACKERS] [Fwd: Index Advisor]      /Gurjeet Singh/

I don't think this can be applied in anything like its current state.
The internal interface to the planner is not very good, and ditto for the
user API.  What I would suggest trying to do is get a set of plugin hooks
defined for the planner that would allow the advisor to be implemented
entirely as an external module, and then let it be worked on as a
pgfoundry project.  I have some ideas about appropriate design of the
plugin hooks (as mentioned in my review message of a month ago), but I
have to say that getting that done for 8.3 is lower-priority than some of
the other patches.

* [pgsql-patches] Ctid chain following enhancement      /Pavan Deolasee/

I'm not very excited about this --- it seems to me to complicate the code
in some places that are not in fact performance-critical.  While it
doesn't seem likely to break things, I'm not in favor of reducing code
readability unless a measurable performance improvement can be shown.
Can we have some tests showing this is worthwhile?

* [PATCHES] Error correction for n_dead_tuples      /ITAGAKI Takahiro/

Waiting for OldestXmin patch to be accepted or rejected.  However,
regardless of the fate of that patch, I'm concerned that this one creates
an open-loop behavior in which the n_dead_tuples estimate will diverge
arbitrarily far from reality over time.  I criticized the original
proposal on that basis, and I'm not convinced this version fixes it,
because of the fact that stats counter updates occur much later than the
actions they count.  (My recent patch to rate-limit tabstat messages made
that problem worse, but it existed anyway.)  What might make sense is for
vacuum to count the number of dead-but-not-removable tuples it skips over,
and apply that as the value of n_dead_tuples on receipt of the vacuum
message (instead of setting to zero as now).  This is likely to be wrong
with respect to the actions of transactions running concurrently with the
vacuum, but I think so is the proposed patch; and at least in this form
the error certainly cannot accumulate across vacuum cycles.

* [PATCHES] Table function support  /Pavel Stehule/

Neil has promised to review this.  AFAIK there are no showstoppers but
there are some disagreements on the details of the functionality.

* [HACKERS] Grouped Index Tuples  /Heikki Linnakangas/
* [HACKERS] Grouped Index Tuples / Clustered Indexes      /Heikki Linnakangas/

Needs review.  I'm not sure how many people besides Heikki have really
looked at this (I know I haven't).

* Re: [PATCHES] POSIX shared memory support      /Chris Marcellino/

I'm not convinced that we want this at all.  The original proposal was
to provide an alternative for platforms without SysV shmem support,
but the working versions of the patch fail to remove the SysV dependency,
and the portability of this code is itself quite unproven.  Do we really
want to take on maintenance of yet-another shmem implementation just to
let people be lazy about changing their SHMMAX settings?  As best I can
tell the main problem in that department is Darwin, and it'd be a whole
lot simpler if Apple just raised their darn default SHMMAX to something
that's sensible for the 21st century.

[BTW, there was a patch submitted recently to use SysV semaphores instead
of Posix on Darwin versions that have 'em; why isn't that in the queue?
It makes sense to me to do that to save on file descriptors ...]

* [HACKERS] Automatic adjustment of bgwriter_lru_maxpages (was:     Dead Space Map version 2)  /ITAGAKI Takahiro/

Heikki is reviewing this.  AIUI there are no known showstoppers but
we'd like more performance measurements, plus there are interactions
with other open patches to investigate.

* [HACKERS] Stream bitmaps  /Heikki Linnakangas/

I think this is on hold unless a finished bitmap-index patch shows up;
however there was some discussion of using this in support of clustered
indexes, so maybe it's live anyway?  Heikki?

* [PATCHES] Heap page diagnostic/test functions (v2)      /Simon Riggs/

Heikki is reviewing this and refactoring it as a contrib module.

* [PATCHES] non-recursive WITH clause support      /Gregory Stark/

I think the consensus is that we should wait for a more complete WITH
implementation to be submitted, since this one creates compatibility
issues without any real return in the form of added functionality.

* Re: [HACKERS] Bug: Buffer cache is not scan resistant      /Simon Riggs/

Heikki is reviewing this one, in conjunction with the sync-scan patch.
Performance testing is needed for both.

* [PATCHES] Updated bitmap index patch  /Gavin Sherry/

We're still waiting to see if a finished patch will be submitted.
I'm very close to saying this has to wait for 8.4, though.

* Re: [HACKERS] [PATCHES] Bitmapscan changes      /Heikki Linnakangas/

I had objected to this on the grounds that it seemed to be covering
only a narrow territory between HOT and bitmap indexes, but given the
probability that one or even both of those won't make it, we need to
give this one a second look.  So: live, needs review.

* Re: [PATCHES] Synchronized Scan WIP patch      /Simon Riggs/

Heikki is reviewing this one.  Also I believe Greg Smith is doing more
performance testing.

* Re: [HACKERS] [PATCHES] xml2 contrib patch supporting default XML     namespaces  /Mike Rylander/

I would like to get Peter to deal with this and the other XML-related
patch.

* [HACKERS] Indexam interface proposal  /Heikki Linnakangas/

AFAICS this discussion died out with no actual patch submitted.

* [PATCHES] Load distributed checkpoint V3      /ITAGAKI Takahiro/

Heikki is reviewing this one.  More performance testing is needed,
I think.

* [HACKERS] Group Commit  /Heikki Linnakangas/

I believe this is on hold for 8.4; in any case it should wait till
after Simon's deferred-commit patch is in and we have more info about
the performance implications of that.

* [PATCHES] Patch for circular buffer in tuplestore to optimize     merge joins (v1)  /stark/

I will handle this --- AFAIK there are no showstopper objections.

* [PATCHES] Concurrent psql v4 [WIP]  /stark/

This is waiting on the author to change the API per list discussions; we
can't apply something that is about to have some commands removed ...

* Re: [PATCHES] Recalculating OldestXmin in a long-running vacuum      /Heikki Linnakangas/

This is on hold until the other vacuum-related patches are accepted
or rejected; at that point we need to redo the performance tests.
As things stand it should *not* be accepted because the only available
test shows a performance decrease, but maybe that will change after
the other patches go in.  (In any case it'd be interesting to find
out why there's no performance improvement, but it seems unlikely
we have time for that investigation for 8.3.)

* Re: [HACKERS] Modifying TOAST thresholds  /Tom Lane/

At this point it seems nothing will be done about this issue for 8.3.

* [PATCHES] plpgpsm  /Pavel Stehule/

I think this has to be held for 8.4: it was submitted too late for the 8.3
feature deadline, and in fact I don't recall that there was any community
discussion/consensus on accepting it into core at all.  Another big
problem is that it largely copies-and-pastes the plpgsql code, which I
think is an unacceptable maintenance burden in the long run.  We need to
consider whether we can't refactor to avoid code duplication.

* [HACKERS] tsearch_core patch for inclusion      /Teodor Sigaev/

Have we resolved whether the API and the dump/restore strategy are
acceptable?  The code needs review too, but not till we've settled
on the basic question whether we like the feature set.

* Re: [PATCHES] [HACKERS] Full page writes improvement, code update      /Koichi Suzuki/

I feel that we have to insist that this be modified to not create any WAL
bloat in the pre-compression form.  That will be more efficient and will
avoid the whole argument about whether a switch is needed.  There was also
doubt about whether the external programs (pg_compresslog etc) were ready
for prime time.  At this point we could accept a patch that makes whatever
small tweaks are needed to ensure that an incremental-format WAL record
can be extracted from a full-page-write record, at least for the most
common WAL record types for which compression is actually important.  I
think the actual compression/decompression programs could undergo further
development on pgfoundry with an eye to merging them into core before 8.4
release.

* [PATCHES] Dead Space Map version 3 (simplified)      /ITAGAKI Takahiro/

I want to reject this in its current form.  It's been clear almost since
the FSM code was first released to the field that its dependence on a
fixed-size shared memory area was a serious design blunder (one for which
I take full blame; I thought it'd be workable, I was wrong).  We can't
sanely take a patch that repeats that mistake.  It seems fairly clear that
DSM could be redesigned to be stored in "special space" in every Nth heap
page and thereby avoid the problem.  We should do that and look at whether
FSM couldn't be integrated into the same data structure.  (I'm not
entirely sure how to handle FSM for indexes in this fashion, but maybe we
should just throw that problem back onto the shoulders of the index AMs;
for instance make them keep linked lists of free index pages.)
Obviously this won't happen for 8.3, but I don't think DSM as it stands
would be a net plus for users.

* Re: [PATCHES] COPY-able sql log outputs  /FAST PostgreSQL/

Are we agreed on the API for this?  AFAIR there were no showstoppers
but there was some considerable discussion about API details.

* [PATCHES] Updateable cursors patch  /FAST PostgreSQL/

This is incomplete, and I fear at this point has to be held over to 8.4.

* [PATCHES] Preliminary GSSAPI Patches  /Henry B. Hotz/

Magnus is reviewing this one.

* [PATCHES] Logging checkpoints and other slowdown causes      /Greg Smith/

Heikki is reviewing this one.

* Re: [PATCHES] [Fwd: Deferred Transactions, Transaction Guarantee     and COMMITwithout waiting]  /Simon Riggs/

Simon is on the hook to submit an updated patch.  I hope this one
makes it in, as it looks like a really nice performance improvement
for those who can use it; but the original patch seems overcomplicated.

* Re: [HACKERS] Arrays of Complex Types  /David Fetter/

Waiting on the author to respond to open issues; in particular the
problem of not making array types for system catalogs.

* Re: [HACKERS] Implicit casts to text  /Peter Eisentraut/

I had volunteered to finish this, but don't be surprised if it gets
put off till 8.4.

* Re: [GENERAL] dropping role w/dependent objects      /Ed L./

Patch seems ok, comment maybe overkill.  Will apply this shortly.

* Re: [HACKERS] pg_index updates and SI invalidation      /Pavan Deolasee/

Needs some review, I don't think there are any showstoppers though.

* [PATCHES] HOT Patch - Ready for review  /Pavan Deolasee/

This needs a *lot* of review.  Can we break it down into more manageable
chunks?  I'm not sure that anyone's got a full grasp of the implications
of this patch, and that's a scary thought.

* [PATCHES] Minor recovery changes  /Simon Riggs/

The submission mentions open issues --- when will those be resolved?
Should we apply the patch-so-far anyway?

* Re: [PATCHES] UTF8MatchText  /ITAGAKI Takahiro/

I'd like someone who uses UTF8 more than I do to double-check whether this
is sane.  Also, can we avoid having so much code duplication between the
various forms of LIKE?

* [PATCHES] Clear up strxfrm() in UTF-8 with locale on Windows      /ITAGAKI Takahiro/

Someone else needs to look at this; I can't test it.  Magnus?

* Re: [PATCHES] xpath_array with namespaces support      /Nikolay Samokhvalov/

I would like to get Peter to deal with this and the other XML-related
patch.

* Re: [PATCHES] CREATE TABLE LIKE INCLUDING INDEXES support      /Trevor Hardcastle/

Neil will review this.

* [PORTS] m32r tas() implementation  /Martin Pitt/

This lacks a configure probe for sys/tas.h, but I have no other
objection to it.

* [HACKERS] [PATCH] A crash and subsequent recovery of the master     can cause theslave to get out-of-sync  /Florian
G.Pflug/
 

This seems like a real problem that needs review.  The given patch
was done hastily and might or might not be OK, though.

* [COMMITTERS] pgsql: Remove some of the most blatant brain-fade in     the recent guc  /Tom Lane/

The recent guc patch to reset variables when they are removed from
postgresql.conf was broken.  Again :-(.  It needs to be fixed
or reverted.  (Why are we having such problems here?  Maybe the
GUC code as a whole is too complex to understand?)

* [HACKERS] Avoiding unnecessary reads in recovery      /Heikki Linnakangas/

I think we need to apply this, or else do something else with the
wired-in assumption about full page writes that was noted in the
discussion.

* Re: [PATCHES] [HACKERS] autovacuum does not start in HEAD      /ITAGAKI Takahiro/

I assume Alvaro will deal with this.

* Re: [PATCHES] LIMIT/SORT optimization  /Gregory Stark/
* [PATCHES] updated SORT/LIMIT patch  /Gregory Stark/

I will look at this.  I recall being concerned about whether there
wasn't a cleaner way to introduce the required inter-node communication.

* Re: [PATCHES] New version of GENERATED/IDENTITY, was Re: parser     dilemma  /Zoltan Boszormenyi/

I reviewed this earlier and can deal with the update.


There is one large item that is not in the patch queue at all, because
it's not been written, and that's the change to support per-function
settings of search_path.  I think this is critical to get done for 8.3,
but adding a column to pg_proc will create merge issues for any patch that
adds anything to pg_proc, and a lot of the open ones do.  So I've been
holding off writing it until more of them are in.  I think it's going to
be straightforward so I'm not too worried about the delay, but it needs to
happen.
        regards, tom lane

On Tue, 1 May 2007, Tom Lane wrote:

> * Re: [PATCHES] Synchronized Scan WIP patch
>      /Simon Riggs/
> Heikki is reviewing this one.  Also I believe Greg Smith is doing more
> performance testing.

Actually it was the "Automatic adjustment of bgwriter_lru_maxpages" patch 
from Itagaki Takahiro I've been working on--which, by the way, Bruce sent 
out a note about the other day saying was already booted into 8.4.  I've 
integrated a part of the LRU patch usefully into the new pg_stat_bgwriter, 
renamed some variables around accordingly, cleanup I felt had to preceed 
performance testing.  Heikki might want to wait for me to finish that 
before putting time into it.  The buffer allocation monitoring part would 
be useful to apply even if the GUC and behavior changes are deemed outside 
of the 8.3 scope.

Also, I think I was the most vocal participant in the "COPY-able sql log 
outputs" feature discussion and therefore was planning to do what review 
what I can of the final patch API immediately afterwards.  I try not to 
complain about things unless I'm willing to help fix them.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Greg Smith <gsmith@gregsmith.com> writes:
> On Tue, 1 May 2007, Tom Lane wrote:
>> * Re: [PATCHES] Synchronized Scan WIP patch
>> /Simon Riggs/
>> Heikki is reviewing this one.  Also I believe Greg Smith is doing more
>> performance testing.

> Actually it was the "Automatic adjustment of bgwriter_lru_maxpages" patch 
> from Itagaki Takahiro I've been working on--which, by the way, Bruce sent 
> out a note about the other day saying was already booted into 8.4.

It's still on the patch queue list, and Heikki and I were assuming it
was still live.

> I've 
> integrated a part of the LRU patch usefully into the new pg_stat_bgwriter, 
> renamed some variables around accordingly, cleanup I felt had to preceed 
> performance testing.  Heikki might want to wait for me to finish that 
> before putting time into it.  The buffer allocation monitoring part would 
> be useful to apply even if the GUC and behavior changes are deemed outside 
> of the 8.3 scope.

> Also, I think I was the most vocal participant in the "COPY-able sql log 
> outputs" feature discussion and therefore was planning to do what review 
> what I can of the final patch API immediately afterwards.  I try not to 
> complain about things unless I'm willing to help fix them.

Great, you're on the hook for both of those ;-)
        regards, tom lane

On Tue, 1 May 2007, Tom Lane wrote:
> * [HACKERS] tsearch_core patch for inclusion
>      /Teodor Sigaev/
>
> Have we resolved whether the API and the dump/restore strategy are
> acceptable?  The code needs review too, but not till we've settled
> on the basic question whether we like the feature set.
>

There were several discussion and we tried to satisfy a claims.
We added the ability of creation of FTS with one command
(just create GIN index on text attribute), which many 
people like a lot, we changed SQL syntax to be more SQL-ish and we like it.
We discussed the new FTS on two conferences in Moscow and got positive 
opinions. Also, we wrote FTSBOOK
( http://mira.sai.msu.su/~megera/pgsql/ftsdoc/ ) in english and FTS introduction in russian
( http://www.sai.msu.su/~megera/postgres/talks/fts_pgsql_intro.html ).

For the period from the patch was announced there were > 3100 unique IP, which
downloaded FTSBOOK.
    Regards,        Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

> * [PATCHES] Updateable cursors patch  /FAST PostgreSQL/
> 
> This is incomplete, and I fear at this point has to be held over to 8.4.
> 

It is true that my original patch post said that I need to modify the 
patch to work with tidscan. Since then I have realized that this 
modification is not needed as it would have the same result as the 
'branching out from sequential scan' solution currently implemented.

I was hoping that I could discuss this with whoever picks up the patch 
for review before doing modifications if any is needed. So in my humble 
opinion, it would be great if this can be considered for 8.3 as there 
are not many modifications needed.

P.S. Only Simon commented on my original patch. Simon, do you have time 
to revisit the patch so that we could discuss this?

Rgds,
Arul Shaji

"Pavan Deolasee" <pavan.deolasee@gmail.com> writes:

> On 5/2/07, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>
>> This needs a *lot* of review.  Can we break it down into more manageable
>> chunks?  
>
> Sure, we can do that. I actually did that when I posted the
> incremental versions of the HOT-patch, each version implementing
> the next big chunk of the code. I can reverse engineer that again.

Can we? I mean, sure you can break the patch up into chunks which might make
it easier to read, but are any of the chunks useful alone?

I suppose inserting HOT tuples without index maintenance is useful even if no
changes to the space allocation is made is useful. It won't get the space
usage but it would save on index thrashing. But that still implies all the
code to handle scans, updates, index builds, etc. Those chunks could be
separated out but you can't commit without them.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com

"Tom Lane" <tgl@sss.pgh.pa.us> writes:

> Bruce Momjian <bruce@momjian.us> writes:
>> FYI, Tom, Heikki, I need one of you to post the list of patches and
>> where we think we are on each one, even if the list is imperfect.
>
> This message is an attempt to sort out which patch queue entries have
> no hope of getting into 8.3 (and so we shouldn't spend more time on them
> right now), which ones can get in but are waiting on their authors for
> more work, and which ones are ready for immediate review.

Thanks for this. This is exactly what we've been missing recently I think.

> * [PATCHES] non-recursive WITH clause support 
>       /Gregory Stark/
>
> I think the consensus is that we should wait for a more complete WITH
> implementation to be submitted, since this one creates compatibility
> issues without any real return in the form of added functionality.

I submitted it because it filled in a missing ANSI feature but nobody's piped
up saying they missed that feature so, sure.

> * [PATCHES] Concurrent psql v4 [WIP]  /stark/
>
> This is waiting on the author to change the API per list discussions; we
> can't apply something that is about to have some commands removed ...

I did finish the api changes -- though I'm not too happy with the names. I was
expecting the list to play the name game so I just put in placeholder names
originally. I'm adding documentation and example regression tests now. 

Also I'm trying to fix the cursor-mode FETCH_COUNT support which it breaks.
I'm thinking of once the first batch of rows arrives just going into a
synchronous function to fetch the rest of the resultsets.

> * Re: [HACKERS] Modifying TOAST thresholds  /Tom Lane/
>
> At this point it seems nothing will be done about this issue for 8.3.

I'm not sure anyone has an idea how to test it. TPCC isn't really useful
because it has a fixed size (500 byte) string buffer. Perhaps if we modified
it to have a random string length uniformly distributed between 0-2k ? But
even then it never does any scans based on that buffer. But the problem with
going with something more natural is that it'll be harder to tell exactly what
it's testing.

> * [HACKERS] tsearch_core patch for inclusion 
>       /Teodor Sigaev/
>
> Have we resolved whether the API and the dump/restore strategy are
> acceptable?  The code needs review too, but not till we've settled
> on the basic question whether we like the feature set.

Personally I actually do like the idea of promoting tsearch to first-class
citizen by giving it keywords and a place in the grammar. I think it's a more
fully integrated interface than the function based one. The only reason I
might think otherwise was if it was just a crutch for missing features it had
exposed that would be better implemented more generically. But I don't think
that's the case.

> * Re: [PATCHES] [Fwd: Deferred Transactions, Transaction Guarantee
>       and COMMITwithout waiting]  /Simon Riggs/
>
> Simon is on the hook to submit an updated patch.  I hope this one
> makes it in, as it looks like a really nice performance improvement
> for those who can use it; but the original patch seems overcomplicated.

I know Simon's really busy. I might be able to help with it if he wants.

> * Re: [PATCHES] LIMIT/SORT optimization  /Gregory Stark/
> * [PATCHES] updated SORT/LIMIT patch  /Gregory Stark/
>
> I will look at this.  I recall being concerned about whether there
> wasn't a cleaner way to introduce the required inter-node communication.

The next big thing to keep in mind in this area is a unique sort which would
need to know if there's a Unique node above it with the same key. If the
resulting inter-node communication arrangement has your blessing and can
handle that as well then I'll probably do that for 8.4.

Incidentally I would prefer it if you want to make changes that you explain
the changes to me and let me make them. It gives me a better chance to
understand what the changes really are and the motivation than trying to read
a patch later and understand why you made the changes you did. I understand
sometimes it's easier to just write the code than to explain the idea to
someone else and then review the resulting code though and there's already
enough work your plate so if that's the case then so be it.


--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com

On Tue, May 01, 2007 at 10:44:38PM -0400, Tom Lane wrote:

> * [PATCHES] Preliminary GSSAPI Patches  /Henry B. Hotz/
> 
> Magnus is reviewing this one.

Check.

> * [PATCHES] Clear up strxfrm() in UTF-8 with locale on Windows
>        /ITAGAKI Takahiro/
> 
> Someone else needs to look at this; I can't test it.  Magnus?

Yup, it's on my list.


//Magnus

Gregory Stark wrote:
> "Tom Lane" <tgl@sss.pgh.pa.us> writes:
>
> > Bruce Momjian <bruce@momjian.us> writes:
> >> FYI, Tom, Heikki, I need one of you to post the list of patches and
> >> where we think we are on each one, even if the list is imperfect.
> >
> > This message is an attempt to sort out which patch queue entries have
> > no hope of getting into 8.3 (and so we shouldn't spend more time on them
> > right now), which ones can get in but are waiting on their authors for
> > more work, and which ones are ready for immediate review.
>
> Thanks for this. This is exactly what we've been missing recently I think.

100% agree.

> > * Re: [HACKERS] Modifying TOAST thresholds  /Tom Lane/
> >
> > At this point it seems nothing will be done about this issue for 8.3.
>
> I'm not sure anyone has an idea how to test it. TPCC isn't really useful
> because it has a fixed size (500 byte) string buffer. Perhaps if we modified
> it to have a random string length uniformly distributed between 0-2k ? But
> even then it never does any scans based on that buffer. But the problem with
> going with something more natural is that it'll be harder to tell exactly what
> it's testing.

My idea on this was to create two backends, one with the default TOAST
value, and a second with a value of 50 bytes.  Create a table with one
TEXT field, and several other columns, each column < 50 bytes.

Then, fill the table with random data (script attached that might help),
and the try 2000, 1500, 1000, etc, bytes in the TEXT column for each row
(use random data so the compression code doesn't shrink it).  Then run a
test with both backends acessing the TEXT column and non-TEXT column and
measure the difference between the two backends, i.e. the backend with a
TOAST value of 50 should show faster access on the non-TEXT field, but
slower access on the TEXT field.

Then, figure out where the gains on the non-TEXT field seem to diminish
in usefulness.  Basically, with a lower TOAST value, we are going to
spend more time accessing the TEXT field, but the speedup for the
non-TEXT field should be large enough win that we don't care. As the
TEXT column becomes shorter, it has less affect on the non-TEXT access.

--
  Bruce Momjian  <bruce@momjian.us>          http://momjian.us
  EnterpriseDB                               http://www.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +
:

ROWS=100000
COLSIZE=2500

echo "
    DROP TABLE test;
    CREATE TABLE test(i SERIAL, t text);
    INSERT INTO test (t)
    SELECT    array_to_string(ARRAY(
        SELECT chr(32 + (95 * random())::integer)
        FROM generate_series(1,$COLSIZE)),'')
    FROM    generate_series(1, $ROWS);
    SELECT     pg_relation_size('test') AS "HEAP",
        pg_total_relation_size('test') - pg_relation_size('test') AS "TOAST";
    SET log_min_duration_statement = 0;
    SET client_min_messages = 'log';
    SELECT t FROM test WHERE i = 234329823;" |
sql test

"Bruce Momjian" <bruce@momjian.us> writes:

> Then, figure out where the gains on the non-TEXT field seem to diminish
> in usefulness.  Basically, with a lower TOAST value, we are going to
> spend more time accessing the TEXT field, but the speedup for the
> non-TEXT field should be large enough win that we don't care. As the
> TEXT column becomes shorter, it has less affect on the non-TEXT access.

I guess the key is to break down what it is we want to measure into several
parts. These can each be measured precisely for various sized TOASTed data.

Costs:

1) cost of retrieving data from TOAST pointer versus retrieving data from  inline tuple. We just want the absolute time
differencebetween the two  operations, not the percentage difference.
 

2) cost of creating TOAST pointer (ie, inserting a new tuple with a TOAST  pointer or updating a previously inlined
tupleto have a TOASTed column). 
 

3) cost of deleting a TOAST pointer (ie, deleting a tuple or updating a tuple  to no longer have a TOASTed column)

3) cost of deleting a tuple with an existing TOAST pointer (or updating a  tuple to be all inlined) versus deleting an
plaintuple or updating a plain  tuple.
 

Savings:

1) time savings accessing a tuple without retrieving the TOAST pointer versus  having to access the tuple with the data
inlined.

2) time savings updating a tuple without modifying toasted data versus  updating same tuple with the data inlined in
bothversions.
 

The plan you described would be testing costs 1 and savings 1 but I think we
need to continue to the others as well.

Then the trick is to somehow make some argument about the frequency of the
various operations and the acceptable tradeoff. I think you're right that the
time spent accessing the data would be the most important metric.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com

Another complexity is testing cases where the table fits in shared
memory/RAM, and those that don't, and testing cases where heap fits in
RAM only because we pushed things to TOAST, and cases where we have to
do lots of TOAST access that doesn't fit in RAM.  I wonder if it is even
worth testing it because pushing more to TOAST probably means the more
frequently accessed data is in RAM.

Anyway, how is going to run these tests?

---------------------------------------------------------------------------

Gregory Stark wrote:
> 
> "Bruce Momjian" <bruce@momjian.us> writes:
> 
> > Then, figure out where the gains on the non-TEXT field seem to diminish
> > in usefulness.  Basically, with a lower TOAST value, we are going to
> > spend more time accessing the TEXT field, but the speedup for the
> > non-TEXT field should be large enough win that we don't care. As the
> > TEXT column becomes shorter, it has less affect on the non-TEXT access.
> 
> I guess the key is to break down what it is we want to measure into several
> parts. These can each be measured precisely for various sized TOASTed data.
> 
> Costs:
> 
> 1) cost of retrieving data from TOAST pointer versus retrieving data from
>    inline tuple. We just want the absolute time difference between the two
>    operations, not the percentage difference.
> 
> 2) cost of creating TOAST pointer (ie, inserting a new tuple with a TOAST
>    pointer or updating a previously inlined tuple to have a TOASTed column). 
> 
> 3) cost of deleting a TOAST pointer (ie, deleting a tuple or updating a tuple
>    to no longer have a TOASTed column)
> 
> 3) cost of deleting a tuple with an existing TOAST pointer (or updating a
>    tuple to be all inlined) versus deleting an plain tuple or updating a plain
>    tuple.
> 
> Savings:
> 
> 1) time savings accessing a tuple without retrieving the TOAST pointer versus
>    having to access the tuple with the data inlined.
> 
> 2) time savings updating a tuple without modifying toasted data versus
>    updating same tuple with the data inlined in both versions.
> 
> The plan you described would be testing costs 1 and savings 1 but I think we
> need to continue to the others as well.
> 
> Then the trick is to somehow make some argument about the frequency of the
> various operations and the acceptable tradeoff. I think you're right that the
> time spent accessing the data would be the most important metric.
> 
> -- 
>   Gregory Stark
>   EnterpriseDB          http://www.enterprisedb.com
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend

--  Bruce Momjian  <bruce@momjian.us>          http://momjian.us EnterpriseDB
http://www.enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
>   
>> FYI, Tom, Heikki, I need one of you to post the list of patches and
>> where we think we are on each one, even if the list is imperfect.
>>     
>
> This message is an attempt to sort out which patch queue entries have
> no hope of getting into 8.3 (and so we shouldn't spend more time on them
> right now), which ones can get in but are waiting on their authors for
> more work, and which ones are ready for immediate review.
>   

Excellent list.

I have a little time available now, so I'll work on some, starting with 
trying to complete arrays of complex types.

cheers

andrew

Tom Lane wrote:
> * [pgsql-patches] Ctid chain following enhancement
>        /Pavan Deolasee/
> 
> I'm not very excited about this --- it seems to me to complicate the code
> in some places that are not in fact performance-critical.  While it
> doesn't seem likely to break things, I'm not in favor of reducing code
> readability unless a measurable performance improvement can be shown.
> Can we have some tests showing this is worthwhile?

IIRC this patch was originally part of an old HOT patch, and it was 
submitted as a separate patch because it has some benefit on its own but 
more importantly getting it applied first would make the HOT patch 
slightly smaller. I'm not sure if the latest HOT patch requires or 
includes this change anymore. If not we should drop this. If it does, 
then let's deal with this before attacking the hard core of HOT.

> * [HACKERS] Grouped Index Tuples  /Heikki Linnakangas/
> * [HACKERS] Grouped Index Tuples / Clustered Indexes
>        /Heikki Linnakangas/
> 
> Needs review.  I'm not sure how many people besides Heikki have really
> looked at this (I know I haven't).

The patch is ugly as it is. We need API changes to make it less ugly, I 
had hoped to discuss and reach consensus on them well before feature 
freeze, that's what the "indexam API proposal" and "Stream bitmaps" 
threads in the patch queue are all about. But those discussions and 
patches stalled, so the clustered index patch is still in the same ugly 
state.

I'm afraid we're getting "past due" on clustered indexes. The patch 
isn't ready for committing as it is, and we still don't have agreement 
on the API changes or even on the design in general. :(

> * [HACKERS] Stream bitmaps  /Heikki Linnakangas/
> 
> I think this is on hold unless a finished bitmap-index patch shows up;
> however there was some discussion of using this in support of clustered
> indexes, so maybe it's live anyway?  Heikki?

This particular thread is closely related to bitmap indexes. But see 
next item:

> * Re: [HACKERS] [PATCHES] Bitmapscan changes 
>       /Heikki Linnakangas/
> 
> I had objected to this on the grounds that it seemed to be covering
> only a narrow territory between HOT and bitmap indexes, but given the
> probability that one or even both of those won't make it, we need to
> give this one a second look.  So: live, needs review.

Are you talking about the patch I submitted at the beginning of that 
thread? Because the mail in the patch queue is actually about whether or 
not we want clustered indexes.

I think the original "bitmapscan changes" patch I submitted is live and 
needs review, even if clustered indexes and bitmap indexes are rejected. 
It should give some performance benefit when you do bitmap ANDs with 
partially lossy bitmaps, and from setting bits directly in the bitmap in 
the indexam in one call, instead of calling amgetmulti many times. 
Though I never measured that.

> * [HACKERS] Indexam interface proposal  /Heikki Linnakangas/
> 
> AFAICS this discussion died out with no actual patch submitted.

This is part of clustered indexes.. This was my proposal of what the 
indexam API changes would be like. This patch is either live or dead 
together with clustered indexes.

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com

http://www.sigaev.ru/misc/tsearch_core-0.46.gz
Patch is synced with current CVS HEAD and synced with bugfixes in contrib/tsearch2



-- 
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
  WWW: http://www.sigaev.ru/
 

On Tue, 2007-05-01 at 22:44 -0400, Tom Lane wrote:

Nice summary Tom.

> * Re: [PATCHES] [Fwd: Deferred Transactions, Transaction Guarantee
>       and COMMITwithout waiting]  /Simon Riggs/
> 
> Simon is on the hook to submit an updated patch.  I hope this one
> makes it in, as it looks like a really nice performance improvement
> for those who can use it; but the original patch seems overcomplicated.

It will, but it will be next week now. Changes agreed though.

> * [PATCHES] Minor recovery changes  /Simon Riggs/
> 
> The submission mentions open issues --- when will those be resolved?
> Should we apply the patch-so-far anyway?

Patch is OK to go in as-is. There are open issues, but they need more
thought and discussion and I regret won't happen in a reasonable
timescale due to other work.

--  Simon Riggs              EnterpriseDB   http://www.enterprisedb.com

Sorry for late responce due to very long vacation season in Japan.

Tom Lane wrote:

> > * Re: [PATCHES] [HACKERS] Full page writes improvement, code update
> >        /Koichi Suzuki/
> >
> > I feel that we have to insist that this be modified to not create
any WAL
> > bloat in the pre-compression form.  That will be more efficient and will
> > avoid the whole argument about whether a switch is needed.  There
was also
> > doubt about whether the external programs (pg_compresslog etc) were
ready
> > for prime time.  At this point we could accept a patch that makes
whatever
> > small tweaks are needed to ensure that an incremental-format WAL record
> > can be extracted from a full-page-write record, at least for the most
> > common WAL record types for which compression is actually important.  I
> > think the actual compression/decompression programs could undergo
further
> > development on pgfoundry with an eye to merging them into core
before 8.4
> > release.

As suggested by Tom, I agree that WAL should not include "both" full
page write and incremental (logical) log.   I began to examine WAL
record format to see if incremental log can be made from full page
writes.   It will be okay even before 8.4, if simplified patch to the
core is accepted.   I will post simplified patch to the core as follows:

1. Mark the flag to indicate that the WAL record is compressable from
full page writes to incremental log.  This flag will be set if
a) It is not written during the hot backup, and
b) Archive command is active, and
c) WAL contains full page writes, and
d) full_page_writes=on.
No logical log will be written to WAL in this case, and
2. During recovery, xl_tot_len check will be skipped for compressed WAL
records.

With this patch, compress/decompress can be developped outside the core.I'd like to post them to PG foundary first for
thereview for 8.4.
 

I'd be very grateful if this patch can be considered again.

Best Regards;

------------
Koichi Suzuki

Tom Lane wrote:

> At this point it seems nothing will be done about this issue for 8.3.
> 
> * [PATCHES] plpgpsm  /Pavel Stehule/
> 
> I think this has to be held for 8.4: it was submitted too late for the 8.3
> feature deadline, and in fact I don't recall that there was any community
> discussion/consensus on accepting it into core at all.  Another big
> problem is that it largely copies-and-pastes the plpgsql code, which I
> think is an unacceptable maintenance burden in the long run.  We need to
> consider whether we can't refactor to avoid code duplication.

Per my updated patch email to the lists yesterday, plus Toms' above 
comments *and* Alvaro's comments to me when I asked Alexey to review 
it... may I strongly suggest that we bounce this for further development 
in 8.4.

Joshua D. Drake





> 



-- 
      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive  PostgreSQL solutions since 1997             http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Pavan Deolasee wrote:

> 
>     I suppose inserting HOT tuples without index maintenance is useful
>     even if no
>     changes to the space allocation is made is useful. It won't get the
>     space
>     usage but it would save on index thrashing. But that still implies
>     all the
>     code to handle scans, updates, index builds, etc. Those chunks could be
>     separated out but you can't commit without them.
> 
> 
> 
> There are few things that we can separate easily, like CREATE INDEX
> related changes, VACUUM and VACUUM FULL related changed, space
> reuse related changes etc. Let me give it a shot.

Did we ever get a broken up patch for this?

Joshua D. Drake


> 
> Thanks,
> Pavan
> 
> 
> -- 
> 
> EnterpriseDB     http://www.enterprisedb.com


-- 
      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive  PostgreSQL solutions since 1997             http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Joshua D. Drake wrote:
> Tom Lane wrote:
> 
> > At this point it seems nothing will be done about this issue for 8.3.
> > 
> > * [PATCHES] plpgpsm  /Pavel Stehule/
> > 
> > I think this has to be held for 8.4: it was submitted too late for the 8.3
> > feature deadline, and in fact I don't recall that there was any community
> > discussion/consensus on accepting it into core at all.  Another big
> > problem is that it largely copies-and-pastes the plpgsql code, which I
> > think is an unacceptable maintenance burden in the long run.  We need to
> > consider whether we can't refactor to avoid code duplication.
> 
> Per my updated patch email to the lists yesterday, plus Toms' above 
> comments *and* Alvaro's comments to me when I asked Alexey to review 
> it... may I strongly suggest that we bounce this for further development 
> in 8.4.

Agreed.  Done.

--  Bruce Momjian  <bruce@momjian.us>          http://momjian.us EnterpriseDB
http://www.enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Joshua D. Drake wrote:
> Pavan Deolasee wrote:
>>
>>     I suppose inserting HOT tuples without index maintenance is useful
>>     even if no
>>     changes to the space allocation is made is useful. It won't get the
>>     space
>>     usage but it would save on index thrashing. But that still implies
>>     all the
>>     code to handle scans, updates, index builds, etc. Those chunks 
>> could be
>>     separated out but you can't commit without them.
>>
>> There are few things that we can separate easily, like CREATE INDEX
>> related changes, VACUUM and VACUUM FULL related changed, space
>> reuse related changes etc. Let me give it a shot.
> 
> Did we ever get a broken up patch for this?

Yes:

http://archives.postgresql.org/pgsql-patches/2007-05/msg00065.php

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com

Heikki Linnakangas wrote:

>>> There are few things that we can separate easily, like CREATE INDEX
>>> related changes, VACUUM and VACUUM FULL related changed, space
>>> reuse related changes etc. Let me give it a shot.
>>
>> Did we ever get a broken up patch for this?
> 
> Yes:
> 
> http://archives.postgresql.org/pgsql-patches/2007-05/msg00065.php

Thanks :).

Sincerely,

Joshua D. Drake





-- 
      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive  PostgreSQL solutions since 1997             http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

For those who have forgotten the progress we have made toward 8.3, here
are the open patches we had for 8.3 as of May 1, 2006:

---------------------------------------------------------------------------

Tom Lane wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> > FYI, Tom, Heikki, I need one of you to post the list of patches and
> > where we think we are on each one, even if the list is imperfect.
> 
> This message is an attempt to sort out which patch queue entries have
> no hope of getting into 8.3 (and so we shouldn't spend more time on them
> right now), which ones can get in but are waiting on their authors for
> more work, and which ones are ready for immediate review.
> 
> You'll notice that numerically quite a lot of the patches fall into the
> "dead" category.  This isn't actually all that surprising because if
> they were apply-able they'd have gotten in.  (It's not like we've
> completely neglected applying patches for the last six months...)
> However, many of the remaining live patches are huge ones, like HOT and
> delayed commit, and are going to consume considerable review effort
> (again, if they didn't, they might have been in already).
> 
> The bottom line is that we are desperately in need of more review
> manpower.  I'm pleased to report that Heikki Linnakangas has promised to
> devote most of the next few weeks to helping review, and has already
> picked out some patches he feels qualified to work on, as you'll see
> below.  I have also been coordinating reviewing assignments off-list with
> Neil and Magnus to avoid duplication of effort.  But I'd like to encourage
> everyone who's done any backend hacking to look at the live patches not
> shown as assigned to someone specific.  The more eyeballs the better;
> anything you catch is something someone else might miss.  Also, several
> of the open patches are in need of more performance testing, so if you
> can help out in that fashion please do so.
> 
> With that, the patches:
> 
> 
> * Re: [pgsql-patches] [PATCHES] [HACKERS] [Fwd: Index Advisor]
>        /Gurjeet Singh/
> 
> I don't think this can be applied in anything like its current state.
> The internal interface to the planner is not very good, and ditto for the
> user API.  What I would suggest trying to do is get a set of plugin hooks
> defined for the planner that would allow the advisor to be implemented
> entirely as an external module, and then let it be worked on as a
> pgfoundry project.  I have some ideas about appropriate design of the
> plugin hooks (as mentioned in my review message of a month ago), but I
> have to say that getting that done for 8.3 is lower-priority than some of
> the other patches.
> 
> * [pgsql-patches] Ctid chain following enhancement
>        /Pavan Deolasee/
> 
> I'm not very excited about this --- it seems to me to complicate the code
> in some places that are not in fact performance-critical.  While it
> doesn't seem likely to break things, I'm not in favor of reducing code
> readability unless a measurable performance improvement can be shown.
> Can we have some tests showing this is worthwhile?
> 
> * [PATCHES] Error correction for n_dead_tuples 
>       /ITAGAKI Takahiro/
> 
> Waiting for OldestXmin patch to be accepted or rejected.  However,
> regardless of the fate of that patch, I'm concerned that this one creates
> an open-loop behavior in which the n_dead_tuples estimate will diverge
> arbitrarily far from reality over time.  I criticized the original
> proposal on that basis, and I'm not convinced this version fixes it,
> because of the fact that stats counter updates occur much later than the
> actions they count.  (My recent patch to rate-limit tabstat messages made
> that problem worse, but it existed anyway.)  What might make sense is for
> vacuum to count the number of dead-but-not-removable tuples it skips over,
> and apply that as the value of n_dead_tuples on receipt of the vacuum
> message (instead of setting to zero as now).  This is likely to be wrong
> with respect to the actions of transactions running concurrently with the
> vacuum, but I think so is the proposed patch; and at least in this form
> the error certainly cannot accumulate across vacuum cycles.
> 
> * [PATCHES] Table function support  /Pavel Stehule/
> 
> Neil has promised to review this.  AFAIK there are no showstoppers but
> there are some disagreements on the details of the functionality.
> 
> * [HACKERS] Grouped Index Tuples  /Heikki Linnakangas/
> * [HACKERS] Grouped Index Tuples / Clustered Indexes
>        /Heikki Linnakangas/
> 
> Needs review.  I'm not sure how many people besides Heikki have really
> looked at this (I know I haven't).
> 
> * Re: [PATCHES] POSIX shared memory support 
>       /Chris Marcellino/
> 
> I'm not convinced that we want this at all.  The original proposal was
> to provide an alternative for platforms without SysV shmem support,
> but the working versions of the patch fail to remove the SysV dependency,
> and the portability of this code is itself quite unproven.  Do we really
> want to take on maintenance of yet-another shmem implementation just to
> let people be lazy about changing their SHMMAX settings?  As best I can
> tell the main problem in that department is Darwin, and it'd be a whole
> lot simpler if Apple just raised their darn default SHMMAX to something
> that's sensible for the 21st century.
> 
> [BTW, there was a patch submitted recently to use SysV semaphores instead
> of Posix on Darwin versions that have 'em; why isn't that in the queue?
> It makes sense to me to do that to save on file descriptors ...]
> 
> * [HACKERS] Automatic adjustment of bgwriter_lru_maxpages (was:
>       Dead Space Map version 2)  /ITAGAKI Takahiro/
> 
> Heikki is reviewing this.  AIUI there are no known showstoppers but
> we'd like more performance measurements, plus there are interactions
> with other open patches to investigate.
> 
> * [HACKERS] Stream bitmaps  /Heikki Linnakangas/
> 
> I think this is on hold unless a finished bitmap-index patch shows up;
> however there was some discussion of using this in support of clustered
> indexes, so maybe it's live anyway?  Heikki?
> 
> * [PATCHES] Heap page diagnostic/test functions (v2)
>        /Simon Riggs/
> 
> Heikki is reviewing this and refactoring it as a contrib module.
> 
> * [PATCHES] non-recursive WITH clause support 
>       /Gregory Stark/
> 
> I think the consensus is that we should wait for a more complete WITH
> implementation to be submitted, since this one creates compatibility
> issues without any real return in the form of added functionality.
> 
> * Re: [HACKERS] Bug: Buffer cache is not scan resistant
>        /Simon Riggs/
> 
> Heikki is reviewing this one, in conjunction with the sync-scan patch.
> Performance testing is needed for both.
> 
> * [PATCHES] Updated bitmap index patch  /Gavin Sherry/
> 
> We're still waiting to see if a finished patch will be submitted.
> I'm very close to saying this has to wait for 8.4, though.
> 
> * Re: [HACKERS] [PATCHES] Bitmapscan changes 
>       /Heikki Linnakangas/
> 
> I had objected to this on the grounds that it seemed to be covering
> only a narrow territory between HOT and bitmap indexes, but given the
> probability that one or even both of those won't make it, we need to
> give this one a second look.  So: live, needs review.
> 
> * Re: [PATCHES] Synchronized Scan WIP patch 
>       /Simon Riggs/
> 
> Heikki is reviewing this one.  Also I believe Greg Smith is doing more
> performance testing.
> 
> * Re: [HACKERS] [PATCHES] xml2 contrib patch supporting default XML
>       namespaces  /Mike Rylander/
> 
> I would like to get Peter to deal with this and the other XML-related
> patch.
> 
> * [HACKERS] Indexam interface proposal  /Heikki Linnakangas/
> 
> AFAICS this discussion died out with no actual patch submitted.
> 
> * [PATCHES] Load distributed checkpoint V3 
>       /ITAGAKI Takahiro/
> 
> Heikki is reviewing this one.  More performance testing is needed,
> I think.
> 
> * [HACKERS] Group Commit  /Heikki Linnakangas/
> 
> I believe this is on hold for 8.4; in any case it should wait till
> after Simon's deferred-commit patch is in and we have more info about
> the performance implications of that.
> 
> * [PATCHES] Patch for circular buffer in tuplestore to optimize
>       merge joins (v1)  /stark/
> 
> I will handle this --- AFAIK there are no showstopper objections.
> 
> * [PATCHES] Concurrent psql v4 [WIP]  /stark/
> 
> This is waiting on the author to change the API per list discussions; we
> can't apply something that is about to have some commands removed ...
> 
> * Re: [PATCHES] Recalculating OldestXmin in a long-running vacuum
>        /Heikki Linnakangas/
> 
> This is on hold until the other vacuum-related patches are accepted
> or rejected; at that point we need to redo the performance tests.
> As things stand it should *not* be accepted because the only available
> test shows a performance decrease, but maybe that will change after
> the other patches go in.  (In any case it'd be interesting to find
> out why there's no performance improvement, but it seems unlikely
> we have time for that investigation for 8.3.)
> 
> * Re: [HACKERS] Modifying TOAST thresholds  /Tom Lane/
> 
> At this point it seems nothing will be done about this issue for 8.3.
> 
> * [PATCHES] plpgpsm  /Pavel Stehule/
> 
> I think this has to be held for 8.4: it was submitted too late for the 8.3
> feature deadline, and in fact I don't recall that there was any community
> discussion/consensus on accepting it into core at all.  Another big
> problem is that it largely copies-and-pastes the plpgsql code, which I
> think is an unacceptable maintenance burden in the long run.  We need to
> consider whether we can't refactor to avoid code duplication.
> 
> * [HACKERS] tsearch_core patch for inclusion 
>       /Teodor Sigaev/
> 
> Have we resolved whether the API and the dump/restore strategy are
> acceptable?  The code needs review too, but not till we've settled
> on the basic question whether we like the feature set.
> 
> * Re: [PATCHES] [HACKERS] Full page writes improvement, code update
>        /Koichi Suzuki/
> 
> I feel that we have to insist that this be modified to not create any WAL
> bloat in the pre-compression form.  That will be more efficient and will
> avoid the whole argument about whether a switch is needed.  There was also
> doubt about whether the external programs (pg_compresslog etc) were ready
> for prime time.  At this point we could accept a patch that makes whatever
> small tweaks are needed to ensure that an incremental-format WAL record
> can be extracted from a full-page-write record, at least for the most
> common WAL record types for which compression is actually important.  I
> think the actual compression/decompression programs could undergo further
> development on pgfoundry with an eye to merging them into core before 8.4
> release.
> 
> * [PATCHES] Dead Space Map version 3 (simplified) 
>       /ITAGAKI Takahiro/
> 
> I want to reject this in its current form.  It's been clear almost since
> the FSM code was first released to the field that its dependence on a
> fixed-size shared memory area was a serious design blunder (one for which
> I take full blame; I thought it'd be workable, I was wrong).  We can't
> sanely take a patch that repeats that mistake.  It seems fairly clear that
> DSM could be redesigned to be stored in "special space" in every Nth heap
> page and thereby avoid the problem.  We should do that and look at whether
> FSM couldn't be integrated into the same data structure.  (I'm not
> entirely sure how to handle FSM for indexes in this fashion, but maybe we
> should just throw that problem back onto the shoulders of the index AMs;
> for instance make them keep linked lists of free index pages.)
> Obviously this won't happen for 8.3, but I don't think DSM as it stands
> would be a net plus for users.
> 
> * Re: [PATCHES] COPY-able sql log outputs  /FAST PostgreSQL/
> 
> Are we agreed on the API for this?  AFAIR there were no showstoppers
> but there was some considerable discussion about API details.
> 
> * [PATCHES] Updateable cursors patch  /FAST PostgreSQL/
> 
> This is incomplete, and I fear at this point has to be held over to 8.4.
> 
> * [PATCHES] Preliminary GSSAPI Patches  /Henry B. Hotz/
> 
> Magnus is reviewing this one.
> 
> * [PATCHES] Logging checkpoints and other slowdown causes
>        /Greg Smith/
> 
> Heikki is reviewing this one.
> 
> * Re: [PATCHES] [Fwd: Deferred Transactions, Transaction Guarantee
>       and COMMITwithout waiting]  /Simon Riggs/
> 
> Simon is on the hook to submit an updated patch.  I hope this one
> makes it in, as it looks like a really nice performance improvement
> for those who can use it; but the original patch seems overcomplicated.
> 
> * Re: [HACKERS] Arrays of Complex Types  /David Fetter/
> 
> Waiting on the author to respond to open issues; in particular the
> problem of not making array types for system catalogs.
> 
> * Re: [HACKERS] Implicit casts to text  /Peter Eisentraut/
> 
> I had volunteered to finish this, but don't be surprised if it gets
> put off till 8.4.
> 
> * Re: [GENERAL] dropping role w/dependent objects 
>       /Ed L./
> 
> Patch seems ok, comment maybe overkill.  Will apply this shortly.
> 
> * Re: [HACKERS] pg_index updates and SI invalidation
>        /Pavan Deolasee/
> 
> Needs some review, I don't think there are any showstoppers though.
> 
> * [PATCHES] HOT Patch - Ready for review  /Pavan Deolasee/
> 
> This needs a *lot* of review.  Can we break it down into more manageable
> chunks?  I'm not sure that anyone's got a full grasp of the implications
> of this patch, and that's a scary thought.
> 
> * [PATCHES] Minor recovery changes  /Simon Riggs/
> 
> The submission mentions open issues --- when will those be resolved?
> Should we apply the patch-so-far anyway?
> 
> * Re: [PATCHES] UTF8MatchText  /ITAGAKI Takahiro/
> 
> I'd like someone who uses UTF8 more than I do to double-check whether this
> is sane.  Also, can we avoid having so much code duplication between the
> various forms of LIKE?
> 
> * [PATCHES] Clear up strxfrm() in UTF-8 with locale on Windows
>        /ITAGAKI Takahiro/
> 
> Someone else needs to look at this; I can't test it.  Magnus?
> 
> * Re: [PATCHES] xpath_array with namespaces support
>        /Nikolay Samokhvalov/
> 
> I would like to get Peter to deal with this and the other XML-related
> patch.
> 
> * Re: [PATCHES] CREATE TABLE LIKE INCLUDING INDEXES support
>        /Trevor Hardcastle/
> 
> Neil will review this.
> 
> * [PORTS] m32r tas() implementation  /Martin Pitt/
> 
> This lacks a configure probe for sys/tas.h, but I have no other
> objection to it.
> 
> * [HACKERS] [PATCH] A crash and subsequent recovery of the master
>       can cause theslave to get out-of-sync  /Florian G. Pflug/
> 
> This seems like a real problem that needs review.  The given patch
> was done hastily and might or might not be OK, though.
> 
> * [COMMITTERS] pgsql: Remove some of the most blatant brain-fade in
>       the recent guc  /Tom Lane/
> 
> The recent guc patch to reset variables when they are removed from
> postgresql.conf was broken.  Again :-(.  It needs to be fixed
> or reverted.  (Why are we having such problems here?  Maybe the
> GUC code as a whole is too complex to understand?)
> 
> * [HACKERS] Avoiding unnecessary reads in recovery
>        /Heikki Linnakangas/
> 
> I think we need to apply this, or else do something else with the
> wired-in assumption about full page writes that was noted in the
> discussion.
> 
> * Re: [PATCHES] [HACKERS] autovacuum does not start in HEAD
>        /ITAGAKI Takahiro/
> 
> I assume Alvaro will deal with this.
> 
> * Re: [PATCHES] LIMIT/SORT optimization  /Gregory Stark/
> * [PATCHES] updated SORT/LIMIT patch  /Gregory Stark/
> 
> I will look at this.  I recall being concerned about whether there
> wasn't a cleaner way to introduce the required inter-node communication.
> 
> * Re: [PATCHES] New version of GENERATED/IDENTITY, was Re: parser
>       dilemma  /Zoltan Boszormenyi/
> 
> I reviewed this earlier and can deal with the update.
> 
> 
> There is one large item that is not in the patch queue at all, because
> it's not been written, and that's the change to support per-function
> settings of search_path.  I think this is critical to get done for 8.3,
> but adding a column to pg_proc will create merge issues for any patch that
> adds anything to pg_proc, and a lot of the open ones do.  So I've been
> holding off writing it until more of them are in.  I think it's going to
> be straightforward so I'm not too worried about the delay, but it needs to
> happen.
> 
>             regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
> 
>                http://www.postgresql.org/docs/faq

--  Bruce Momjian  <bruce@momjian.us>          http://momjian.us EnterpriseDB
http://www.enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

On Wed, 2007-09-12 at 17:47 -0400, Bruce Momjian wrote:
> For those who have forgotten the progress we have made toward 8.3, here
> are the open patches we had for 8.3 as of May 1, 2006:

Could you please issue a list of open items for 8.3?

I want to check whether you are waiting on me for anything and which
things have been deferred to next release.

Thanks,

--  Simon Riggs 2ndQuadrant  http://www.2ndQuadrant.com

Pavan Deolasee wrote:
> On 9/13/07, Bruce Momjian <bruce@momjian.us> wrote:
> >
> >
> > For those who have forgotten the progress we have made toward 8.3, here
> > are the open patches we had for 8.3 as of May 1, 2006:
> >
> >
> You mean May 1, 2007 ;-)

Yea, sorry.

--  Bruce Momjian  <bruce@momjian.us>          http://momjian.us EnterpriseDB
http://www.enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +

Simon Riggs wrote:
> On Wed, 2007-09-12 at 17:47 -0400, Bruce Momjian wrote:
> > For those who have forgotten the progress we have made toward 8.3, here
> > are the open patches we had for 8.3 as of May 1, 2006:
> 
> Could you please issue a list of open items for 8.3?
> 
> I want to check whether you are waiting on me for anything and which
> things have been deferred to next release.

I am working on putting them all in the patch queue now.

--  Bruce Momjian  <bruce@momjian.us>          http://momjian.us EnterpriseDB
http://www.enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +