Thread: Draft release notes for 9.3.2

I've put up draft notes at
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=23e796de15567e9d31e8f9e8661828179f24a7be
and they should be visible on the docs website at
http://www.postgresql.org/docs/devel/static/release-9-3-2.html
in half an hour or so, after guaibasaurus does its next buildfarm run.
I'd like to do any required editing on the notes at this stage,
before I start extracting relevant subsets for the older branches.
So:

* is it useful to go into more detail than this about the data corruption
bugs?  It's not clear to me that we can say more than "vacuum and re-clone
your standbys" as far as recovery actions go, at least not within the
couple of sentences that we traditionally allow for a release note item.

* is there anything important to be said about the fourth and fifth bullet
points ("Fix bugs in setting the visibility-map bit for an empty page"
and "Fix multiple bugs in update chain traversal")?  The commit log
entries these were summarized from didn't spend much breath on saying what
the user-visible consequences were.  Are any of those likely to be sources
of data corruption that would persist past the upgrade?

* any other items that need to be corrected or expanded?
        regards, tom lane

On 2013-12-01 18:56:19 -0500, Tom Lane wrote:
> I've put up draft notes at
> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=23e796de15567e9d31e8f9e8661828179f24a7be
> and they should be visible on the docs website at
> http://www.postgresql.org/docs/devel/static/release-9-3-2.html
> in half an hour or so, after guaibasaurus does its next buildfarm run.
> I'd like to do any required editing on the notes at this stage,
> before I start extracting relevant subsets for the older branches.

When do you plan to backpatch the documentation?

> * is it useful to go into more detail than this about the data corruption
> bugs?  It's not clear to me that we can say more than "vacuum and re-clone
> your standbys" as far as recovery actions go, at least not within the
> couple of sentences that we traditionally allow for a release note item.

I think it might be worth mentioning that (parts) of the data are
potentially recoverable without too much effort in all of the bugs.

> * is there anything important to be said about the fourth and fifth bullet
> points ("Fix bugs in setting the visibility-map bit for an empty page"

I doubt it's worth mentioning that one. Pretty much the only possible
consequence is hitting an Assert() in assert enabled builds in a pretty
rare scenario. There's no data corruption.
Except maybe the fact that the original fix (a6370fd9) wasn't mentioned
in the last round of release notes, and that was more frequent and could
persistently break crash recovery, by erroring out with "PANIC: WAL
contains references to invalid pages" errors.

> and "Fix multiple bugs in update chain traversal")?

4c71f48f4c6c923d1b3f6e6c788e8a0995f4241a
2a4b6eed0916a8fdc57b4778ecb1463fe752d369
663d2e485e4d526770ab618469d875c404714119

These have quite some possible consequences though. Not sure how to
describe it in few words, but it could lead to updating, locking or
returning the wrong row (by following into a different ctid chain),
unneccessary deadlocks (by unneccesarily waiting for an entire
multixact's member, instead of just the updater), missed and superflous
serialization failures in repeatable read and, slightly differently, in
serializable.  All need concurrency to manifest.

In that context, commit fbface6f946242571f4acbfa9a74727c073748ba maybe
should be mentioned as well: spurious "unable to fetch updated version
of tuple" ERRORs.

Not sure whether f5f92bdc44ffdf577244e0d055825cacd0cdea10,
d9484ab5f3cbcfea64536fec333723f9aa4c0b2c shouldn't be mentioned
separately, especially the first could cause autovacuum to crazily spawn
children without ever actually doing anything useful.

> Are any of those likely to be sources of data corruption that would
> persist past the upgrade?

In so far that a query could have updated/returned something wrong,
yes. I don't think we usually list that kind of problem as corruption?

> * any other items that need to be corrected or expanded?
    <para>     Fix <command>VACUUM</>'s tests to see whether it can     update <structfield>relfrozenxid</> (Andres
Freund,Sergey Burladyn,     Jeff Janes)    </para>
 

I think Sergey's and Jeff's fix
(4c697d8f4845823a8af67788b219ffa4516ad14c) deserves its own
headline. The problem there was that a scan_all vacuum could end up not
updating relfrozenxid. That doesn't cause corruption but possibly
superflous further full table/anti-wraparound vacuums or even not
accepting writes anymore outside of single user mode thingy (do we have
a good name for that?). Only affects 9.2+.
    <para>     Fix initialization of <filename>pg_clog</> and     <filename>pg_subtrans</>     during hot standby
startup(Andres Freund)    </para>
 
I think Heikki spent a fair amount of time looking at the code, so it
seems fair to also name him as well..

Maybe we should mention that hot_standby=on is a prerequisite?
    <para>     Truncate <filename>pg_multixact</> contents during crash recovery     (Andres Freund)    </para>
    <para>     This avoids ever-increasing disk space consumption in hot standby     mode.    </para>
It's not really related to hot standby - anything that never comes out
of crash recovery is affected. We sometime should come up with a
coherent name that covers HS, SR, PITR, warm standbys et al...


Should the strerror() improvements be mentioned
(e3480438e89f74019f271b1b5501bb9eed2e0d2a)?


Greetings,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services

Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-12-01 18:56:19 -0500, Tom Lane wrote:
>> I'd like to do any required editing on the notes at this stage,
>> before I start extracting relevant subsets for the older branches.

> When do you plan to backpatch the documentation?

Tomorrow afternoon (my time).
        regards, tom lane

Sergey Burladyan <eshkinkot@gmail.com> writes:
> Is it possible to fix my surname in changelog?
> -Sergey Burladyn
> +Sergey Burladyan

Oh, sorry about that!  I can't do anything about the typo in the commit
log, but certainly we can get it right in the release notes.
        regards, tom lane

Andres Freund <andres@2ndquadrant.com> writes:
> On 2013-12-01 18:56:19 -0500, Tom Lane wrote:
>> * is it useful to go into more detail than this about the data corruption
>> bugs?  It's not clear to me that we can say more than "vacuum and re-clone
>> your standbys" as far as recovery actions go, at least not within the
>> couple of sentences that we traditionally allow for a release note item.

> I think it might be worth mentioning that (parts) of the data are
> potentially recoverable without too much effort in all of the bugs.

I thought about that but was afraid that it'd come off like a commercial
for data recovery services, which didn't seem appropriate.  People who
need this type of service can get advice on the mailing lists, anyway.

>> * is there anything important to be said about the fourth and fifth bullet
>> points ("Fix bugs in setting the visibility-map bit for an empty page"

> I doubt it's worth mentioning that one. Pretty much the only possible
> consequence is hitting an Assert() in assert enabled builds in a pretty
> rare scenario. There's no data corruption.

OK, removed.  We usually mention Assert-fixing commits but this update
certainly has enough other reasons to get installed :-(

>> and "Fix multiple bugs in update chain traversal")?

> These have quite some possible consequences though. Not sure how to
> describe it in few words, but it could lead to updating, locking or
> returning the wrong row (by following into a different ctid chain),
> unneccessary deadlocks (by unneccesarily waiting for an entire
> multixact's member, instead of just the updater), missed and superflous
> serialization failures in repeatable read and, slightly differently, in
> serializable.  All need concurrency to manifest.

I put in a little bit here.

> Not sure whether f5f92bdc44ffdf577244e0d055825cacd0cdea10,
> d9484ab5f3cbcfea64536fec333723f9aa4c0b2c shouldn't be mentioned
> separately, especially the first could cause autovacuum to crazily spawn
> children without ever actually doing anything useful.

Agreed, significant autovacuum activity is a separate symptom, so it
seems worth mentioning separately.

> I think Sergey's and Jeff's fix
> (4c697d8f4845823a8af67788b219ffa4516ad14c) deserves its own
> headline.

Yeah, I had lumped it with the "wrong relfrozenxid accounting" issue
but again the symptom is different.

>      <para>
>       Fix initialization of <filename>pg_clog</> and
>       <filename>pg_subtrans</>
>       during hot standby startup (Andres Freund)
>      </para>
> I think Heikki spent a fair amount of time looking at the code, so it
> seems fair to also name him as well..

Done.

> Maybe we should mention that hot_standby=on is a prerequisite?

Well, it already says hot standby, but I repeated that term in the
body to emphasize it.

>      <para>
>       This avoids ever-increasing disk space consumption in hot standby
>       mode.
>      </para>
> It's not really related to hot standby - anything that never comes out
> of crash recovery is affected. We sometime should come up with a
> coherent name that covers HS, SR, PITR, warm standbys et al...

OK, I said "standby server" instead, which should cover the most
interesting cases.

> Should the strerror() improvements be mentioned
> (e3480438e89f74019f271b1b5501bb9eed2e0d2a)?

I intentionally left that out because it seemed like a reasonable
explanation would take more space than was justified.

Thanks for looking at the notes!
        regards, tom lane

On 2013-12-02 12:26:25 -0500, Tom Lane wrote:
> Andres Freund <andres@2ndquadrant.com> writes:
> > I think it might be worth mentioning that (parts) of the data are
> > potentially recoverable without too much effort in all of the bugs.
> 
> I thought about that but was afraid that it'd come off like a commercial
> for data recovery services, which didn't seem appropriate.  People who
> need this type of service can get advice on the mailing lists, anyway.

Fair argument. Especially when suggested by a person working for a
company providing such services. On that note I'd would take the
following with a grain of salt: I think it's a letting users a bit alone
with their problems. Maybe: "Data might be recoverable."?

Thanks!

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services

On 02 Dec 2013, at 01:56, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> * any other items that need to be corrected or expanded?

I think 2103430 (Fix parsing of xlog file name in pg_receivexlog) is worth mentioning,
as all past pg_receivexlog 9.3.x versions fail to resume interrupted streaming after ~4 GiB
of xlogs have been generated.

Cheers,

--
Mika Eloranta
Ohmu Ltd.  http://www.ohmu.fi/

Tom,

"The issue can be ameliorated by, after upgrading, vacuuming all tables
in all databases while having vacuum_freeze_table_age set to zero. "

Why not say:

"This issue can be ameliorated by, after upgrading, running a
database-wide VACUUM FREEZE."

Or is there a difference in this case?  If so, what?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 2013-12-02 10:51:28 -0800, Josh Berkus wrote:
> Tom,
> 
> "The issue can be ameliorated by, after upgrading, vacuuming all tables
> in all databases while having vacuum_freeze_table_age set to zero. "
> 
> Why not say:
> 
> "This issue can be ameliorated by, after upgrading, running a
> database-wide VACUUM FREEZE."
> 
> Or is there a difference in this case?  If so, what?

vacuum_freeze_table age causes a full table scan, but doesn't freeze
rows younger than vacuum_freeze_min_age. I.e. it finishes much faster.

Greetings,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services

Mika Eloranta <mel@ohmu.fi> writes:
> On 02 Dec 2013, at 01:56, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> * any other items that need to be corrected or expanded?

> I think 2103430 (Fix parsing of xlog file name in pg_receivexlog) is worth mentioning, 
> as all past pg_receivexlog 9.3.x versions fail to resume interrupted streaming after ~4 GiB
> of xlogs have been generated.

OK, I'd misunderstood the severity of that problem.  Item added, thanks!
        regards, tom lane