Thread: BYTEA / DBD::Pg change in 9.0 beta

BYTEA / DBD::Pg change in 9.0 beta

From
Jesper Krogh
Date:
Hi.

I'm trying to do a test move of one of our applications onto 9.0beta1.
We use storable and serializes data into a bytea column in the database.
This script uses that:

#!/usr/bin/perl
use strict;
use warnings;
use Storable;
use DBI;
use DBD::Pg;
use Data::Dumper;

my $dbh = DBI->connect("dbi:Pg:dbname=testdb","","",{AutoCommit => 1});
my $sql = <<END
create table testtable (id serial, testbytea bytea);
END
;

eval {
$dbh->do($sql);
};
$dbh->do("delete from testtable");
my $href = { this => "1", that => "2"};
print "Before: " . Dumper($href) . "\n";
my $sth = $dbh->prepare("insert into testtable (testbytea) values (?)");
my $frozen = Storable::nfreeze($href);
$sth->bind_param(1, $frozen, { pg_type=>DBD::Pg::PG_BYTEA });
$sth->execute;
$sth = $dbh->prepare("select testbytea from testtable");
$sth->execute();
my $row = $sth->fetchrow_hashref();
my $href2 = Storable::thaw($row->{testbytea});

print Dumper($href2);


Running it against 8.4 gives:

$ perl bin/test-bytea
NOTICE:  CREATE TABLE will create implicit sequence "testtable_id_seq1" 
for serial column "testtable.id"
DBD::Pg::db do failed: ERROR:  relation "testtable" already exists at 
bin/efam/test-bytea line 16.
Before: $VAR1 = {          'that' => '2',          'this' => '1'        };

$VAR1 = {          'that' => '2',          'this' => '1'        };

Whereas 9.0beta1 gives:
$ perl bin/test-bytea
NOTICE:  CREATE TABLE will create implicit sequence "testtable_id_seq1" 
for serial column "testtable.id"
DBD::Pg::db do failed: ERROR:  relation "testtable" already exists at 
bin/efam/test-bytea line 16.
Before: $VAR1 = {          'that' => '2',          'this' => '1'        };

Storable binary image v60.48 more recent than I am (v2.7) at 
../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/thaw.al) 
line 366, at bin/test-bytea line 28

Inspecting the data seems that it is the insert that does something to 
the data:

8.4 id |                                       testbytea
----+----------------------------------------------------------------------------------------  9 | 
\005\007\003\000\000\000\002\012\0012\000\000\000\004that\012\0011\000\000\000\004this
(1 row)

9.0beta1 id |                          testbytea
----+--------------------------------------------------------------  3 |
\x050703000000020a013200000004746861740a01310000000474686973
(1 row)


Jesper


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Alex Hunsaker
Date:
On Tue, May 18, 2010 at 14:54, Jesper Krogh <jesper@krogh.cc> wrote:
> Hi.
>
> I'm trying to do a test move of one of our applications onto 9.0beta1.
> We use storable and serializes data into a bytea column in the database.
> [ snip insert/select using bytea ]

> 8.4
>  id |                                       testbytea
> ----+----------------------------------------------------------------------------------------
>  9 |
> \005\007\003\000\000\000\002\012\0012\000\000\000\004that\012\0011\000\000\000\004this
> (1 row)
>
> 9.0beta1
>  id |                          testbytea
> ----+--------------------------------------------------------------
>  3 | \x050703000000020a013200000004746861740a01310000000474686973
> (1 row)


Try adding $db->do("set bytea_output 'escape';"); as 9.0 defaults to
hex encoding.  Id bet DBD::Pg does not account for that yet.


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Jesper Krogh
Date:
On 2010-05-18 23:12, Alex Hunsaker wrote:
> set bytea_output 'escape';

That was it. Knowing what the problem was I had no problem finding it in 
the release notes.

May I ask whats the reason is for "breaking" the compatibillity?

-- 
Jesper


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Alex Hunsaker
Date:
On Tue, May 18, 2010 at 15:20, Jesper Krogh <jesper@krogh.cc> wrote:
> On 2010-05-18 23:12, Alex Hunsaker wrote:
>>
>> set bytea_output 'escape';
>
> That was it. Knowing what the problem was I had no problem finding it in the
> release notes.
>
> May I ask whats the reason is for "breaking" the compatibillity?

There were a couple IIRC, the big ones being speed and size.  Id look
at the archives for more.

I imagine at some point DBD::Pg will handle this transparently.  I
also imagine Greg would happily accept patches :-)


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Bernd Helmle
Date:

--On 18. Mai 2010 23:20:26 +0200 Jesper Krogh <jesper@krogh.cc> wrote:

> That was it. Knowing what the problem was I had no problem finding it in
> the release notes.
>
> May I ask whats the reason is for "breaking" the compatibillity?

"Efficency", if i am allowed to call it this way. The new hex 
representation should be more efficient to retrieve and to handle than the 
old one. I think bytea_output was set to hex for testing purposes on the 
first hand, but not sure wether there was a consensus to leave it there 
finally later.

-- 
Thanks
Bernd


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Kenneth Marshall
Date:
On Tue, May 18, 2010 at 03:26:17PM -0600, Alex Hunsaker wrote:
> On Tue, May 18, 2010 at 15:20, Jesper Krogh <jesper@krogh.cc> wrote:
> > On 2010-05-18 23:12, Alex Hunsaker wrote:
> >>
> >> set bytea_output 'escape';
> >
> > That was it. Knowing what the problem was I had no problem finding it in the
> > release notes.
> >
> > May I ask whats the reason is for "breaking" the compatibillity?
> 
> There were a couple IIRC, the big ones being speed and size.  Id look
> at the archives for more.
> 
> I imagine at some point DBD::Pg will handle this transparently.  I
> also imagine Greg would happily accept patches :-)
> 

Yes, the new format is much faster, more space efficient, and uses
less CPU to do the encoding. The older format caused the COPY for
bytea to be CPU limited in many more situations.

Regards,
Ken


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Tom Lane
Date:
Bernd Helmle <mailings@oopsware.de> writes:
> --On 18. Mai 2010 23:20:26 +0200 Jesper Krogh <jesper@krogh.cc> wrote:
>> May I ask whats the reason is for "breaking" the compatibillity?

> "Efficency", if i am allowed to call it this way. The new hex 
> representation should be more efficient to retrieve and to handle than the 
> old one. I think bytea_output was set to hex for testing purposes on the 
> first hand, but not sure wether there was a consensus to leave it there 
> finally later.

Yeah, we intentionally set it that way initially to help find stuff that
needs to be updated (as DBD::Pg evidently does).  It's still TBD whether
9.0.0 will ship with that default or not.
        regards, tom lane


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Stefan Kaltenbrunner
Date:
On 05/19/2010 08:13 AM, Tom Lane wrote:
> Bernd Helmle <mailings@oopsware.de> writes:
>> --On 18. Mai 2010 23:20:26 +0200 Jesper Krogh <jesper@krogh.cc> wrote:
>>> May I ask whats the reason is for "breaking" the compatibillity?
> 
>> "Efficency", if i am allowed to call it this way. The new hex 
>> representation should be more efficient to retrieve and to handle than the 
>> old one. I think bytea_output was set to hex for testing purposes on the 
>> first hand, but not sure wether there was a consensus to leave it there 
>> finally later.
> 
> Yeah, we intentionally set it that way initially to help find stuff that
> needs to be updated (as DBD::Pg evidently does).  It's still TBD whether
> 9.0.0 will ship with that default or not.

given how much faster the new format is (or rather how slow the old one
was) and the number of people I have seen complaining "why is bytea so
slow) I would like to see it staying turned on by default. However this
also depends on how quickly database driver developers can adapt.



Stefan


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Robert Haas
Date:
On Wed, May 19, 2010 at 10:17 AM, Stefan Kaltenbrunner
<stefan@kaltenbrunner.cc> wrote:
> On 05/19/2010 08:13 AM, Tom Lane wrote:
>> Bernd Helmle <mailings@oopsware.de> writes:
>>> --On 18. Mai 2010 23:20:26 +0200 Jesper Krogh <jesper@krogh.cc> wrote:
>>>> May I ask whats the reason is for "breaking" the compatibillity?
>>
>>> "Efficency", if i am allowed to call it this way. The new hex
>>> representation should be more efficient to retrieve and to handle than the
>>> old one. I think bytea_output was set to hex for testing purposes on the
>>> first hand, but not sure wether there was a consensus to leave it there
>>> finally later.
>>
>> Yeah, we intentionally set it that way initially to help find stuff that
>> needs to be updated (as DBD::Pg evidently does).  It's still TBD whether
>> 9.0.0 will ship with that default or not.
>
> given how much faster the new format is (or rather how slow the old one
> was) and the number of people I have seen complaining "why is bytea so
> slow) I would like to see it staying turned on by default. However this
> also depends on how quickly database driver developers can adapt.

I would favor waiting a release to turn it on by default, precisely to
give driver developers time to adapt.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Kenneth Marshall
Date:
On Wed, May 19, 2010 at 10:54:01AM -0400, Robert Haas wrote:
> On Wed, May 19, 2010 at 10:17 AM, Stefan Kaltenbrunner
> <stefan@kaltenbrunner.cc> wrote:
> > On 05/19/2010 08:13 AM, Tom Lane wrote:
> >> Bernd Helmle <mailings@oopsware.de> writes:
> >>> --On 18. Mai 2010 23:20:26 +0200 Jesper Krogh <jesper@krogh.cc> wrote:
> >>>> May I ask whats the reason is for "breaking" the compatibillity?
> >>
> >>> "Efficency", if i am allowed to call it this way. The new hex
> >>> representation should be more efficient to retrieve and to handle than the
> >>> old one. I think bytea_output was set to hex for testing purposes on the
> >>> first hand, but not sure wether there was a consensus to leave it there
> >>> finally later.
> >>
> >> Yeah, we intentionally set it that way initially to help find stuff that
> >> needs to be updated (as DBD::Pg evidently does). ?It's still TBD whether
> >> 9.0.0 will ship with that default or not.
> >
> > given how much faster the new format is (or rather how slow the old one
> > was) and the number of people I have seen complaining "why is bytea so
> > slow) I would like to see it staying turned on by default. However this
> > also depends on how quickly database driver developers can adapt.
> 
> I would favor waiting a release to turn it on by default, precisely to
> give driver developers time to adapt.
> 
Changing something like that within the minor release arc is
not a good idea. It would be better to have it on by default and
if the driver developers are not up to use it, they can have that
as a setting that they will need to change when going to 9.0. I
would be very upset to have a minor upgrade break my database. At
least the major upgrades have more testing.

Regards,
Ken


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Robert Haas
Date:
On Wed, May 19, 2010 at 11:00 AM, Kenneth Marshall <ktm@rice.edu> wrote:
> On Wed, May 19, 2010 at 10:54:01AM -0400, Robert Haas wrote:
>> On Wed, May 19, 2010 at 10:17 AM, Stefan Kaltenbrunner
>> <stefan@kaltenbrunner.cc> wrote:
>> > On 05/19/2010 08:13 AM, Tom Lane wrote:
>> >> Bernd Helmle <mailings@oopsware.de> writes:
>> >>> --On 18. Mai 2010 23:20:26 +0200 Jesper Krogh <jesper@krogh.cc> wrote:
>> >>>> May I ask whats the reason is for "breaking" the compatibillity?
>> >>
>> >>> "Efficency", if i am allowed to call it this way. The new hex
>> >>> representation should be more efficient to retrieve and to handle than the
>> >>> old one. I think bytea_output was set to hex for testing purposes on the
>> >>> first hand, but not sure wether there was a consensus to leave it there
>> >>> finally later.
>> >>
>> >> Yeah, we intentionally set it that way initially to help find stuff that
>> >> needs to be updated (as DBD::Pg evidently does). ?It's still TBD whether
>> >> 9.0.0 will ship with that default or not.
>> >
>> > given how much faster the new format is (or rather how slow the old one
>> > was) and the number of people I have seen complaining "why is bytea so
>> > slow) I would like to see it staying turned on by default. However this
>> > also depends on how quickly database driver developers can adapt.
>>
>> I would favor waiting a release to turn it on by default, precisely to
>> give driver developers time to adapt.
>>
> Changing something like that within the minor release arc is
> not a good idea. It would be better to have it on by default and
> if the driver developers are not up to use it, they can have that
> as a setting that they will need to change when going to 9.0. I
> would be very upset to have a minor upgrade break my database. At
> least the major upgrades have more testing.

I meant, wait for the next MAJOR release to turn it on by default.
Changing it in a minor release is clearly a bad idea.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company


Re: BYTEA / DBD::Pg change in 9.0 beta

From
"Greg Sabino Mullane"
Date:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160


>> given how much faster the new format is (or rather how slow the old one
>> was) and the number of people I have seen complaining "why is bytea so
>> slow) I would like to see it staying turned on by default. However this
>> also depends on how quickly database driver developers can adapt.

DBD::Pg is already patched, and will very likely be released before 9.0

- -- 
Greg Sabino Mullane greg@turnstep.com
End Point Corporation http://www.endpoint.com/
PGP Key: 0x14964AC8 201005191105
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAkvz/mgACgkQvJuQZxSWSsiQ+ACg5B61+bJ4fNaJI8kTNIjyV2lS
Y0IAnR9tB86upmY5JufsVvcithHOUtjP
=rgH4
-----END PGP SIGNATURE-----




Re: BYTEA / DBD::Pg change in 9.0 beta

From
Magnus Hagander
Date:
On Wed, May 19, 2010 at 11:06 AM, Greg Sabino Mullane <greg@turnstep.com> wrote:
>
>>> given how much faster the new format is (or rather how slow the old one
>>> was) and the number of people I have seen complaining "why is bytea so
>>> slow) I would like to see it staying turned on by default. However this
>>> also depends on how quickly database driver developers can adapt.
>
> DBD::Pg is already patched, and will very likely be released before 9.0

How do the distros generaly deal with that? E.g. do we have to wait
for RHEL7 for it to actually show up in redhat?

-- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Robert Haas
Date:
On Wed, May 19, 2010 at 11:07 AM, Magnus Hagander <magnus@hagander.net> wrote:
> On Wed, May 19, 2010 at 11:06 AM, Greg Sabino Mullane <greg@turnstep.com> wrote:
>>
>>>> given how much faster the new format is (or rather how slow the old one
>>>> was) and the number of people I have seen complaining "why is bytea so
>>>> slow) I would like to see it staying turned on by default. However this
>>>> also depends on how quickly database driver developers can adapt.
>>
>> DBD::Pg is already patched, and will very likely be released before 9.0
>
> How do the distros generaly deal with that? E.g. do we have to wait
> for RHEL7 for it to actually show up in redhat?

Yeah, that's what I'm worried about.  I remember going through this
with E'' quoting.  It wasn't fun.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Aidan Van Dyk
Date:
* Magnus Hagander <magnus@hagander.net> [100519 11:08]:
> How do the distros generaly deal with that? E.g. do we have to wait
> for RHEL7 for it to actually show up in redhat?

Don't worry, 9.0 won't show up in redhat for a while yet either...

;-)

-- 
Aidan Van Dyk                                             Create like a god,
aidan@highrise.ca                                       command like a king,
http://www.highrise.ca/                                   work like a slave.

Re: BYTEA / DBD::Pg change in 9.0 beta

From
Magnus Hagander
Date:
On Wed, May 19, 2010 at 11:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, May 19, 2010 at 11:07 AM, Magnus Hagander <magnus@hagander.net> wrote:
>> On Wed, May 19, 2010 at 11:06 AM, Greg Sabino Mullane <greg@turnstep.com> wrote:
>>>
>>>>> given how much faster the new format is (or rather how slow the old one
>>>>> was) and the number of people I have seen complaining "why is bytea so
>>>>> slow) I would like to see it staying turned on by default. However this
>>>>> also depends on how quickly database driver developers can adapt.
>>>
>>> DBD::Pg is already patched, and will very likely be released before 9.0
>>
>> How do the distros generaly deal with that? E.g. do we have to wait
>> for RHEL7 for it to actually show up in redhat?
>
> Yeah, that's what I'm worried about.  I remember going through this
> with E'' quoting.  It wasn't fun.

Right. So do we know what the policy is? As long as DBD::Pg is
released before pg 9.0 we'd be fine, *provided* that they
(redhat/novell/debian/whatever) actually pull in the latest version at
that point...

-- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Alex Hunsaker
Date:
On Wed, May 19, 2010 at 09:05, Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, May 19, 2010 at 11:00 AM, Kenneth Marshall <ktm@rice.edu> wrote:
>> Changing something like that within the minor release arc is
>> not a good idea. It would be better to have it on by default and
>> if the driver developers are not up to use it, they can have that
>> as a setting that they will need to change when going to 9.0. I
>> would be very upset to have a minor upgrade break my database. At
>> least the major upgrades have more testing.
>
> I meant, wait for the next MAJOR release to turn it on by default.
> Changing it in a minor release is clearly a bad idea.

I think with this release already being clearly marked as a bit more
than the usual major release (9.0 vs 8.5), we can get away with it
leaving the default the way it is.


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Robert Haas
Date:
On Wed, May 19, 2010 at 11:31 AM, Alex Hunsaker <badalex@gmail.com> wrote:
> On Wed, May 19, 2010 at 09:05, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Wed, May 19, 2010 at 11:00 AM, Kenneth Marshall <ktm@rice.edu> wrote:
>>> Changing something like that within the minor release arc is
>>> not a good idea. It would be better to have it on by default and
>>> if the driver developers are not up to use it, they can have that
>>> as a setting that they will need to change when going to 9.0. I
>>> would be very upset to have a minor upgrade break my database. At
>>> least the major upgrades have more testing.
>>
>> I meant, wait for the next MAJOR release to turn it on by default.
>> Changing it in a minor release is clearly a bad idea.
>
> I think with this release already being clearly marked as a bit more
> than the usual major release (9.0 vs 8.5), we can get away with it
> leaving the default the way it is.

I think it just depends on whether we're likely to get releases from
Linux vendors that include PG 9.0 but not the updated drivers.  I'm
not sure their schedule will be affected by whether we call it 8.5 or
9.0.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Stefan Kaltenbrunner
Date:
On 05/19/2010 11:45 AM, Robert Haas wrote:
> On Wed, May 19, 2010 at 11:31 AM, Alex Hunsaker <badalex@gmail.com> wrote:
>> On Wed, May 19, 2010 at 09:05, Robert Haas <robertmhaas@gmail.com> wrote:
>>> On Wed, May 19, 2010 at 11:00 AM, Kenneth Marshall <ktm@rice.edu> wrote:
>>>> Changing something like that within the minor release arc is
>>>> not a good idea. It would be better to have it on by default and
>>>> if the driver developers are not up to use it, they can have that
>>>> as a setting that they will need to change when going to 9.0. I
>>>> would be very upset to have a minor upgrade break my database. At
>>>> least the major upgrades have more testing.
>>>
>>> I meant, wait for the next MAJOR release to turn it on by default.
>>> Changing it in a minor release is clearly a bad idea.
>>
>> I think with this release already being clearly marked as a bit more
>> than the usual major release (9.0 vs 8.5), we can get away with it
>> leaving the default the way it is.
> 
> I think it just depends on whether we're likely to get releases from
> Linux vendors that include PG 9.0 but not the updated drivers.  I'm
> not sure their schedule will be affected by whether we call it 8.5 or
> 9.0.

that's a fair point (like I expect debian to provide 9.0 as a backport)
though the packages could just change the default for that backport.
The precedence for that is standard_conforming_strings which we now have
for a while(since 8.2 iirc) - though I don't think we have a firm plan
on when we are actually going to turn it on...
Not sure if we really need to wait 4 major releases to allow driver
developers to adapt...
So one idea would be to turn it off for 9.1 and enable that and scs for
9.1 and try to get driver developers attention early in the release cycle.


Stefan


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Stefan Kaltenbrunner
Date:
On 05/19/2010 11:19 AM, Magnus Hagander wrote:
> On Wed, May 19, 2010 at 11:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> On Wed, May 19, 2010 at 11:07 AM, Magnus Hagander <magnus@hagander.net> wrote:
>>> On Wed, May 19, 2010 at 11:06 AM, Greg Sabino Mullane <greg@turnstep.com> wrote:
>>>>
>>>>>> given how much faster the new format is (or rather how slow the old one
>>>>>> was) and the number of people I have seen complaining "why is bytea so
>>>>>> slow) I would like to see it staying turned on by default. However this
>>>>>> also depends on how quickly database driver developers can adapt.
>>>>
>>>> DBD::Pg is already patched, and will very likely be released before 9.0
>>>
>>> How do the distros generaly deal with that? E.g. do we have to wait
>>> for RHEL7 for it to actually show up in redhat?
>>
>> Yeah, that's what I'm worried about.  I remember going through this
>> with E'' quoting.  It wasn't fun.
> 
> Right. So do we know what the policy is? As long as DBD::Pg is
> released before pg 9.0 we'd be fine, *provided* that they
> (redhat/novell/debian/whatever) actually pull in the latest version at
> that point...

well the next debian release (squeeze) is likely to end up with 8.4
anyway, same for RHEL6 I believe. so I don't think we really have a
"problem" there. It might actually be not to bad a time to break
compatibility because there is a long time for distros to catch up after
their next releases. For debian 9.0 will likely show up in backports but
i would expect them to provide a backport of the relevant drivers as
well (or change the default for the backport).


Stefan


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Robert Haas
Date:
On Wed, May 19, 2010 at 12:16 PM, Stefan Kaltenbrunner
<stefan@kaltenbrunner.cc> wrote:
>> I think it just depends on whether we're likely to get releases from
>> Linux vendors that include PG 9.0 but not the updated drivers.  I'm
>> not sure their schedule will be affected by whether we call it 8.5 or
>> 9.0.
>
> that's a fair point (like I expect debian to provide 9.0 as a backport)
> though the packages could just change the default for that backport.
> The precedence for that is standard_conforming_strings which we now have
> for a while(since 8.2 iirc) - though I don't think we have a firm plan
> on when we are actually going to turn it on...
> Not sure if we really need to wait 4 major releases to allow driver
> developers to adapt...
> So one idea would be to turn it off for 9.1 and enable that and scs for
> 9.1 and try to get driver developers attention early in the release cycle.

I think we previously discussed flipping standard_conforming_strings
at the beginning of the 9.1 cycle, and I'm still OK with that.  Not
sure it bears on the present issue, though.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Stefan Kaltenbrunner
Date:
On 05/19/2010 12:32 PM, Robert Haas wrote:
> On Wed, May 19, 2010 at 12:16 PM, Stefan Kaltenbrunner
> <stefan@kaltenbrunner.cc> wrote:
>>> I think it just depends on whether we're likely to get releases from
>>> Linux vendors that include PG 9.0 but not the updated drivers.  I'm
>>> not sure their schedule will be affected by whether we call it 8.5 or
>>> 9.0.
>>
>> that's a fair point (like I expect debian to provide 9.0 as a backport)
>> though the packages could just change the default for that backport.
>> The precedence for that is standard_conforming_strings which we now have
>> for a while(since 8.2 iirc) - though I don't think we have a firm plan
>> on when we are actually going to turn it on...
>> Not sure if we really need to wait 4 major releases to allow driver
>> developers to adapt...
>> So one idea would be to turn it off for 9.1 and enable that and scs for
>> 9.1 and try to get driver developers attention early in the release cycle.
> 
> I think we previously discussed flipping standard_conforming_strings
> at the beginning of the 9.1 cycle, and I'm still OK with that.  Not
> sure it bears on the present issue, though.

well we might want to get a bit more formal with deprecating things now
that we have an official EOL policy for the server. Maybe we should
consider adding stuff like scs, bytea output format,add_missing_from
there as well with a depcreation & removal/cnage notice.
On the other side we tend to break drivers with other stuff in almost
every release in some way or another anyway (iirc we broke JDBC in 9.0
already) so maybe that is a mood point.


Stefan


Re: BYTEA / DBD::Pg change in 9.0 beta

From
Florian Pflug
Date:
On May 19, 2010, at 18:32 , Robert Haas wrote:
> On Wed, May 19, 2010 at 12:16 PM, Stefan Kaltenbrunner
> <stefan@kaltenbrunner.cc> wrote:
>>> I think it just depends on whether we're likely to get releases from
>>> Linux vendors that include PG 9.0 but not the updated drivers.  I'm
>>> not sure their schedule will be affected by whether we call it 8.5 or
>>> 9.0.
>>
>> that's a fair point (like I expect debian to provide 9.0 as a backport)
>> though the packages could just change the default for that backport.
>> The precedence for that is standard_conforming_strings which we now have
>> for a while(since 8.2 iirc) - though I don't think we have a firm plan
>> on when we are actually going to turn it on...
>> Not sure if we really need to wait 4 major releases to allow driver
>> developers to adapt...
>> So one idea would be to turn it off for 9.1 and enable that and scs for
>> 9.1 and try to get driver developers attention early in the release cycle.
>
> I think we previously discussed flipping standard_conforming_strings
> at the beginning of the 9.1 cycle, and I'm still OK with that.  Not
> sure it bears on the present issue, though.

Well, since both issues are related in that they deal with data representation and force driver upgrades and/or
reviewingand testing of application code to ensure correct encoding and decoding, flipping both defaults simultaneously
mightreduce the overall effort required. If 9.0 ships with the new bytea encoding enabled by default, people will need
toadapt applications for 9.0 to deal with bytea issues and then again for 9.1 to deal with string encoding issues. 

Since updated drivers can choose to override the default on a per-connection basis if they're ready to deal with the
newrepresentation, flipping the default doesn't have much of a performance advantage either. 

So +1 for flipping both with the release of 9.1, and warning people well ahead of time. Maybe there could even be a
warningin the 9.0 release notes about the scheduled change? 

best regards,
Florian Pflug





Re: BYTEA / DBD::Pg change in 9.0 beta

From
Tom Lane
Date:
Magnus Hagander <magnus@hagander.net> writes:
> On Wed, May 19, 2010 at 11:11 AM, Robert Haas <robertmhaas@gmail.com> wrote:
>> Yeah, that's what I'm worried about. �I remember going through this
>> with E'' quoting. �It wasn't fun.

> Right. So do we know what the policy is? As long as DBD::Pg is
> released before pg 9.0 we'd be fine, *provided* that they
> (redhat/novell/debian/whatever) actually pull in the latest version at
> that point...

Well, as far as Red Hat goes, I'll make a point of not shipping 9.0
before DBD::Pg is updated.  I'm not sure how tense we need to be about
this, though, considering that users can easily turn off the option
if they need to run clients with old drivers.

BTW, standard_conforming_strings is really a different case because of
the SQL-injection security hazards with non-scs-aware client code.
I don't see any comparable risk for bytea format.
        regards, tom lane


Re: BYTEA / DBD::Pg change in 9.0 beta

From
"Andrew Dunstan"
Date:
On Wed, May 19, 2010 1:31 pm, Tom Lane wrote:
> BTW, standard_conforming_strings is really a different case because of
> the SQL-injection security hazards with non-scs-aware client code.
> I don't see any comparable risk for bytea format.
>


Yeah, and the impact of this will be much more limited. I'd want quite a
bit of convincing to agree that we shouldn't turn it on right away.

cheers

andrew