Thread: Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)

[ Blind CC to general added for comment below.]


> [Taken off GENERAL, added HACKERS to cc:]
>
> Bruce Momjian wrote:
> > > He's meaning the libpq version for dynamic link loading.  Is the
> > > libpq.so lib changing versions (like the change from 6.5.x to 7.0.x
> > > changed from libpq.so.2.0 to libpq.so.2.1, which broke binary RPM
> > > compatibility for other RPM's linked against libpq.so.2.0, which failed
> > > when libpq.so.2.1 came on the scene).  I think the answer is no, but I
> > > haven't checked the details yet.
>
> > I usually up the .so version numbers before entering beta.  That way,
> > they get marked as newer than older versions.
>
> May I ask: is it necessary?  Have there been version-bumping changes to
> libpq since 7.0.x? (With the rate that necessary improvement is
> happening to PostgreSQL, probably).

No, only major releases have bumps.

>
> But, enough rant.  That _is_ I believe what Trond was asking about.  We
> have been bitten before with people installing the PHP from RedHat 6.2
> after installing the PostgreSQL 7.0.x RPMset -- and dependency failures
> wreaked havoc.
>
> So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then?
>
> Actually, Bruce, it would do me and Trond a great favor if a list of
> what so's are getting bumped and to what version were to be posted.  At
> least we can plan for a transition at that point.

See pgsql/src/tools/RELEASE_CHANGES.  I edit interfaces/*/Makefile and
increase the minor number for every interface by one.



Let me add one thing on this RPM issue.  There has been a lot of talk
recently about RPM's, and what they should do, and what they don't do,
and who should be blamed.  Unfortunately, much of the discussion has
been very unproductive and more like 'venting'.

I really don't appreciate people 'venting' on these lists, especially
since we have _nothing_ to do with RPM's.  All we do is make the
PostgreSQL software.

If people want to discuss RPM's on the ports list, or want to create a
new list just about RPM's, that's OK, but venting is bad, and venting on
a list that has nothing to do with RPM's is even worse.

What would be good is for someone to constructively make a posting about
the known problems, and come up with acceptible solutions.  Asking us to
fix it really isn't going to help because we don't deal with RPM's here,
and we don't have enough free time to make significant changes to meet
the needs of RPM's.

Also, remember we support many Unix platforms, and Linux is only one of
them.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:
> Lamar Owen wrote:
> > May I ask: is it necessary?  Have there been version-bumping changes to
> > libpq since 7.0.x? (With the rate that necessary improvement is
> > happening to PostgreSQL, probably).
> No, only major releases have bumps.
> See pgsql/src/tools/RELEASE_CHANGES.  I edit interfaces/*/Makefile and
> increase the minor number for every interface by one.

Thanks for the pointer.
> Let me add one thing on this RPM issue.  There has been a lot of talk
> recently about RPM's, and what they should do, and what they don't do,
> and who should be blamed.  Unfortunately, much of the discussion has
> been very unproductive and more like 'venting'.
> I really don't appreciate people 'venting' on these lists, especially
> since we have _nothing_ to do with RPM's.  All we do is make the
> PostgreSQL software.

> What would be good is for someone to constructively make a posting about
> the known problems, and come up with acceptible solutions.  Asking us to
> fix it really isn't going to help because we don't deal with RPM's here,
> and we don't have enough free time to make significant changes to meet
> the needs of RPM's.

Which is why I stepped up to the plate last year to help with RPM's.

I apologize if you took my post (which I edited greatly) as 'venting' --
it was not my intention to 'vent', much less offend.  I just want to
know what to expect from the 7.1 release.  I feel that that is germane
to the Hackers list, as the knowledge necessary to answer the question
is to be found on the list. (and you answered the question above).

Like it or not, in the eyes of many people having solid RPM's is a core
issue.  If there are gotchas, I want to document them so people don't
get blindsided.  Or work around them.  Or ask why the change is
necessary in the first place.

I appreciate the fact that we are not here to make it easy for
distributors to package our software.  I also appreciate the fact that
if you don't at least make an effort to work with major distributors
(and RedHat, TurboLinux, Caldera, and SuSE together comprise a major
userbase) that you run the risk of not being distributed in favor of an
inferior product.

I also appreciate and applaud the cross-platform mentality of the
PostgreSQL developers.  Linux is far from the only OS to be supported by
PostgreSQL, true.  But Linux is also the most popular OS for PostgreSQL
deployment.

However, there are known problems that can bite people who are not using
RPM's and are not running Linux.  Some of those problems are such that
it will take someone with more knowledge than I currently possess to
solve.  One is the issue of upgrading/migrating tools.  This is not an
RPM-specific issue.  To me, that is the only big issue that I have
spoken about in a way that could even remotely be construed as
'venting'.  And it is not a Linux-specific issue.  It is a core issue.

I'll shut up now, as I have cross-distribution RPM problems to solve.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

> > What would be good is for someone to constructively make a posting about
> > the known problems, and come up with acceptable solutions.  Asking us to
> > fix it really isn't going to help because we don't deal with RPM's here,
> > and we don't have enough free time to make significant changes to meet
> > the needs of RPM's.
> 
> Which is why I stepped up to the plate last year to help with RPM's.
> 
> I apologize if you took my post (which I edited greatly) as 'venting' --
> it was not my intention to 'vent', much less offend.  I just want to
> know what to expect from the 7.1 release.  I feel that that is germane
> to the Hackers list, as the knowledge necessary to answer the question
> is to be found on the list. (and you answered the question above).

No, I was not pointing to you when I mentioned venting.  There have been
other RPM threads lately.  I just want information on how to make things
better for RPM's, not vents.

> Like it or not, in the eyes of many people having solid RPM's is a core
> issue.  If there are gotchas, I want to document them so people don't
> get blindsided.  Or work around them.  Or ask why the change is
> necessary in the first place.

Sure.

> I appreciate the fact that we are not here to make it easy for
> distributors to package our software.  I also appreciate the fact that
> if you don't at least make an effort to work with major distributors
> (and RedHat, TurboLinux, Caldera, and SuSE together comprise a major
> userbase) that you run the risk of not being distributed in favor of an
> inferior product.

Let them.  It is their decision.  Frankly, I have seen this attitude
before, and I don't like it.  Just the mention that "Gee, if you don't
cooperate, we may yank you," is really a veiled threat.  Now, I know you
aren't saying that, but the "if you don't play nice, we will drop you"
argument sounds a lot more like MS that a Linux vendor should be acting,
especially since they are not telling us what they want or assisting in
the work.

The "We are big.  Just fix it and let us know when it is ready" attitude
does not work here, and that is what I am hearing mostly from the RPM
people.

> I also appreciate and applaud the cross-platform mentality of the
> PostgreSQL developers.  Linux is far from the only OS to be supported by
> PostgreSQL, true.  But Linux is also the most popular OS for PostgreSQL
> deployment.

True, it is the most popular, but that doesn't make the others less
important. 

This whole statement comes across as, "You run on Linux, and look, you
took the time to run on other OS's too.  How quaint."

In the history of this project, Linux was an after-thought.  None of our
platforms are inferior or superior, except to the extent that the
platform does not support Unix standard functions (like NT/Cygwin).

> However, there are known problems that can bite people who are not using
> RPM's and are not running Linux.  Some of those problems are such that
> it will take someone with more knowledge than I currently possess to
> solve.  One is the issue of upgrading/migrating tools.  This is not an
> RPM-specific issue.  To me, that is the only big issue that I have
> spoken about in a way that could even remotely be construed as
> 'venting'.  And it is not a Linux-specific issue.  It is a core issue.

Again, your comments where quite helpful.  We need more of them.  We
need to hear more about the problems people are having with RPM's, and
how to make them better.

There must be a list of known problems.  Let's hear them, so we can try
to solve them as a group.  However, in general, we do not make dramatic
change to work around OS bugs, and do not plan to make major changes to
work around the limitations of RPM's.  My bet is that some middle layer
can be created that will fix that for us.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 

Bruce Momjian <pgman@candle.pha.pa.us> writes:

> Let them.  It is their decision.  Frankly, I have seen this attitude
> before, and I don't like it.  Just the mention that "Gee, if you don't
> cooperate, we may yank you," is really a veiled threat.  Now, I know you
> aren't saying that, but the "if you don't play nice, we will drop you"
> argument sounds a lot more like MS that a Linux vendor should be acting,
> especially since they are not telling us what they want or assisting in
> the work.

FWIW, I've never threatened to do so. If I wanted to, I would just do
it[1] - threats are bad and never cause anything but bad feelings.

That being said, my favorite wishes (in addition to as much SQL
compliance and performance as possible, of course) are:

* migration on upgrade
* old libraries being able to speak to newer databases, so old binaries can continue working after database upgrades
* good sonames on libraries - if a library hasn't changed, bumping the number to show it's part of a new version isn't
necesarry.If it is backwards compatible, just bump the minor version, if it isn't, bump the major version. Or even
better,use versioned symbols (I don't know how many other OSes than Linux and Solaris supports this, though). 
 

As for assisting, at least Red Hat contributes to a lot of projects,
some of which are important to postgres on one or more platforms: gdb,
gcc, glibc and the linux kernel. There just isn't enough resources to
do everything, but I try to help out with the RPMs.

When we make patches for packages, we try to cooperate with the
author(s) to get them in - happily, we haven't had much of a need for
that with postgresql.

> The "We are big.  Just fix it and let us know when it is ready" attitude
> does not work here, and that is what I am hearing mostly from the RPM
> people.

I haven't heard anyone say that.

> There must be a list of known problems.  Let's hear them, so we can try
> to solve them as a group.  However, in general, we do not make dramatic
> change to work around OS bugs, and do not plan to make major changes to
> work around the limitations of RPM's.

I don't think there are any apart from the upgrade issues - if library
versioning follows the standard, that certainly won't be a problem.


[1] which I'm not even close to doing - I've spent a bit of time lately
hunting down aliasing bugs in MySQL which causes wrong SQL query
results if compiled with "-O2". Ouch.
-- 
Trond Eivind Glomsrød
Red Hat, Inc.

Bruce Momjian wrote:
> > I appreciate the fact that we are not here to make it easy for
> > distributors to package our software.  I also appreciate the fact that
> > if you don't at least make an effort to work with major distributors
> > (and RedHat, TurboLinux, Caldera, and SuSE together comprise a major
> > userbase) that you run the risk of not being distributed in favor of an
> > inferior product.
> Let them.  It is their decision.  Frankly, I have seen this attitude
> before, and I don't like it.  Just the mention that "Gee, if you don't
> cooperate, we may yank you," is really a veiled threat.

I don't even see it as a veiled threat, Bruce.  It simply _is_ a
threat.  There are other RDBMS choices.  Currently PostgreSQL is the
Officially Sanctioned RDBMS for multiple Linux distributions.  As our
capabilities increase, it will make us more and more attractive as the
Choice, Top Shelf Open Source RDBMS.

However, the upgrade gotcha has left a very bitter taste in more than
one user's mouth.  I'll not say more about that now, as I've said quite
enough in the past.  And I'm still trying to figure out enough of the
internals of the storage manager to try to write the migration tools
myself.  But, I have other fish to fry right now, the biggest being
cross-distribution RPM's.

> > Linux is far from the only OS to be supported by
> > PostgreSQL, true.  But Linux is also the most popular OS for PostgreSQL
> > deployment.
> True, it is the most popular, but that doesn't make the others less
> important.

No, it doesn't.  
> This whole statement comes across as, "You run on Linux, and look, you
> took the time to run on other OS's too.  How quaint."
I ran Unix before there was linux.  I ran Unix years before Linus was
even out of High School.  Well, that is if you count Tandy Xenix V7 and
System III as Unix.  Or AT&T 3B1 SysVR2.  Or Apollo DomainOS SR10.2.  Or
Ultrix on a VAX 11/750 (running in tandem with VMS). And I'm considering
moving my most critical public servers from Linux over to OpenBSD.  A
Linux bigot I'm not.
> > However, there are known problems that can bite people who are not using
> > RPM's and are not running Linux.  Some of those problems are such that
> > it will take someone with more knowledge than I currently possess to
> Again, your comments where quite helpful.  We need more of them.  We
> need to hear more about the problems people are having with RPM's, and
> how to make them better.

Bruce, sometimes I fear my own lack of communications skills.  If I can
make my wife fighting mad at me with me having no clue as to what I said
that made her mad, I fear I can make anyone mad, without knowing what I
said to do so.  So, I guess you could say I'm a little paranoid about my
communications skills.  So, I'm glad you considered my comments helpful
-- I was beginning to get worried.
> There must be a list of known problems.  Let's hear them, so we can try
> to solve them as a group.  However, in general, we do not make dramatic
> change to work around OS bugs, and do not plan to make major changes to
> work around the limitations of RPM's.  My bet is that some middle layer
> can be created that will fix that for us.

Meet Mr. Middle Layer. :-)  The PostgreSQL spec file that controls the
RPM build is one of the most complex ones in the RedHat distribution,
AFAIK. There's the middle layer.  It does quite a bit of finagling
already.

And the work that Peter E is doing is helping my cause significantly.

Bruce, when I recover fully from the illness I've had the last few days,
I'll try to come up with a coherent listing of what I've had to work
around in the past.  My current headache won't let me think straight
right now, which makes it likely that I won't effectively communicate
the issues.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

> However, the upgrade gotcha has left a very bitter taste in more than
> one user's mouth.  I'll not say more about that now, as I've said quite
> enough in the past.  And I'm still trying to figure out enough of the
> internals of the storage manager to try to write the migration tools
> myself.  But, I have other fish to fry right now, the biggest being
> cross-distribution RPM's.

Actually, I would prefer to see how we can improve what we have before
making a binary conversion utility that will have to be updated for
every release.


> Meet Mr. Middle Layer. :-)  The PostgreSQL spec file that controls the
> RPM build is one of the most complex ones in the RedHat distribution,
> AFAIK. There's the middle layer.  It does quite a bit of finagling
> already.

Yes, I suspected the RPM was the middle layer.  To the extent we can
make that easier, let's hear it.  Tell us what you need to do, and what
you can't do, and see if any of us can figure out how to make things
easier.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 

Bruce Momjian wrote:
> 
> > However, the upgrade gotcha has left a very bitter taste in more than
> > one user's mouth.  I'll not say more about that now, as I've said quite
> > enough in the past.  And I'm still trying to figure out enough of the
> > internals of the storage manager to try to write the migration tools
> > myself.  But, I have other fish to fry right now, the biggest being
> > cross-distribution RPM's.
> 
> Actually, I would prefer to see how we can improve what we have before
> making a binary conversion utility that will have to be updated for
> every release.
> 
> > Meet Mr. Middle Layer. :-)  The PostgreSQL spec file that controls the
> > RPM build is one of the most complex ones in the RedHat distribution,
> > AFAIK. There's the middle layer.  It does quite a bit of finagling
> > already.
> 
> Yes, I suspected the RPM was the middle layer.  To the extent we can
> make that easier, let's hear it.  Tell us what you need to do, and what
> you can't do, and see if any of us can figure out how to make things
> easier.

Ok, here goes:
*    Location-agnostic installation.  Documentation (which I'll be happy to
contribute) on that.  Peter E is already working in this area. Getting
the installation that 'make install' spits out massaged into an FHS
compliant setup is the majority of the RPM's spec file.

*    Upgrades that don't require an ASCII database dump for migration. This
can either be implemented as a program to do a pg_dump of an arbitrary
version of data, or as a binary migration utility.  Currently, I'm
saving old executables to run under a special environment to pull a dump
-- but it is far from optimal.  What if the OS upgrade behind 99% of the
upgrades makes it where those old executables can't run due to binary
incompatibility (say I'm going from RedHat 3.0.3 to RedHat 7 -- 3.0.3,
IIRC, as a.out...( and I know 3.0.3 didn't have PostgreSQL RPMs).)? 
What I could actually do to prevent that problem is build all of
PostgreSQL's 6.1.x, 6.2.x, 6.3.x, 6.4.x, and 6.5.x and include the
necessary backend executables as part of the RPM.... But I think you see
the problem there.  However, that would in my mind be better than the
current situation, albeit taking up a lot of space.

*    A less source-centric mindset.  Let's see, how to explain?  The
regression tests are a good example.  You need make. You need the source
installed, configured, and built in the usual location.  You need
portions of contrib.  RPM's need to be installable on compiler-crippled
servers for security.  While the demand for regression testing on such a
box may not be there, it certainly does give the user something to use
to get standard output for bug reports.  As a point, I run PostgreSQL in
production on a compilerless machine.  No compiler == more security. 
And Linux has enough security problems without a compiler being
available :-(.  Oh, and I have no make on that machine either.

The documentation as well as many of the examples assume too much, IMHO,
about the install location and the install methodology.

I think I may have a solution for the library versioning problem. 
Rather than symlink libpq.so->libpq.so.2->libpq.so.2.x, I'll copy
libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that.  A little more
code for me.  There is no real danger in version confusion with RPM's
versioning and upgrade methodology, as long as you consistently use the
RPMset.  The PostgreSQL version number is readily found from an RPM
database query, making the so version immaterial.

The upgrade issue is the hot trigger for me at this time.  It is and has
been a major drain on my time and effort, as well as Trond's and others,
to get the RPM upgrade working even remotely smoothly.  And I am willing
to code -- once I know how to go about doing it in the backend.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen writes:

> Getting the installation that 'make install' spits out massaged into
> an FHS compliant setup is the majority of the RPM's spec file.

./configure --prefix=/usr --sysconfdir=/etc

Off you go...  (I'll refrain from commenting further on the FHS.)

> *    Upgrades that don't require an ASCII database dump for migration.

Let me ask you this question:  When any given RPM-based Linux distribution
will update their system from ext2 to, say, ReiserFS across the board, how
are they going to do it?  Sincere question.

> *    A less source-centric mindset.  Let's see, how to explain?  The
> regression tests are a good example.  You need make. You need the source
> installed, configured, and built in the usual location.

This is not an excuse, but almost every package behaves this way.  Test
suites are designed to be run after "make all" and before "make install".  
When you ship a binary package then you're saying to users "I did the
building and installation (and presumably everything else that the authors
recommend along the way) for you."  RPM packages usually don't work very
well on systems that are not exactly like the one they were built on, so
this seems to be a fair assumption.

Getting the regression tests to work from anywhere is not very hard, but
it's not the most interesting project for most people. :-)

> I think I may have a solution for the library versioning problem. 
> Rather than symlink libpq.so->libpq.so.2->libpq.so.2.x, I'll copy
> libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that.

I'd still claim that if RPM thinks it's smarter than the dynamic loader,
then it's broken.  All the shared libraries on Linux have a symlink from
more general to more specific names.  PostgreSQL can't be the first to hit
this problem.

-- 
Peter Eisentraut      peter_e@gmx.net       http://yi.org/peter-e/

Peter Eisentraut wrote:
> Lamar Owen writes:
> > Getting the installation that 'make install' spits out massaged into
> > an FHS compliant setup is the majority of the RPM's spec file.
> ./configure --prefix=/usr --sysconfdir=/etc
> Off you go...  (I'll refrain from commenting further on the FHS.)

I know alot of people don't like LSB/FHS, but, like it or not, I have to
work with it.  And, many many thanks for putting in the work on the
configuration as you have.
> > *     Upgrades that don't require an ASCII database dump for migration.
> Let me ask you this question:  When any given RPM-based Linux distribution
> will update their system from ext2 to, say, ReiserFS across the board, how
> are they going to do it?  Sincere question.

Like the TRS-80 model III, whose TRSDOS 1.3 could not read the TRS-80
Model I's disks, written on TRSDOS 2.3 (TRSDOS's versioning was
absolutely horrendous).  TRSDOS 1.3 included a CONVERT utility that
could read files from the old filesystem.  

I'm sure that the newer distributions using ReiserFS as the primary
filesystem will include legacy Ext2/3 support, at least for read-only,
for many versions to come.

And that's my big beef -- a newer version of PostgreSQL can't even
pg_dump an old database.  If that single function was supported, I would
have no problem with the upgrade whatsoever.  
> > *     A less source-centric mindset.  Let's see, how to explain?  The
> > regression tests are a good example.  You need make. You need the source
> > installed, configured, and built in the usual location.
> This is not an excuse, but almost every package behaves this way.  Test
> suites are designed to be run after "make all" and before "make install".
> When you ship a binary package then you're saying to users "I did the
> building and installation (and presumably everything else that the authors
> recommend along the way) for you."

Yes, and I do just that.  Regression testing is a regular part of my
build process here.

>  RPM packages usually don't work very
> well on systems that are not exactly like the one they were built on

Boy, don't I know it.....~;-/
> Getting the regression tests to work from anywhere is not very hard, but
> it's not the most interesting project for most people. :-)

I know.  I'll probably do it myself, as that is something I _can_ do.  
> > I think I may have a solution for the library versioning problem.
> > Rather than symlink libpq.so->libpq.so.2->libpq.so.2.x, I'll copy
> > libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that.
> I'd still claim that if RPM thinks it's smarter than the dynamic loader,
> then it's broken.  All the shared libraries on Linux have a symlink from
> more general to more specific names.  PostgreSQL can't be the first to hit
> this problem.

RPM is getting it's .so dependency list straight from the mouth of the
dynamic loader itself.  RPM uses shell scripts, customizable for each
system on which RPM runs, to determine the automatic dependencies --
those shell scripts run the dynamic loader to get the list of requires. 
So, the dynamic loader itself is providing the list.  
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11