Thread: Big 7.1 open items

Big 7.1 open items

From
Bruce Momjian
Date:
Here is the list I have gotten of open 7.1 items:bit typeinheritancedrop columnvacuum index speedcached query
plansmemorycontext cleanupTOASTWALfmgr redesignencrypt pg_shadow passwordsredesign pg_hba.conf password file optionnew
locationfor config files
 

I have some of my own that are not on the list, as do others who are
working on their own items.  Just thought a list of major items that
need work would be helpful.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Karel Zak
Date:
On Tue, 13 Jun 2000, Bruce Momjian wrote:

> Here is the list I have gotten of open 7.1 items:
>     
>     bit type
>     inheritance
>     drop column
>     vacuum index speed
>     cached query plans^^^^^^^^^^^^^^^^^

I have already down it and I send patch for _testing_ next week (or
later), but I think that not will for 7.1, but 7.2.

>     memory context cleanup
>     TOAST
>     WAL
>     fmgr redesign
>     encrypt pg_shadow passwords
>     redesign pg_hba.conf password file option
>     new location for config files
+ new ACL? (please :-)


BTW. --- really cool list.
                    Karel



Re: Big 7.1 open items

From
Vince Vielhaber
Date:
On Tue, 13 Jun 2000, Bruce Momjian wrote:

> Here is the list I have gotten of open 7.1 items:
>     
>     encrypt pg_shadow passwords

This will be for 7.1?  For some reason I thought it was being pushed 
off to 7.2.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net128K ISDN from $22.00/mo - 56K Dialup from
$16.00/moat Pop4 Networking       Online Campground Directory    http://www.camping-usa.com      Online Giftshop
Superstore   http://www.cloudninegifts.com
 
==========================================================================





Re: Big 7.1 open items

From
"Sergio A. Kessler"
Date:
Bruce Momjian <pgman@candle.pha.pa.us> el día Tue, 13 Jun 2000 05:05:53 
-0400 (EDT), escribió:

[...]
>    new location for config files

can I suggest /etc/postgresql ?


sergio



Re: Big 7.1 open items

From
The Hermit Hacker
Date:
On Tue, 13 Jun 2000, Sergio A. Kessler wrote:

> Bruce Momjian <pgman@candle.pha.pa.us> el d�a Tue, 13 Jun 2000 05:05:53 
> -0400 (EDT), escribi�:
> 
> [...]
> >    new location for config files
> 
> can I suggest /etc/postgresql ?

you can ... but everything related to postgresql has always been designed
not to require any special permissions to install, and /etc/postgresql
would definitely require root access to install :(




Re: Big 7.1 open items

From
Vince Vielhaber
Date:
On Tue, 13 Jun 2000, The Hermit Hacker wrote:

> On Tue, 13 Jun 2000, Sergio A. Kessler wrote:
> 
> > Bruce Momjian <pgman@candle.pha.pa.us> el d�a Tue, 13 Jun 2000 05:05:53 
> > -0400 (EDT), escribi�:
> > 
> > [...]
> > >    new location for config files
> > 
> > can I suggest /etc/postgresql ?
> 
> you can ... but everything related to postgresql has always been designed
> not to require any special permissions to install, and /etc/postgresql
> would definitely require root access to install :(

~postgres/etc ??

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net128K ISDN from $22.00/mo - 56K Dialup from
$16.00/moat Pop4 Networking       Online Campground Directory    http://www.camping-usa.com      Online Giftshop
Superstore   http://www.cloudninegifts.com
 
==========================================================================





Re: Big 7.1 open items

From
The Hermit Hacker
Date:
that one works ...

On Tue, 13 Jun 2000, Vince Vielhaber wrote:

> On Tue, 13 Jun 2000, The Hermit Hacker wrote:
> 
> > On Tue, 13 Jun 2000, Sergio A. Kessler wrote:
> > 
> > > Bruce Momjian <pgman@candle.pha.pa.us> el d�a Tue, 13 Jun 2000 05:05:53 
> > > -0400 (EDT), escribi�:
> > > 
> > > [...]
> > > >    new location for config files
> > > 
> > > can I suggest /etc/postgresql ?
> > 
> > you can ... but everything related to postgresql has always been designed
> > not to require any special permissions to install, and /etc/postgresql
> > would definitely require root access to install :(
> 
> ~postgres/etc ??
> 
> Vince.
> -- 
> ==========================================================================
> Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net
>  128K ISDN from $22.00/mo - 56K Dialup from $16.00/mo at Pop4 Networking
>         Online Campground Directory    http://www.camping-usa.com
>        Online Giftshop Superstore    http://www.cloudninegifts.com
> ==========================================================================
> 
> 
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy@hub.org           secondary: scrappy@{freebsd|postgresql}.org 



Re: Big 7.1 open items

From
Tom Lane
Date:
Vince Vielhaber <vev@michvhf.com> writes:
>> encrypt pg_shadow passwords

> This will be for 7.1?  For some reason I thought it was being pushed 
> off to 7.2.

I don't know of anything that would force delaying it --- it's not
dependent on querytree redesign, for example.  The real question is,
do we have anyone who's committed to do the work?  I heard a lot of
discussion but I didn't hear anyone taking responsibility for it...
        regards, tom lane


Re: Big 7.1 open items

From
Vince Vielhaber
Date:
On Tue, 13 Jun 2000, Tom Lane wrote:

> Vince Vielhaber <vev@michvhf.com> writes:
> >> encrypt pg_shadow passwords
> 
> > This will be for 7.1?  For some reason I thought it was being pushed 
> > off to 7.2.
> 
> I don't know of anything that would force delaying it --- it's not
> dependent on querytree redesign, for example.  The real question is,
> do we have anyone who's committed to do the work?  I heard a lot of
> discussion but I didn't hear anyone taking responsibility for it...

I offered to do the work and I have the md5 routine here and tested on 
a number of platforms.  But as I said, I thought someone wanted to delay
it until 7.2, if that's not the case then I'll get to it.  There was also
a lack of interest in testing it, but I think we have most platforms 
covered.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net128K ISDN from $22.00/mo - 56K Dialup from
$16.00/moat Pop4 Networking       Online Campground Directory    http://www.camping-usa.com      Online Giftshop
Superstore   http://www.cloudninegifts.com
 
==========================================================================





Re: Big 7.1 open items

From
Ed Loehr
Date:
Vince Vielhaber wrote:
> 
> > > [...]
> > > >   new location for config files
> > >
> > > can I suggest /etc/postgresql ?
> >
> > you can ... but everything related to postgresql has always been designed
> > not to require any special permissions to install, and /etc/postgresql
> > would definitely require root access to install :(
> 
> ~postgres/etc ??

I would suggest you don't *require* or assume the creation of a postgres
user, except as an overridable default.

Regards,
Ed Loehr


Re: Big 7.1 open items

From
Vince Vielhaber
Date:
On Tue, 13 Jun 2000, Ed Loehr wrote:

> Vince Vielhaber wrote:
> > 
> > > > [...]
> > > > >   new location for config files
> > > >
> > > > can I suggest /etc/postgresql ?
> > >
> > > you can ... but everything related to postgresql has always been designed
> > > not to require any special permissions to install, and /etc/postgresql
> > > would definitely require root access to install :(
> > 
> > ~postgres/etc ??
> 
> I would suggest you don't *require* or assume the creation of a postgres
> user, except as an overridable default.

I *knew* somebody would bring this up.  Before I sent that I tried to 
describe the intent a few ways and just opted for simple.  PostgreSQL
has to run as SOMEONE.  Substitute that SOMEONE for ~postgres above.

Vince.
-- 
==========================================================================
Vince Vielhaber -- KA8CSH    email: vev@michvhf.com    http://www.pop4.net128K ISDN from $22.00/mo - 56K Dialup from
$16.00/moat Pop4 Networking       Online Campground Directory    http://www.camping-usa.com      Online Giftshop
Superstore   http://www.cloudninegifts.com
 
==========================================================================





Re: Big 7.1 open items

From
Tom Lane
Date:
The Hermit Hacker <scrappy@hub.org> writes:
>>>> new location for config files
>> 
>> can I suggest /etc/postgresql ?

> you can ... but everything related to postgresql has always been designed
> not to require any special permissions to install, and /etc/postgresql
> would definitely require root access to install :(

Even more to the point, the config files are always kept in the data
directory so that it's possible to run multiple installations on the
same machine.  Keeping the config files under /etc (or any other fixed
location) would destroy that capability.
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
Vince Vielhaber <vev@michvhf.com> writes:
>> do we have anyone who's committed to do the work?  I heard a lot of
>> discussion but I didn't hear anyone taking responsibility for it...

> I offered to do the work and I have the md5 routine here and tested on 
> a number of platforms.  But as I said, I thought someone wanted to delay
> it until 7.2, if that's not the case then I'll get to it.

Far as I can see, you should go for it.
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Here is the list I have gotten of open 7.1 items:

There were a whole bunch of issues about the type system --- automatic
coercion rules, default type selection for both numeric and string
literals, etc.  Not sure how to describe this in five words or less...
        regards, tom lane


Re: Big 7.1 open items

From
Kaare Rasmussen
Date:
> Here is the list I have gotten of open 7.1 items:

I thought that someone was working on
outer joins
better views (or rewriting the rules system, not sure what the direction was)
better SQL92 compliance
also, I think that at some time there was discussion about a better interface
for procedures, enabling them to work on several tuples. May be wrong though.

But if all, or just most, of the items on your list will be finished, it ought
to be a 8.0 release :-)

-- 
Kaare Rasmussen            --Linux, spil,--        Tlf:        3816 2582
Kaki Data                tshirts, merchandize      Fax:        3816 2582
Howitzvej 75               �ben 14.00-18.00        Email: kar@webline.dk
2000 Frederiksberg        L�rdag 11.00-17.00       Web:      www.suse.dk


Re: Big 7.1 open items

From
Thomas Lockhart
Date:
Since there are several people interested in contributing, we should
list:
 Support multiple simultaneous character sets, per SQL92
                - Thomas


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> >     memory context cleanup
> >     TOAST
> >     WAL
> >     fmgr redesign
> >     encrypt pg_shadow passwords
> >     redesign pg_hba.conf password file option
> >     new location for config files
> 
>     + new ACL? (please :-)
> 
> 
> BTW. --- really cool list.

Updated TODO.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
>     + new ACL? (please :-)

Updated TODO.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> On Tue, 13 Jun 2000, Bruce Momjian wrote:
> 
> > Here is the list I have gotten of open 7.1 items:
> >     
> >     bit type
> >     inheritance
> >     drop column
> >     vacuum index speed
> >     cached query plans
> >     memory context cleanup
> >     TOAST
> >     WAL
> >     fmgr redesign
> >     encrypt pg_shadow passwords
> >     redesign pg_hba.conf password file option
> 
> Any details?

I would like to remove our pg_passwd script that allows
username/passwords to be specified in a file, change that file to lists
of users, or allow lists of users in pg_hba.conf.


> 
> >     new location for config files
> 
> Are you referring to pushing internal files to `$PGDATA/global'?

Yes.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> that one works ...
> 
> On Tue, 13 Jun 2000, Vince Vielhaber wrote:
> 
> > On Tue, 13 Jun 2000, The Hermit Hacker wrote:
> > 
> > > On Tue, 13 Jun 2000, Sergio A. Kessler wrote:
> > > 
> > > > Bruce Momjian <pgman@candle.pha.pa.us> el d�a Tue, 13 Jun 2000 05:05:53 
> > > > -0400 (EDT), escribi�:
> > > > 
> > > > [...]
> > > > >    new location for config files
> > > > 
> > > > can I suggest /etc/postgresql ?
> > > 
> > > you can ... but everything related to postgresql has always been designed
> > > not to require any special permissions to install, and /etc/postgresql
> > > would definitely require root access to install :(
> > 
> > ~postgres/etc ??

Remember, that file has to be specific for each data tree, so it has to
be under /data.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Vince Vielhaber <vev@michvhf.com> writes:
> >> encrypt pg_shadow passwords
> 
> > This will be for 7.1?  For some reason I thought it was being pushed 
> > off to 7.2.
> 
> I don't know of anything that would force delaying it --- it's not
> dependent on querytree redesign, for example.  The real question is,
> do we have anyone who's committed to do the work?  I heard a lot of
> discussion but I didn't hear anyone taking responsibility for it...

Agreed.  No reason not to be in 7.1.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
I just kept your e-mails.  I will make a TODO.detail mailbox with them.

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Here is the list I have gotten of open 7.1 items:
> 
> There were a whole bunch of issues about the type system --- automatic
> coercion rules, default type selection for both numeric and string
> literals, etc.  Not sure how to describe this in five words or less...
> 
>             regards, tom lane
> 


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > Here is the list I have gotten of open 7.1 items:
> 
> I thought that someone was working on
> outer joins
> better views (or rewriting the rules system, not sure what the direction was)
> better SQL92 compliance
> also, I think that at some time there was discussion about a better interface
> for procedures, enabling them to work on several tuples. May be wrong though.
> 
> But if all, or just most, of the items on your list will be finished, it ought
> to be a 8.0 release :-)
> 

Most of these are planned for 7.2.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
Added to TODO.

> Since there are several people interested in contributing, we should
> list:
> 
>   Support multiple simultaneous character sets, per SQL92
> 
>                  - Thomas
> 


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Peter Eisentraut
Date:
On Tue, 13 Jun 2000, Vince Vielhaber wrote:

> > > >    new location for config files
> > > 
> > > can I suggest /etc/postgresql ?
> > 
> > you can ... but everything related to postgresql has always been designed
> > not to require any special permissions to install, and /etc/postgresql
> > would definitely require root access to install :(
> 
> ~postgres/etc ??

You need root access to create a postgres user. What's wrong with just
keeping it in $PGDATA and making symlinks whereever you would prefer it?

-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Big 7.1 open items

From
Peter Eisentraut
Date:
On Tue, 13 Jun 2000, Bruce Momjian wrote:

> Here is the list I have gotten of open 7.1 items:
>     
>     bit type
>     inheritance
>     drop column
>     vacuum index speed
>     cached query plans
>     memory context cleanup
>     TOAST
>     WAL
>     fmgr redesign
>     encrypt pg_shadow passwords
>     redesign pg_hba.conf password file option

Any details?

>     new location for config files

Are you referring to pushing internal files to `$PGDATA/global'?


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Big 7.1 open items

From
Michael Robinson
Date:
While people are working on that, they might want to add some sanity checking
to the multibyte character decoders.  Currently they fail to check for
"illegal" character sequences (i.e. sequences with no valid multibyte mapping),
and fail to do something reasonable (like return an error, silently drop the
offending characters, or anything else besides just returning random garbage
and crashing the backend).

The last time this failed to get on the TODO list because Bruce wanted
more than one person to verify that it was an issue.  If several people 
are going to work on the NATIONAL CHARACTER stuff, maybe they could look
into this issue, too.
-Michael Robinson

>Added to TODO.
>
>> Since there are several people interested in contributing, we should
>> list:
>> 
>>   Support multiple simultaneous character sets, per SQL92



Re: Big 7.1 open items

From
"Oliver Elphick"
Date:
>On Tue, 13 Jun 2000, Bruce Momjian wrote: > >> Here is the list I have gotten of open 7.1 items:

Rolling back a transaction after dropping a table creates a corrupted
database.  (Yes, I know it warns you not to do that, but users are
fallible and sometimes just plain stupid.)  Although the system catalog
entries are rolled back, the file on disk is permanently destroyed.

I suggest that DROP TABLE in a transaction should not be allowed.



-- 
Oliver Elphick                                Oliver.Elphick@lfix.co.uk
Isle of Wight                              http://www.lfix.co.uk/oliver
PGP: 1024R/32B8FAA1: 97 EA 1D 47 72 3F 28 47  6B 7E 39 CC 56 E4 C1 47
GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839  932A 614D 4C34 3E1D 0C1C
========================================   "I beseech you therefore, brethren, by the mercies of      God, that ye
presentyour bodies a living sacrifice,      holy, acceptable unto God, which is your reasonable      service."
Romans12:1 
 




Re: Re: Big 7.1 open items

From
Bruce Momjian
Date:
> While people are working on that, they might want to add some sanity checking
> to the multibyte character decoders.  Currently they fail to check for
> "illegal" character sequences (i.e. sequences with no valid multibyte mapping),
> and fail to do something reasonable (like return an error, silently drop the
> offending characters, or anything else besides just returning random garbage
> and crashing the backend).
> 
> The last time this failed to get on the TODO list because Bruce wanted
> more than one person to verify that it was an issue.  If several people 
> are going to work on the NATIONAL CHARACTER stuff, maybe they could look
> into this issue, too.

The issue is that some people felt we shouldn't be performing such
checks, and some did.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> On Tue, 13 Jun 2000, Karel Zak wrote:
> 
> >     + new ACL? (please :-)
> 
> Not if we're shipping in August. :(
> 

I hear you.
--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
"Oliver Elphick" <olly@lfix.co.uk> writes:
> I suggest that DROP TABLE in a transaction should not be allowed.

I had actually made it do that for a short time early this year,
and was shouted down.  On reflection I have to agree; it's too useful
to be able to do
begin;drop table foo;create table foo(new schema);...end;

You do indeed lose big if you suffer an error partway through, but
the answer to that is to fix our file naming conventions so that we
can support rollback of drop table.

Also note the complaints we've been getting about CREATE USER not
working inside a transaction block.  That is a case where someone
(Peter IIRC) took the more hard-line approach of emitting an error
instead of a warning.  I think it was not the right choice to make.
        regards, tom lane


Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Tom Lane writes:

> Also note the complaints we've been getting about CREATE USER not
> working inside a transaction block.  That is a case where someone
> (Peter IIRC) took the more hard-line approach of emitting an error
> instead of a warning.  I think it was not the right choice to make.

Probably. Remember that you can claim your lunch any time. :)

In all truth, the problem is that the ODBC driver isn't very flexible
about putting BEGIN/END blocks around things. Perhaps that is also
something to look at.


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Big 7.1 open items

From
JanWieck@t-online.de (Jan Wieck)
Date:
Tom Lane wrote:
> "Oliver Elphick" <olly@lfix.co.uk> writes:
> > I suggest that DROP TABLE in a transaction should not be allowed.
>
> I had actually made it do that for a short time early this year,
> and was shouted down.  On reflection I have to agree; it's too useful
> to be able to do
>
>    begin;
>    drop table foo;
>    create table foo(new schema);
>    ...
>    end;
>
> You do indeed lose big if you suffer an error partway through, but
> the answer to that is to fix our file naming conventions so that we
> can support rollback of drop table.
   Belongs  IMHO  to  the  discussion  to  keep separate what is   separate  (having  indices/toast-relations/etc.  in
separate  directories and whatnot).
 
   I've   never   been   really   happy  with  the  file  naming   conventions. The need of a filesystem entry to have
the same   name of the DB object that is associated with it isn't right.   I know, some people love to be able to
easily identify  the   files with ls(1). OTOH what is that good for?
 
   Well,  someone  can  easily see how big the disk footprint of   his data is.  Whow - what an info. Anything else?
   Why not changing the naming to be something like this:
       <dbroot>/catalog_tables/pg_...       <dbroot>/catalog_index/pg_...       <dbroot>/user_tables/oid_...
<dbroot>/user_index/oid_...      <dbroot>/temp_tables/oid_...       <dbroot>/temp_index/oid_...
<dbroot>/toast_tables/oid_...      <dbroot>/toast_index/oid_...       <dbroot>/whatnot_???/...
 
   This way, it  would  be  much  easier  to  separate  all  the   different  object types to different physical media.
Wewould   loose some  transparency,  but  I've  allways  wondered  what   people  USE  that  for  (except  for  just
wanna know). For   convinience we could implement another  little  utility  that   tells the object size like
 
       DESCRIBE TABLE/VIEW/whatnot <object-name>
   that returns the physical location and storage details of the   object. And psql could use it to print this  info
additional  on  the  \d commands. Would give unprivileged users access to   this info, so be it, it's not a security
issueIMHO.
 
   The subdirectory an object goes into has to be controlled  by   the relkind. So we need to tidy up that a little
too.I think   it's worth it.
 
   The objects  storage  location  (the  bare  file)  now  would   contain  the  OID.  So  we  avoid  naming  conflicts
fortemp   tables, naming conflicts during DROP/CREATE in a  transaction   and all the like.
 
   Comments?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #




Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Tom Lane wrote:
> > "Oliver Elphick" <olly@lfix.co.uk> writes:
> > > I suggest that DROP TABLE in a transaction should not be allowed.
> >
> > I had actually made it do that for a short time early this year,
> > and was shouted down.  On reflection I have to agree; it's too useful
> > to be able to do
> >
> >    begin;
> >    drop table foo;
> >    create table foo(new schema);
> >    ...
> >    end;
> >
> > You do indeed lose big if you suffer an error partway through, but
> > the answer to that is to fix our file naming conventions so that we
> > can support rollback of drop table.
> 
>     Belongs  IMHO  to  the  discussion  to  keep separate what is
>     separate  (having  indices/toast-relations/etc.  in  separate
>     directories and whatnot).
> 
>     I've   never   been   really   happy  with  the  file  naming
>     conventions. The need of a filesystem entry to have the  same
>     name of the DB object that is associated with it isn't right.
>     I know, some people love to be able to  easily  identify  the
>     files with ls(1). OTOH what is that good for?

Well, I have no problem just appending some serial number to the end of
our existing names.  That solves both purposes, no?  Seems Vadim is
going to have a new storage manager in 7.2 anyway.

If/when we lose file name/object mapping, we will have to write
command-line utilities to report the mappings so people can do
administration properly.  It certainly makes it hard for administrators.

> 
>     Well,  someone  can  easily see how big the disk footprint of
>     his data is.  Whow - what an info. Anything else?
> 
>     Why not changing the naming to be something like this:
> 
>         <dbroot>/catalog_tables/pg_...
>         <dbroot>/catalog_index/pg_...
>         <dbroot>/user_tables/oid_...
>         <dbroot>/user_index/oid_...
>         <dbroot>/temp_tables/oid_...
>         <dbroot>/temp_index/oid_...
>         <dbroot>/toast_tables/oid_...
>         <dbroot>/toast_index/oid_...
>         <dbroot>/whatnot_???/...
> 
>     This way, it  would  be  much  easier  to  separate  all  the
>     different  object types to different physical media. We would
>     loose some  transparency,  but  I've  allways  wondered  what
>     people  USE  that  for  (except  for  just  wanna  know). For
>     convinience we could implement another  little  utility  that
>     tells the object size like

Yes, we could do that.

> 
>         DESCRIBE TABLE/VIEW/whatnot <object-name>
> 
>     that returns the physical location and storage details of the
>     object. And psql could use it to print this  info  additional
>     on  the  \d commands. Would give unprivileged users access to
>     this info, so be it, it's not a security issue IMHO.

You need something that works from the command line, and something that
works if PostgreSQL is not running.  How would you restore one file from
a tape.  I guess you could bring back the whole thing, then do the
query, and move the proper table file back in, but that is a pain.



--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Don Baccus
Date:
At 07:13 PM 6/14/00 -0400, Bruce Momjian wrote:

>> 
>>     This way, it  would  be  much  easier  to  separate  all  the
>>     different  object types to different physical media. We would
>>     loose some  transparency,  but  I've  allways  wondered  what
>>     people  USE  that  for  (except  for  just  wanna  know). For
>>     convinience we could implement another  little  utility  that
>>     tells the object size like
>
>Yes, we could do that.

It's a poor man's substitute for a proper create tablespace on
storage 'filesystem' - style dml statement, but it's a step in
the right direction.



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
Tom Lane
Date:
JanWieck@t-online.de (Jan Wieck) writes:
>     I've   never   been   really   happy  with  the  file  naming
>     conventions. The need of a filesystem entry to have the  same
>     name of the DB object that is associated with it isn't right.
>     I know, some people love to be able to  easily  identify  the
>     files with ls(1). OTOH what is that good for?

I agree with Jan on this: let's just change the file names over to
be OIDs.  Then we can have rollbackable DROP and RENAME TABLE easily.
Naming the files after the logical names of the tables is nice if it
doesn't cost anything, but it is *not* worth the trouble to preserve
a relationship between filename and tablename when it is costing us.
And it's costing us big time.  That single feature is hurting us on
functionality, robustness, and portability, and for what benefit?
Not nearly enough.  It's time to just let go of it.

>     Why not changing the naming to be something like this:

>         <dbroot>/catalog_tables/pg_...
>         <dbroot>/catalog_index/pg_...
>         <dbroot>/user_tables/oid_...
>         <dbroot>/user_index/oid_...
>         <dbroot>/temp_tables/oid_...
>         <dbroot>/temp_index/oid_...
>         <dbroot>/toast_tables/oid_...
>         <dbroot>/toast_index/oid_...
>         <dbroot>/whatnot_???/...

I don't see a lot of value in that.  Better to do something like
tablespaces:
<dbroot>/<oidoftablespace>/<oidofobject>
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> You need something that works from the command line, and something that
> works if PostgreSQL is not running.  How would you restore one file from
> a tape.

"Restore one file from a tape"?  How are you going to do that anyway?
You can't save and restore portions of a database like that, because
of transaction commit status problems.  To restore table X correctly,
you'd have to restore pg_log as well, and then your other tables are
hosed --- unless you also restore all of them from the backup.  Only
a complete database restore from tape would work, and for that you
don't need to tell which file is which.  So the above argument is a
red herring.

I realize it's nice to be able to tell which table file is which by
eyeball, but the price we are paying for that small convenience is
just too high.  Give that up, and we can have rollbackable DROP and
RENAME now (I'll personally commit to making it happen for 7.1).
Continue to insist on it, and I don't think we'll *ever* have those
features in a really robust form.  It's just not possible to do
multiple file renames atomically.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > You need something that works from the command line, and something that
> > works if PostgreSQL is not running.  How would you restore one file from
> > a tape.
> 
> "Restore one file from a tape"?  How are you going to do that anyway?
> You can't save and restore portions of a database like that, because
> of transaction commit status problems.  To restore table X correctly,
> you'd have to restore pg_log as well, and then your other tables are
> hosed --- unless you also restore all of them from the backup.  Only
> a complete database restore from tape would work, and for that you
> don't need to tell which file is which.  So the above argument is a
> red herring.
> 
> I realize it's nice to be able to tell which table file is which by
> eyeball, but the price we are paying for that small convenience is
> just too high.  Give that up, and we can have rollbackable DROP and
> RENAME now (I'll personally commit to making it happen for 7.1).
> Continue to insist on it, and I don't think we'll *ever* have those
> features in a really robust form.  It's just not possible to do
> multiple file renames atomically.
> 

OK, I am flexible.  (Yea, right.)  :-)

But seriously, let me give some background.  I used Ingres, that used
the VMS file system, but used strange sequential AAAF324 numbers for
tables.  When someone deleted a table, or we were looking at what tables
were using disk space, it was impossible to find the Ingres table names
that went with the file.  There was a system table that showed it, but
it was poorly documented, and if you deleted the table, there was no way
to look on the tape to find out which file to restore.

As far as pg_log, you certainly would not expect to get any information
back from the time of the backup table to current, so the current pg_log
would be just fine.

Basically, I guess we have to do it, but we have to print the proper
error messages for cases in the backend we just print the file name. 
Also, we have to now replace the 'ls -l' command with something that
will be meaningful.

Right now, we use 'ps' with args to display backend information, and ls
-l to show disk information.  We are going to lose that here.



--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> But seriously, let me give some background.  I used Ingres, that used
> the VMS file system, but used strange sequential AAAF324 numbers for
> tables.  When someone deleted a table, or we were looking at what tables
> were using disk space, it was impossible to find the Ingres table names
> that went with the file.  There was a system table that showed it, but
> it was poorly documented, and if you deleted the table, there was no way
> to look on the tape to find out which file to restore.

Fair enough, but it seems to me that the answer is to expend some effort
on system admin support tools.  We could do a lot in that line with less
effort than trying to make a fundamentally mismatched filesystem
representation do what we need.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > But seriously, let me give some background.  I used Ingres, that used
> > the VMS file system, but used strange sequential AAAF324 numbers for
> > tables.  When someone deleted a table, or we were looking at what tables
> > were using disk space, it was impossible to find the Ingres table names
> > that went with the file.  There was a system table that showed it, but
> > it was poorly documented, and if you deleted the table, there was no way
> > to look on the tape to find out which file to restore.
> 
> Fair enough, but it seems to me that the answer is to expend some effort
> on system admin support tools.  We could do a lot in that line with less
> effort than trying to make a fundamentally mismatched filesystem
> representation do what we need.

That was my point --- that in doing this change, we are taking on more
TODO items, that may detract from our main TODO items.  I am also
concerned that the filename/tablename mapping is supported by so many
Unix toolks like ls, lsof/fstat, and tar, that we could be in for
needing to support tons of utilities to enable administrators to do what
they can so easily do now.

Even gdb shows us the filename/tablename in backtraces.  We are never
going to be able to reproduce that.  I guess I didn't want to bit off
that much work until we had a _convincing_ need.  I guess I don't
consider table schema commands inside transactions and such to be as big
an items as the utility features we will need to build.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Don Baccus
Date:
At 10:28 PM 6/14/00 -0400, Bruce Momjian wrote:

>As far as pg_log, you certainly would not expect to get any information
>back from the time of the backup table to current, so the current pg_log
>would be just fine.

In reality, very few people are going to be interested in restoring
a table in a way that breaks referential integrity and other 
normal assumptions about what exists in the database.  The reality
is that most people are going to engage in a little time travel
to a past, consistent backup rather than do as you suggest.

This is going to be more and more true as Postgres gains more and
more acceptance in (no offense intended) the real world.

>Right now, we use 'ps' with args to display backend information, and ls
>-l to show disk information.  We are going to lose that here.

Dependence on "ls -l" is, IMO, a very weak argument.



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> That was my point --- that in doing this change, we are taking on more
> TODO items, that may detract from our main TODO items.

True, but they are also TODO items that could be handled by people other
than the inner circle of key developers.  The actual rejiggering of
table-to-filename mapping is going to have to be done by one of the
small number of people who are fully up to speed on backend internals.
But we've got a lot more folks who would be able (and, hopefully,
willing) to design and code whatever tools are needed to make the
dbadmin's job easier in the face of the new filesystem layout.  I'd
rather not expend a lot of core time to avoid needing those tools,
especially when I feel the old approach is fatally flawed anyway.

> Even gdb shows us the filename/tablename in backtraces.  We are never
> going to be able to reproduce that.

Backtraces from *what*, exactly?  99% of the backend is still going
to be dealing with the same data as ever.  It might be that poking
around in fd.c will be a little harder, but considering that fd.c
doesn't really know or care what the files it's manipulating are
anyway, I'm not convinced that this is a real issue.

> I guess I don't consider table schema commands inside transactions and
> such to be as big an items as the utility features we will need to
> build.

You've *got* to be kidding.  We're constantly seeing complaints about
the fact that rolling back DROP or RENAME TABLE fails --- and worse,
leaves the table in a corrupted/inconsistent state.  As far as I can
tell, that's one of the worst robustness problems we've got left to
fix.  This is a big deal IMHO, and I want it to be fixed and fixed
right.  I don't see how to fix it right if we try to keep physical
filenames tied to logical tablenames.

Moreover, that restriction will continue to hurt us if we try to
preserve it while implementing tablespaces, ANSI schemas, etc.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > That was my point --- that in doing this change, we are taking on more
> > TODO items, that may detract from our main TODO items.
> 
> True, but they are also TODO items that could be handled by people other
> than the inner circle of key developers.  The actual rejiggering of
> table-to-filename mapping is going to have to be done by one of the
> small number of people who are fully up to speed on backend internals.
> But we've got a lot more folks who would be able (and, hopefully,
> willing) to design and code whatever tools are needed to make the
> dbadmin's job easier in the face of the new filesystem layout.  I'd
> rather not expend a lot of core time to avoid needing those tools,
> especially when I feel the old approach is fatally flawed anyway.

Yes, it is clearly fatally flawed.  I agree.

> > Even gdb shows us the filename/tablename in backtraces.  We are never
> > going to be able to reproduce that.
> 
> Backtraces from *what*, exactly?  99% of the backend is still going
> to be dealing with the same data as ever.  It might be that poking
> around in fd.c will be a little harder, but considering that fd.c
> doesn't really know or care what the files it's manipulating are
> anyway, I'm not convinced that this is a real issue.

I was just throwing gdb out as an example.  The bigger ones are ls,
lsof/fstat, and tar.

> > I guess I don't consider table schema commands inside transactions and
> > such to be as big an items as the utility features we will need to
> > build.
> 
> You've *got* to be kidding.  We're constantly seeing complaints about
> the fact that rolling back DROP or RENAME TABLE fails --- and worse,
> leaves the table in a corrupted/inconsistent state.  As far as I can
> tell, that's one of the worst robustness problems we've got left to
> fix.  This is a big deal IMHO, and I want it to be fixed and fixed
> right.  I don't see how to fix it right if we try to keep physical
> filenames tied to logical tablenames.
> 
> Moreover, that restriction will continue to hurt us if we try to
> preserve it while implementing tablespaces, ANSI schemas, etc.
> 

Well, we did have someone do a test implementation of oid file names,
and their report was that is looked pretty ugly.  However, if people are
convinced it has to be done, we can get started.  I guess I was waiting
for Vadim's storage manager, where the whole idea of separate files is
going to go away anyway, I suspect.  We would then have to re-write all
our admin tools for the new format.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
JanWieck@t-online.de (Jan Wieck)
Date:
Tom Lane wrote:
> JanWieck@t-online.de (Jan Wieck) writes:
> >     Why not changing the naming to be something like this:
> 
> >         <dbroot>/catalog_tables/pg_...
> >         <dbroot>/catalog_index/pg_...
> >         <dbroot>/user_tables/oid_...
> >         <dbroot>/user_index/oid_...
> >         <dbroot>/temp_tables/oid_...
> >         <dbroot>/temp_index/oid_...
> >         <dbroot>/toast_tables/oid_...
> >         <dbroot>/toast_index/oid_...
> >         <dbroot>/whatnot_???/...
> 
> I don't see a lot of value in that.  Better to do something like
> tablespaces:
> 
>     <dbroot>/<oidoftablespace>/<oidofobject>
   *Slap* - yes!


Jan

-- 

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #



Re: Big 7.1 open items

From
JanWieck@t-online.de (Jan Wieck)
Date:
Bruce Momjian wrote:
> >
> >         DESCRIBE TABLE/VIEW/whatnot <object-name>
> >
> >     that returns the physical location and storage details of the
> >     object. And psql could use it to print this  info  additional
> >     on  the  \d commands. Would give unprivileged users access to
> >     this info, so be it, it's not a security issue IMHO.
>
> You need something that works from the command line, and something that
> works if PostgreSQL is not running.  How would you restore one file from
> a tape.  I guess you could bring back the whole thing, then do the
> query, and move the proper table file back in, but that is a pain.
   Think you messed up some basics of PG here.
   It's  totally useless to restore single files of a PostgreSQL   database. You could either put back anything below
./data,or   nothing - the reason is pg_log.
 
   You  don't  need  something  that work's if PostgreSQL is not   running.  You cannot restore ONE file from a  tape!
You can   restore  a  PostgreSQL  instance (only a complete one - not a   single DB, nor a single table or any  other
object).  While   your  backup  is writing to the tape, each number of backends   could concurrently modify single
blocks of  the  heap,  it's   indices and pg_log. So what does the tape contain the?
 
   I'd  like  to ask you, are you sure the backups you're making   are worth the power consumption of  the  tape
drive? You're   talking  about  restoring  a file - and sould be aware of the   fact, that any file based backup would
neverbe able  to  get   consistent snapshot of the database, like pg_dump is able to.
 
   As long as you don't take  the  postmaster  down  during  the   entire  saving  of ./data, you aren't in a safe
position.And   the only safe RESTORE is  to  restore  ./data  completely  or   nothing.  It's  not  even  (easily)
possible to  initdb and   restore a single DB from tape (it is, but requires some  deep   knowledge and more than just
restoringsome files from tape).
 
   YOU REALLY DON'T NEED ANY FILENAMES IN THERE!
   The more I think about it, the more I feel these file  names,   easily associatable with the objects they represent,
aremore   dangerous than useful in practice. Maybe we should  obfuscate   the  entire  ./data  like  Oracle  does  with
it'stablespace   files.  Just  that  our  tablespaces  will  be   directories,   containing totally cryptic named
files.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #




Re: Re: Big 7.1 open items

From
Michael Robinson
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>The issue is that some people felt we shouldn't be performing such
>checks, and some did.

Well, more precisely, the issue was stalemated at "one person felt we 
should perform such checks" and "one person (who, incidentally, wrote the
code) felt we shouldn't".

Obviously, as it stands, the issue is going nowhere.

I was just hoping to encourage more people to examine the problem, so that
we might get a consensus one way or the other.
-Michael Robinson



Re: Big 7.1 open items

From
"Ross J. Reedstrom"
Date:
On Wed, Jun 14, 2000 at 11:21:15PM -0400, Bruce Momjian wrote:
> 
> Well, we did have someone do a test implementation of oid file names,
> and their report was that is looked pretty ugly.  

That someone would be me. Did my mails from this morning fall into a black
hole? I've got a patch that does either oid filenames or relname_<oid>,
take your pick.  It doesn't do tablespaces, just leaves the files where
they are. TO do relname_<oid>, I add a relphysname field to pg_class.

I'll update it to current and throw it at the PATCHES list this weekend,
unless someone more central wants to do tablespaces first. I tried
out rollinging back ALTER TABLE RENAME. Works fine. Biggest problem
with it is that I played silly buggers with the relcache for no good
reason. Hiroshi stripped that out and said it works fone, otherwise. I
also haven't touched DROP TABLE yet. The physical file be deleted at
transaction commit time, then?  Hmm, we're the 'things to do at commit'
queue?

> convinced it has to be done, we can get started.  I guess I was waiting
> for Vadim's storage manager, where the whole idea of separate files is
> going to go away anyway, I suspect.  We would then have to re-write all
> our admin tools for the new format.

Any strong objections to the mixed relname_oid solution? It gets us
everything oids does, and still lets Bruce use 'ls -l' to find the big
tables, putting off writing any admin tools that'll need to be rewritten,
anyway.

Ross
-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005


Re: Big 7.1 open items

From
Thomas Lockhart
Date:
> But seriously, let me give some background.  I used Ingres, that used
> the VMS file system, but used strange sequential AAAF324 numbers for
> tables.  When someone deleted a table, or we were looking at what 
> tables were using disk space, it was impossible to find the Ingres 
> table names that went with the file.  There was a system table that 
> showed it, but it was poorly documented, and if you deleted the table, 
> there was no way to look on the tape to find out which file to 
> restore.

I had the same experience, but let's put the blame where it belongs: it
wasn't the filename's fault, it was poor design and support from the
Ingres company.
                 - Thomas


Re: Big 7.1 open items

From
Chris Bitmead
Date:
"Ross J. Reedstrom" wrote:

> Any strong objections to the mixed relname_oid solution? It gets us
> everything oids does, and still lets Bruce use 'ls -l' to find the big
> tables, putting off writing any admin tools that'll need to be rewritten,
> anyway.

Doesn't  relname_oid defeat the purpose of oid file names, which is that
they don't change when the table is renamed? Wasn't it going to be oids
with a tool to create a symlink of relname -> oid ?


Re: Re: Big 7.1 open items

From
Thomas Lockhart
Date:
> >The issue is that some people felt we shouldn't be performing such
> >checks, and some did.
> Well, more precisely, the issue was stalemated at "one person felt we
> should perform such checks" and "one person (who, incidentally, wrote 
> the code) felt we shouldn't".
> I was just hoping to encourage more people to examine the problem, so 
> that we might get a consensus one way or the other.

I hope that the issue is clearer once we have a trial implementation to
play with.
                  - Thomas


Re: Big 7.1 open items

From
Tom Lane
Date:
"Ross J. Reedstrom" <reedstrm@rice.edu> writes:
> Any strong objections to the mixed relname_oid solution?

Yes!

You cannot make it work reliably unless the relname part is the original
relname and does not track ALTER TABLE RENAME.  IMHO having an obsolete
relname in the filename is worse than not having the relname at all;
it's a recipe for confusion, it means you still need admin tools to tell
which end is really up, and what's worst is you might think you don't.

Furthermore it requires an additional column in pg_class to keep track
of the original relname, which is a waste of space and effort.

It also creates a portability risk, or at least fails to remove one,
since you are critically dependent on the assumption that the OS
supports long filenames --- on a filesystem that truncates names to less
than about 45 characters you're in very deep trouble.  An OID-only
approach still works on traditional 14-char-filename Unix filesystems
(it'd mostly even work on DOS 8+3, though I doubt we care about that).

Finally, one of the reasons I want to go to filenames based only on OID
is that that'll make life easier for mdblindwrt.  Original relname + OID
doesn't help, in fact it makes life harder (more shmem space needed to
keep track of the filename for each buffer).

Can we *PLEASE JUST LET GO* of this bad idea?  No relname in the
filename.  Period.
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Well, we did have someone do a test implementation of oid file names,
> and their report was that is looked pretty ugly.  However, if people are
> convinced it has to be done, we can get started.  I guess I was waiting
> for Vadim's storage manager, where the whole idea of separate files is
> going to go away anyway, I suspect.  We would then have to re-write all
> our admin tools for the new format.

I seem to recall him saying that he wanted to go to filename == OID
just like I'm suggesting.  But I agree we probably ought to hold off
doing anything until he gets back from Russia and can let us know
whether that's still his plan.  If he is planning one-huge-file or
something like that, we might as well let these issues go unfixed
for one more release cycle.
        regards, tom lane


Re: Big 7.1 open items

From
Karel Zak
Date:
On Wed, 14 Jun 2000, Peter Eisentraut wrote:

> On Tue, 13 Jun 2000, Karel Zak wrote:
> 
> >     + new ACL? (please :-)
> 
> Not if we're shipping in August. :(
I understand you. I said it as dream :-)

BTW. --- Are you sure, how idea for ACL will good? 
1/ your original idea with one-line-for-one-privilage in pg_privilage   (IMHO it's good idea).
2/ more priv. in one-line.
                    Karel



Re: Re: Big 7.1 open items

From
Tatsuo Ishii
Date:
> While people are working on that, they might want to add some sanity checking
> to the multibyte character decoders.  Currently they fail to check for
> "illegal" character sequences (i.e. sequences with no valid multibyte mapping),
> and fail to do something reasonable (like return an error, silently drop the
> offending characters, or anything else besides just returning random garbage
> and crashing the backend).

Hum.. I thought Michael Robinson is the one who is against the idea of
rejecting "illegal" character sequences before they are put in the DB.
I like the idea but I haven't time to do that (However I'm not sure I
would like to do it for EUC-CN, since he dislikes the codes I write).

Bruce, I would like to see followings in the TODO. I also would like
to hear from Thomas and Peter or whoever being interested in
implementing NATIONAL CHARACTER stuffs if they are reasonable.

o Don't accept character sequences those are not valid as their charset (signaling ERROR seems appropriate IMHO)

o Make PostgreSQL more multibyte aware (for example, TRIM function and NAME data type)

o Regard n of CHAR(n)/VARCHAR(n) as the number of letters, rather than the number of bytes
--
Tatsuo Ishii


Re: Big 7.1 open items

From
The Hermit Hacker
Date:
On Wed, 14 Jun 2000, Jan Wieck wrote:

>     Why not changing the naming to be something like this:
> 
>         <dbroot>/catalog_tables/pg_...
>         <dbroot>/catalog_index/pg_...
>         <dbroot>/user_tables/oid_...
>         <dbroot>/user_index/oid_...
>         <dbroot>/temp_tables/oid_...
>         <dbroot>/temp_index/oid_...
>         <dbroot>/toast_tables/oid_...
>         <dbroot>/toast_index/oid_...
>         <dbroot>/whatnot_???/...
> 
>     This way, it  would  be  much  easier  to  separate  all  the
>     different  object types to different physical media. We would
>     loose some  transparency,  but  I've  allways  wondered  what
>     people  USE  that  for  (except  for  just  wanna  know). For
>     convinience we could implement another  little  utility  that
>     tells the object size like

Wow, I've been advocating this one for how many months now? :)  You won't
get any arguments from me ... 




Re: Big 7.1 open items

From
The Hermit Hacker
Date:
On Wed, 14 Jun 2000, Bruce Momjian wrote:

> > Backtraces from *what*, exactly?  99% of the backend is still going
> > to be dealing with the same data as ever.  It might be that poking
> > around in fd.c will be a little harder, but considering that fd.c
> > doesn't really know or care what the files it's manipulating are
> > anyway, I'm not convinced that this is a real issue.
> 
> I was just throwing gdb out as an example.  The bigger ones are ls,
> lsof/fstat, and tar.

You've lost me on this one ... if someone does an lsof of the process, it
will still provide them a list of open files ... are you complaining about
the extra step required to translate the file name to a "valid table"?  

Oh, one point here ... this whole 'filenaming issue' ... as far as ls is
concerned, at least, only affects the superuser, since he's the only one
that can go 'ls'ng around i nthe directories ...

And, ummm, how hard would it be to have \d in psql display the "physical
table name" as part of its output?

Slight tangent here:

One thing that I think would be great if we could add is some sort of:

SELECT db_name, disk_space;

query wher a database owner, not the superuser, could see how much disk
space their tables are using up ... possible?



Re: Big 7.1 open items

From
"Mark Hollomon"
Date:
Ross J. Reedstrom wrote:
> 
> Any strong objections to the mixed relname_oid solution? It gets us
> everything oids does, and still lets Bruce use 'ls -l' to find the big
> tables, putting off writing any admin tools that'll need to be rewritten,
> anyway.

I would object to the mixed name.

Consider:

CREATE TABLE FOO ....
ALTER TABLE FOO RENAME FOO_OLD;
CREATE TABLE FOO ....

For the same atomicity reason, rename can't change the
name of the files. So, which foo_<oid> is the FOO_OLD
and which is FOO?

In other words, in the presence of rename, putting
relname in the filename is misleading at best.

-- 

Mark Hollomon
mhh@nortelnetworks.com
ESN 451-9008 (302)454-9008


Re: Big 7.1 open items

From
Brian E Gallew
Date:
Then <tgl@sss.pgh.pa.us> spoke up and said:
> Precedence: bulk
> 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > But seriously, let me give some background.  I used Ingres, that used
> > the VMS file system, but used strange sequential AAAF324 numbers for
> > tables.  When someone deleted a table, or we were looking at what tables
> > were using disk space, it was impossible to find the Ingres table names
> > that went with the file.  There was a system table that showed it, but
> > it was poorly documented, and if you deleted the table, there was no way
> > to look on the tape to find out which file to restore.
> 
> Fair enough, but it seems to me that the answer is to expend some effort
> on system admin support tools.  We could do a lot in that line with less
> effort than trying to make a fundamentally mismatched filesystem
> representation do what we need.

We've been an Ingres shop as long as there's been an Ingres.  While
we've also had the problem Bruce noticed with table names, we've
*also* used the trivial fix of running a (simple) Report Writer job
each night, immediately before the backup, that lists all of the
database tables/indicies and the underlying files.

True, if someone drops/recreates a table twice between backups we
can't find the intermediate file name, but since we also haven't
backed up that filename, this isn't an issue.

Also, the consistency issue is really not as important as you would
think.  If you are restoring a table, you want the information in it,
whether or not it's consistent with anything else.  I've done hundreds
of table restores (can you say "modify table to heap"?) and never once
has inconsistency been an issue.  Oh, yeah, and we don't shut the
database down for this, either.  (That last isn't my choice, BTW.)

-- 
=====================================================================
| JAVA must have been developed in the wilds of West Virginia.      |
| After all, why else would it support only single inheritance??    |
=====================================================================
| Finger geek@cmu.edu for my public key.                            |
=====================================================================

Re: Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Hum.. I thought Michael Robinson is the one who is against the idea of
> rejecting "illegal" character sequences before they are put in the DB.
> I like the idea but I haven't time to do that (However I'm not sure I
> would like to do it for EUC-CN, since he dislikes the codes I write).
> 
> Bruce, I would like to see followings in the TODO. I also would like
> to hear from Thomas and Peter or whoever being interested in
> implementing NATIONAL CHARACTER stuffs if they are reasonable.
> 
> o Don't accept character sequences those are not valid as their charset
>   (signaling ERROR seems appropriate IMHO)
> 
> o Make PostgreSQL more multibyte aware (for example, TRIM function and
>   NAME data type)
> 
> o Regard n of CHAR(n)/VARCHAR(n) as the number of letters, rather than
>   the number of bytes

Added to TODO:

* Reject character sequences those are not valid in their charset          * Make functions more multi-byte aware, i.e.
trim()
 
* Make n of CHAR(n)/VARCHAR(n) the number of letters, not bytes     


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > But seriously, let me give some background.  I used Ingres, that used
> > the VMS file system, but used strange sequential AAAF324 numbers for
> > tables.  When someone deleted a table, or we were looking at what 
> > tables were using disk space, it was impossible to find the Ingres 
> > table names that went with the file.  There was a system table that 
> > showed it, but it was poorly documented, and if you deleted the table, 
> > there was no way to look on the tape to find out which file to 
> > restore.
> 
> I had the same experience, but let's put the blame where it belongs: it
> wasn't the filename's fault, it was poor design and support from the
> Ingres company.

Yes, that certainly was part of the cause.  Also, if the PostgreSQL
files are backed up using tar while no database activity is happening,
there is no reason the tar restore will not work.  You just create a
table with the same schema, stop the postmaster, have the backup file
replace the newly created table file, and restart the postmaster.

I can't tell you how many times I have said, "Man, whoever did this
Ingres naming schema was an idiot.  Do they know how many problems they
caused for us?"

Also, Informix standard engine uses the tablename_oid setup for its
table names, and it works fine.  It grabs the first 8 characters of the
table, and plops some unique number on the end of it.  Works fine for
administrators.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Re: Big 7.1 open items

From
Thomas Lockhart
Date:
> o Don't accept character sequences those are not valid as their 
>   charset (signaling ERROR seems appropriate IMHO)
> o Make PostgreSQL more multibyte aware (for example, TRIM function and
>   NAME data type)
> o Regard n of CHAR(n)/VARCHAR(n) as the number of letters, rather than
>   the number of bytes

All good, and important features when we are done.

One issue: I can see (or imagine ;) how we can use the Postgres type
system to manage multiple character sets. But allowing arbitrary
character sets in, say, table names forces us to cope with allowing a
mix of character sets in a single column of a system table. afaik this
general capability is not mandated by SQL9x (the SQL_TEXT character set
is used for all system resources??). Would it be acceptable to have a
"default database character set" which is allowed to creep into the
pg_xxx tables? Even that seems to be a difficult thing to accomplish at
the moment (we'd need to get some of the text manipulation functions
from the catalogs, not from hardcoded references as we do now).

We should itemize all of these issues so we can keep track of what is
necessary, possible, and/or "easy".
              - Thomas


Re: Big 7.1 open items

From
"Ross J. Reedstrom"
Date:
On Thu, Jun 15, 2000 at 03:11:52AM -0400, Tom Lane wrote:
> "Ross J. Reedstrom" <reedstrm@rice.edu> writes:
> > Any strong objections to the mixed relname_oid solution?
> 
> Yes!
> 
> You cannot make it work reliably unless the relname part is the original
> relname and does not track ALTER TABLE RENAME.  IMHO having an obsolete
> relname in the filename is worse than not having the relname at all;
> it's a recipe for confusion, it means you still need admin tools to tell
> which end is really up, and what's worst is you might think you don't.

The plan here was to let VACUUM handle renaming the file, since it
will already have all the necessary locks. This shortens the window
of confusion.  ALTER TABLE RENAME doesn't happen that often, really - 
the relname is there just for human consumption, then.

> 
> Furthermore it requires an additional column in pg_class to keep track
> of the original relname, which is a waste of space and effort.
> 

I actually started down this path thinking about implementing SCHEMA,
since tables in the same DB but in different schema can have the same
relname, I figured I needed to change that. We'll need something in
pg_class to keep track of what schema a relation is in, instead.

> It also creates a portability risk, or at least fails to remove one,
> since you are critically dependent on the assumption that the OS
> supports long filenames --- on a filesystem that truncates names to less
> than about 45 characters you're in very deep trouble.  An OID-only
> approach still works on traditional 14-char-filename Unix filesystems
> (it'd mostly even work on DOS 8+3, though I doubt we care about that).

Actually, no. Since I store the filename in a name attribute, I used this
nifty function somebody wrote, makeObjectName, to trim the relname part,
but leave the oid. (Yes, I know it's yours ;-)

> 
> Finally, one of the reasons I want to go to filenames based only on OID
> is that that'll make life easier for mdblindwrt.  Original relname + OID
> doesn't help, in fact it makes life harder (more shmem space needed to
> keep track of the filename for each buffer).

Can you explain in more detail how this helps? Not by letting the bufmgr
know that oid == filename, I hope. We need to improving the abstraction
of the smgr, not add another violation. Ah, sorry, mdblindwrt _is_
in the smgr. 

Hmm, grovelling through that code, I see how it could be simpler if reloid
== filename. Heck, we even get to save shmem in the buffdesc.blind part,
since we only need the dbname in there, now.

Hmm, I see I missed the relpath_blind() in my patch - oops.  (relpath()
is always called with RelationGetPhysicalRelationName(), and that's
where I was putting in the relphysname)

Hmm, what's all this with functions in catalog.c that are only called by
smgr/md.c? seems to me that anything having to do with physical storage
(like the path!) belongs in the smgr abstraction.

> 
> Can we *PLEASE JUST LET GO* of this bad idea?  No relname in the
> filename.  Period.
> 

Gee, so dogmatic. No one besides Bruce and Hiroshi discussed this _at
all_ when I first put up patches two month ago. O.K., I'll do the oids
only version (and fix up relpath_blind)

Ross

-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > Can we *PLEASE JUST LET GO* of this bad idea?  No relname in the
> > filename.  Period.
> > 
> 
> Gee, so dogmatic. No one besides Bruce and Hiroshi discussed this _at
> all_ when I first put up patches two month ago. O.K., I'll do the oids
> only version (and fix up relpath_blind)

Hold on.  I don't think we want that work done yet.  Seems even Tom is
thinking that if Vadim is going to re-do everything later anyway, we may
be better with a relname/oid solution that does require additional
administration apps.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: pgsql-hackers-owner@hub.org 
> [mailto:pgsql-hackers-owner@hub.org]On Behalf Of Bruce Momjian
> 
> > > Can we *PLEASE JUST LET GO* of this bad idea?  No relname in the
> > > filename.  Period.
> > > 
> > 
> > Gee, so dogmatic. No one besides Bruce and Hiroshi discussed this _at
> > all_ when I first put up patches two month ago. O.K., I'll do the oids
> > only version (and fix up relpath_blind)
> 
> Hold on.  I don't think we want that work done yet.  Seems even Tom is
> thinking that if Vadim is going to re-do everything later anyway, we may
> be better with a relname/oid solution that does require additional
> administration apps.
>

Hmm,why is naming rule first ?

I've never enphasized naming rule except that it should be unique.
It has been my main point to reduce the necessity of naming rule
as possible. IIRC,by keeping the stored place in pg_class,Ross's
trial patch remains only 2 places where naming rule is required. 
So wouldn't we be free from naming rule(it would not be so difficult
to change naming rule if the rule is found to be bad) ? 

I've also mentioned many times neither relname nor oid is sufficient
for the uniqueness. In addiiton neither relname nor oid would be
necessary for the uniqueness.
IMHO,it's bad to rely on the item which is neither necessary nor
sufficient.
I proposed relname+unique_id naming once. The unique_id is
independent from oid. The relname is only for convinience for
DBA and so we don't have to change it due to RENAME.
Db's consistency is much more important than dba's satis-
faction.

Comments ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp



Re: Big 7.1 open items

From
Bruce Momjian
Date:
> I've also mentioned many times neither relname nor oid is sufficient
> for the uniqueness. In addiiton neither relname nor oid would be
> necessary for the uniqueness.
> IMHO,it's bad to rely on the item which is neither necessary nor
> sufficient.
> I proposed relname+unique_id naming once. The unique_id is
> independent from oid. The relname is only for convinience for
> DBA and so we don't have to change it due to RENAME.
> Db's consistency is much more important than dba's satis-
> faction.
> 
> Comments ?

I am happy not to rename the file on 'RENAME', but seems no one likes
that.


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
"Ross J. Reedstrom"
Date:
On Thu, Jun 15, 2000 at 05:48:59PM -0400, Bruce Momjian wrote:
> > I've also mentioned many times neither relname nor oid is sufficient
> > for the uniqueness. In addiiton neither relname nor oid would be
> > necessary for the uniqueness.
> > IMHO,it's bad to rely on the item which is neither necessary nor
> > sufficient.
> > I proposed relname+unique_id naming once. The unique_id is
> > independent from oid. The relname is only for convinience for
> > DBA and so we don't have to change it due to RENAME.
> > Db's consistency is much more important than dba's satis-
> > faction.
> > 
> > Comments ?
> 
> I am happy not to rename the file on 'RENAME', but seems no one likes
> that.

Good, 'cause that's how I've implemented it so far. Actually, all
I've done is port my previous patch to current, with one little
change: I added a macro RelationGetRealRelationName which does what
RelationGetPhysicalRelationName used to do: i.e. return the relname with
no temptable funny business, and used that for the relcache macros. It
passes all the serial regression tests: I haven't run the parallel tests
yet. ALTER TABLE RENAME rollsback nicely. I'll need to learn some omre
about xacts to get DROP TABLE rolling back.

I'll drop it on PATCHES right now, for comment.

Ross
-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005


Re: Big 7.1 open items

From
Tom Lane
Date:
"Ross J. Reedstrom" <reedstrm@rice.edu> writes:
> On Thu, Jun 15, 2000 at 03:11:52AM -0400, Tom Lane wrote:
>> "Ross J. Reedstrom" <reedstrm@rice.edu> writes:
>>>> Any strong objections to the mixed relname_oid solution?
>> 
>> Yes!

> The plan here was to let VACUUM handle renaming the file, since it
> will already have all the necessary locks. This shortens the window
> of confusion.  ALTER TABLE RENAME doesn't happen that often, really - 
> the relname is there just for human consumption, then.

Yeah, I've seen tons of discussion of how if we do this, that, and
the other thing, and be prepared to fix up some other things in case
of crash recovery, we can make it work with filename == relname + OID
(where relname tracks logical name, at least at some remove).

Probably.  Assuming nobody forgets anything.

I'm just trying to point out that that's a huge amount of pretty
delicate mechanism.  The amount of work required to make it trustworthy
looks to me to dwarf the admin tools that Bruce is complaining about.
And we only have a few people competent to do the work.  (With all
due respect, Ross, if you weren't already aware of the implications
for mdblindwrt, I have to wonder what else you missed.)

Filename == OID is so simple, reliable, and straightforward by
comparison that I think the decision is a no-brainer.

If we could afford to sink unlimited time into this one issue then
it might make sense to do it the hard way, but we have enough
important stuff on our TODO list to keep us all busy for years ---
I cannot believe that it's an effective use of our time to do this.


> Hmm, what's all this with functions in catalog.c that are only called by
> smgr/md.c? seems to me that anything having to do with physical storage
> (like the path!) belongs in the smgr abstraction.

Yeah, there's a bunch of stuff that should have been implemented by
adding new smgr entry points, but wasn't.  It should be pushed down.
(I can't resist pointing out that one of those things is physical
relation rename, which will go away and not *need* to be pushed down
if we do it the way I want.)
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Gee, so dogmatic. No one besides Bruce and Hiroshi discussed this _at
>> all_ when I first put up patches two month ago. O.K., I'll do the oids
>> only version (and fix up relpath_blind)

> Hold on.  I don't think we want that work done yet.  Seems even Tom is
> thinking that if Vadim is going to re-do everything later anyway, we may
> be better with a relname/oid solution that does require additional
> administration apps.

Don't put words in my mouth, please.  If we are going to throw the
work away later, it'd be foolish to do the much greater amount of
work needed to make filename=relname+OID fly than is needed for
filename=OID.

However, I'm pretty sure I recall Vadim stating that he thought
filename=OID would be required for his smgr changes anyway...
        regards, tom lane


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: pgsql-hackers-owner@hub.org [mailto:pgsql-hackers-owner@hub.org]On
> Behalf Of Tom Lane
> 
> "Ross J. Reedstrom" <reedstrm@rice.edu> writes:
> > On Thu, Jun 15, 2000 at 03:11:52AM -0400, Tom Lane wrote:
> >> "Ross J. Reedstrom" <reedstrm@rice.edu> writes:
> >>>> Any strong objections to the mixed relname_oid solution?
> >> 
> >> Yes!
> 
> > The plan here was to let VACUUM handle renaming the file, since it
> > will already have all the necessary locks. This shortens the window
> > of confusion.  ALTER TABLE RENAME doesn't happen that often, really - 
> > the relname is there just for human consumption, then.
> 
> Yeah, I've seen tons of discussion of how if we do this, that, and
> the other thing, and be prepared to fix up some other things in case
> of crash recovery, we can make it work with filename == relname + OID
> (where relname tracks logical name, at least at some remove).
>

I've seen little discussion of how to avoid the use of naming rule.
I've proposed many times that we should keep the information
where the table is stored in our database itself. I've never seen
clear objections to it. So I could understand my proposal is OK ? 
Isn't it much more important than naming rule ?  Under the
mechanism,we could easily replace bad naming rule.
And I believe that Ross's work is mostly around the mechanism
not naming rule. 

Now I like neither relname nor oid because it's not sufficient 
for my purpose. 
Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> Now I like neither relname nor oid because it's not sufficient 
> for my purpose.

We should probably not do much of anything with this issue until
we have a clearer understanding of what we want to do about
tablespaces and schemas.

My gut feeling is that we will end up with pathnames that look
something like

.../data/base/DBNAME/TABLESPACE/OIDOFRELATION

(with .N attached if a segment of a large relation, of course).

The TABLESPACE "name" should likely be an OID itself, but it wouldn't
have to be if you are willing to say that tablespaces aren't renamable.
(Come to think of it, does anyone care about being able to rename
databases?  ;-))  Note that the TABLESPACE will often be a symlink
to storage on another drive, rather than a plain subdirectory of the
DBNAME, but that shouldn't be an issue at this level of discussion.

I think that schemas probably don't enter into this.  We should instead
rely on the uniqueness of OIDs to prevent filename collisions.  However,
OIDs aren't really unique: different databases in an installation will
use the same OIDs for their system tables.  My feeling is that we can
live with a restriction like "you can't store the system tables of
different databases in the same tablespace".  Alternatively we could
avoid that issue by inverting the pathname order:

.../data/base/TABLESPACE/DBNAME/OIDOFRELATION

Note that in any case, system tables will have to live in a
predetermined tablespace, since you can't very well look in pg_class
to find out which tablespace pg_class lives in.  Perhaps we should
just reserve a tablespace per database for system tables and forget
the whole issue.  If we do that, there's not really any need for
the database in the path!  Just

.../data/base/TABLESPACE/OIDOFRELATION

would do fine and help reduce lookup overhead.

BTW, schemas do make things interesting for the other camp:
is it possible for the same table to be referenced by different
names in different schemas?  If so, just how useful is it to pick
one of those names arbitrarily for the filename?  This is an advanced
version of the main objection to using the original relname and not
updating it at RENAME TABLE --- sooner or later, the filenames are
going to be more confusing than helpful.

Comments?  Have I missed something important about schemas?
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > Now I like neither relname nor oid because it's not sufficient 
> > for my purpose.
> 
> We should probably not do much of anything with this issue until
> we have a clearer understanding of what we want to do about
> tablespaces and schemas.

Here is an analysis of our options:
                         Work required             Disadvantages
----------------------------------------------------------------------------

Keep current system       no work                   rename/create no rollback

relname/oid but           less work                 new pg_class column,
no rename change                                    filename not accurate on
      rename
 

relname/oid with          more work                 complex code
rename change during      
vacuum

oid filename              less work, but            confusing to admins                         need admin tools
 
 

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
Sorry for my previous mail. It was posted by my mistake.

> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > Now I like neither relname nor oid because it's not sufficient 
> > for my purpose.
> 
> We should probably not do much of anything with this issue until
> we have a clearer understanding of what we want to do about
> tablespaces and schemas.
> 
> My gut feeling is that we will end up with pathnames that look
> something like
>
> .../data/base/DBNAME/TABLESPACE/OIDOFRELATION
>

Schema is a logical concept and irrevant to physical location.
I strongly object your suggestion unless above means *default*
location.
Tablespace is an encapsulation of table allocation and the 
name should be irrevant to the location basically. So above
seems very bad for me.

Anyway I don't see any advantage in fixed mapping impleme
ntation. After renewal,we should at least have a possibility to
allocate a specific table in arbitrary separate directory.

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
> 
> > "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > > Now I like neither relname nor oid because it's not sufficient 
> > > for my purpose.
> > 
> > We should probably not do much of anything with this issue until
> > we have a clearer understanding of what we want to do about
> > tablespaces and schemas.
> 
> Here is an analysis of our options:
> 
>                           Work required             Disadvantages
> ------------------------------------------------------------------
> ----------
> 
> Keep current system       no work                   rename/create 
> no rollback
> 
> relname/oid but           less work                 new pg_class column,
> no rename change                                    filename not 
> accurate on
>                                                     rename
> 
> relname/oid with          more work                 complex code
> rename change during      
> vacuum
> 
> oid filename              less work, but            confusing to admins
>                           need admin tools          
>

Please add my opinion for naming rule.

relname/unique_id but    need some work        new pg_class column,    
no relname change.    for unique-id generation    filename not relname

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> Tablespace is an encapsulation of table allocation and the 
> name should be irrevant to the location basically. So above
> seems very bad for me.
> Anyway I don't see any advantage in fixed mapping impleme
> ntation. After renewal,we should at least have a possibility to
> allocate a specific table in arbitrary separate directory.

Call a "directory" a "tablespace" and we're on the same page,
aren't we?  Actually I'd envision some kind of admin command
"CREATE TABLESPACE foo AS /path/to/wherever". That would make
appropriate system catalog entries and also create a symlink
from ".../data/base/foo" (or some such place) to the target
directory.  Then when we make a table in that tablespace,
it's in the right place.  Problem solved, no?

It gets a little trickier if you want to be able to split
multi-gig tables across several tablespaces, though, since
you couldn't just append ".N" to the base table path in that
scenario.

I'd be interested to know what sort of facilities Oracle
provides for managing huge tables...
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> Please add my opinion for naming rule.

> relname/unique_id but    need some work        new pg_class column,    
> no relname change.    for unique-id generation    filename not relname

Why is a unique ID better than --- or even different from ---
using the relation's OID?  It seems pointless to me...
        regards, tom lane


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > Please add my opinion for naming rule.
> 
> > relname/unique_id but    need some work        new 
> pg_class column,    
> > no relname change.    for unique-id generation    filename not relname
> 
> Why is a unique ID better than --- or even different from ---
> using the relation's OID?  It seems pointless to me...
>

For example,in the implementation of CLUSTER command,
we would need another new file for the target relation in
order to put sorted rows but don't we want to change the
OID ? It would be needed for table re-construction generally.
If I remember correectly,you once proposed OID+version
naming for the cases.

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > Tablespace is an encapsulation of table allocation and the 
> > name should be irrevant to the location basically. So above
> > seems very bad for me.
> > Anyway I don't see any advantage in fixed mapping impleme
> > ntation. After renewal,we should at least have a possibility to
> > allocate a specific table in arbitrary separate directory.
> 
> Call a "directory" a "tablespace" and we're on the same page,
> aren't we?  Actually I'd envision some kind of admin command
> "CREATE TABLESPACE foo AS /path/to/wherever". 

Yes,I think 'tablespace -> directory' is the most natural
extension under current file_per_table storage manager.
If many_tables_in_a_file storage manager is introduced,we
may be able to change the definiiton of TABLESPACE
to 'tablespace -> files'  like Oracle.

> That would make
> appropriate system catalog entries and also create a symlink
> from ".../data/base/foo" (or some such place) to the target
> directory.
> Then when we make a table in that tablespace,
> it's in the right place.  Problem solved, no?
> 

I don't like symlink for dbms data files. However it may
be OK,If symlink are limited to 'tablespace->directory'
corrspondence and all tablespaces(including default
etc) are symlink.  It is simple and all debugging would
be processed under tablespace_is_symlink environment.

> It gets a little trickier if you want to be able to split
> multi-gig tables across several tablespaces, though, since
> you couldn't just append ".N" to the base table path in that
> scenario.
>

This seems to be not that easy to solve now.
Ross doesn't change this naming rule for multi-gig
tables either in his trial.
> I'd be interested to know what sort of facilities Oracle
> provides for managing huge tables...
>

In my knowledge about old Oracle,one TABLESPACE
could have many DATAFILEs which could contain
many tables.
Regards.

Hiroshi Inoue
Inoue@tpf.co.jp 


Re: Big 7.1 open items

From
Chris Bitmead
Date:
Tom Lane wrote:

> >         <dbroot>/catalog_tables/pg_...
> >         <dbroot>/catalog_index/pg_...
> >         <dbroot>/user_tables/oid_...
> >         <dbroot>/user_index/oid_...
> >         <dbroot>/temp_tables/oid_...
> >         <dbroot>/temp_index/oid_...
> >         <dbroot>/toast_tables/oid_...
> >         <dbroot>/toast_index/oid_...
> >         <dbroot>/whatnot_???/...
> 
> I don't see a lot of value in that.  Better to do something like
> tablespaces:
> I don't see a lot of value in that.  Better to do something like
> tablespaces:
>
>        <dbroot>/<oidoftablespace>/<oidofobject>

What is the benefit of having oidoftablespace in the directory path?
Isn't tablespace an idea so you can store it somewhere completely
different?
Or is there some symlink idea or something?


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
>> Why is a unique ID better than --- or even different from ---
>> using the relation's OID?  It seems pointless to me...

> For example,in the implementation of CLUSTER command,
> we would need another new file for the target relation in
> order to put sorted rows but don't we want to change the
> OID ? It would be needed for table re-construction generally.
> If I remember correectly,you once proposed OID+version
> naming for the cases.

Hmm, so you are thinking that the pg_class row for the table would
include this uniqueID, and then committing the pg_class update would
be the atomic action that replaces the old table contents with the
new?  It does have some attraction now that I think about it.

But there are other ways we could do the same thing.  If we want to
have tablespaces, there will need to be a tablespace identifier in
each pg_class row.  So we could do CLUSTER in the same way as we'd
move a table from one tablespace to another: create the new files in
the new tablespace directory, and the commit of the new pg_class row
with the new tablespace value is the atomic action that makes the new
files valid and the old files not.

You will probably say "but I didn't want to move my table to a new
tablespace just to cluster it!"  I think we could live with that,
though.  A tablespace doesn't need to have any existence more concrete
than a subdirectory, in my vision of the way things would work.  We 
could do something like making two subdirectories of each place that
the dbadmin designates as a "tablespace", so that we make two logical
tablespaces out of what the dbadmin thinks of as one.  Then we can
ping-pong between those directories to do things like clustering "in
place".

Basically I want to keep the bottom-level mechanisms as simple and
reliable as we possibly can.  The fewer concepts are known down at
the bottom, the better.  If we can keep the pathname constituents
to just "tablespace" and "relation OID" we'll be in great shape ---
but each additional concept that has to be known down there is
another potential problem.
        regards, tom lane


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> >> Why is a unique ID better than --- or even different from ---
> >> using the relation's OID?  It seems pointless to me...
> 
> > For example,in the implementation of CLUSTER command,
> > we would need another new file for the target relation in
> > order to put sorted rows but don't we want to change the
> > OID ? It would be needed for table re-construction generally.
> > If I remember correectly,you once proposed OID+version
> > naming for the cases.
> 
> Hmm, so you are thinking that the pg_class row for the table would
> include this uniqueID, 

No,I just include the place where the table is stored(pathname under
current file_per_table storage manager) in the pg_class row because
I don't want to rely on table allocating rule(naming rule for current)
to access existent relation files. This has always been my main point.
Many_tables_in_a_file storage manager wouldn't be able to live without
keeping this kind of infomation.
This information(where it is stored) is diffrent from tablespace(where
to store) information. There was an idea to keep the information into
opaque entry in pg_class which only a specific storage manager
could handle. There was an idea to have a new system table which
keeps the information. and so on...

> and then committing the pg_class update would
> be the atomic action that replaces the old table contents with the
> new?  It does have some attraction now that I think about it.
> 
> But there are other ways we could do the same thing.  If we want to
> have tablespaces, there will need to be a tablespace identifier in
> each pg_class row.  So we could do CLUSTER in the same way as we'd
> move a table from one tablespace to another: create the new files in
> the new tablespace directory, and the commit of the new pg_class row
> with the new tablespace value is the atomic action that makes the new
> files valid and the old files not.
> 
> You will probably say "but I didn't want to move my table to a new
> tablespace just to cluster it!" 

Yes.

> I think we could live with that,
> though.  A tablespace doesn't need to have any existence more concrete
> than a subdirectory, in my vision of the way things would work.  We 
> could do something like making two subdirectories of each place that
> the dbadmin designates as a "tablespace", so that we make two logical
> tablespaces out of what the dbadmin thinks of as one. 

Certainly we could design TABLESPACE(where to store) as above.

> Then we can
> ping-pong between those directories to do things like clustering "in
> place".
>

But maybe we must keep the directory information where the table was 
*ping-ponged* in (e.g.) pg_class. Is such an implementation cleaner or
more extensible than mine(keeping the stored place exactly) ?   
Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


Re: Big 7.1 open items

From
Tom Lane
Date:
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au> writes:
> Tom Lane wrote:
>> I don't see a lot of value in that.  Better to do something like
>> tablespaces:
>> 
>> <dbroot>/<oidoftablespace>/<oidofobject>

> What is the benefit of having oidoftablespace in the directory path?
> Isn't tablespace an idea so you can store it somewhere completely
> different?
> Or is there some symlink idea or something?

Exactly --- I'm assuming that the tablespace "directory" is likely
to be a symlink to some other mounted volume.  The point here is
to keep the low-level file access routines from having to know very
much about tablespaces or file organization.  In the above proposal,
all they need to know is the relation's OID and the name (or OID)
of the tablespace the relation's assigned to; then they can form
a valid path using a hardwired rule.  There's still plenty of
flexibility of organization, but it's not necessary to know that
where the rubber meets the road (eg, when you're down inside mdblindwrt
trying to dump a dirty buffer to disk with no spare resources to find
out anything about the relation the page belongs to...)
        regards, tom lane


Re: Re: Big 7.1 open items

From
Tatsuo Ishii
Date:
> > o Don't accept character sequences those are not valid as their 
> >   charset (signaling ERROR seems appropriate IMHO)
> > o Make PostgreSQL more multibyte aware (for example, TRIM function and
> >   NAME data type)
> > o Regard n of CHAR(n)/VARCHAR(n) as the number of letters, rather than
> >   the number of bytes
> 
> All good, and important features when we are done.

Glad to hear that.

> One issue: I can see (or imagine ;) how we can use the Postgres type
> system to manage multiple character sets. But allowing arbitrary
> character sets in, say, table names forces us to cope with allowing a
> mix of character sets in a single column of a system table. afaik this
> general capability is not mandated by SQL9x (the SQL_TEXT character set
> is used for all system resources??). Would it be acceptable to have a
> "default database character set" which is allowed to creep into the
> pg_xxx tables? Even that seems to be a difficult thing to accomplish at
> the moment (we'd need to get some of the text manipulation functions
> from the catalogs, not from hardcoded references as we do now).

"default database character set" idea does not seem to be the solution
for cross-db relations such as pg_database. The only solution I can
imagine so far is using SQL_TEXT.

BTW, I've been thinking about SQL_TEXT for a while and it seems
mule_internal_code or Unicode(UTF-8) would be the candidates for
it. Mule_internal_code looks more acceptable for Asian multi-byte
users like me than Unicode. It's clean, simple and does not require
huge conversion tables between Unicode and other encodings. However,
Unicode has a stronger political power in the real world and for most
single-byte users probably it would be enough. My idea is let users
choose one of them. I mean making it a compile time option.

> We should itemize all of these issues so we can keep track of what is
> necessary, possible, and/or "easy".

You are right, probably there would be tons of issues in implementing
multiple charsets support.
--
Tatsuo Ishii


Re: Re: Big 7.1 open items

From
Thomas Lockhart
Date:
> "default database character set" idea does not seem to be the solution
> for cross-db relations such as pg_database. The only solution I can
> imagine so far is using SQL_TEXT.
> BTW, I've been thinking about SQL_TEXT for a while and it seems
> mule_internal_code or Unicode(UTF-8) would be the candidates for
> it. Mule_internal_code looks more acceptable for Asian multi-byte
> users like me than Unicode. It's clean, simple and does not require
> huge conversion tables between Unicode and other encodings. However,
> Unicode has a stronger political power in the real world and for most
> single-byte users probably it would be enough. My idea is let users
> choose one of them. I mean making it a compile time option.

Oh. I was recalling SQL_TEXT as being a "subset" character set which
contains only the characters (more or less) that are required for
implementing the SQL92 query language and standard features.

Are you seeing it as being a "superset" character set which can
represent all other character sets??

And, how would you suggest we start tracking this discussion in a design
document? I could put something into the developer's guide, or we could
have a plain-text FAQ, or ??

I'd propose that we start accumulating a feature list, perhaps ordering
it into categories like

o required/suggested by SQL9x
o required/suggested by experience in the real world
o sure would be nice to have
o really bad idea ;)
                 - Thomas


Re: Big 7.1 open items

From
JanWieck@t-online.de (Jan Wieck)
Date:
Tom Lane wrote:
>
> It gets a little trickier if you want to be able to split
> multi-gig tables across several tablespaces, though, since
> you couldn't just append ".N" to the base table path in that
> scenario.
>
> I'd be interested to know what sort of facilities Oracle
> provides for managing huge tables...
   Oracle  tablespaces  are  a  collection of 1...n preallocated   files.   Each  table  then  is  bound  to  a
tablespace and   allocates extents (chunks) from those files.
 
   There  are  some per table attributes that control the extent   sizes with default values coming from  the
tablespace.  The   initial  extent  size,  the  nextextent  and the pctincrease.   There is a hardcoded limit for the
numberof extents a  table   can  have at all.  In Oracle7 it was 512 (or somewhat below -   don't recall correct).
Maybethat's gone with Oracle8,  don't   know.
 
   This  storage  concept  has  IMHO  a couple of advatages over   ours.
       The tablespace files  are  preallocated,  so  there  will       never  be a change in block allocation during
runtimeand       that's the base  for  fdatasync()  beeing  sufficient  at       syncpoints. All what might be
inaccurateafter a crash is       the last modified time in the inode, and  that's  totally       irrelevant  for
Oracle. The  fsck  will never fail, and       anything is up to Oracle's recovery.
 
       The number of total tablespace  files  is  limited  to  a       value  that  ensures, that the backends can keep
themall       open all the time. It's hard  to  exceed  that  limit.  A       typical   SAP   installation   with
more  than  20,000       tables/indices doesn't need more than 30 or 40 of them.
 
       It  is  perfectly  prepared  for  raw  devices,  since  a       tablespace in a raw device installation is
simplyan area       of blocks on a disk.
 
   There are also disadvantages.
       You can run out of space even if there  are  plenty  GB's       free  on  your  disks.   You  have  to create
tablespaces      explicitly.
 
       If you've choosen inadequate extent size parameters,  you       end  up with high fragmented tables (slowing
down)or get       stuck with running against maxextents, where only a reorg       (export/import) helps.
 


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #




Re: Big 7.1 open items

From
Tom Lane
Date:
JanWieck@t-online.de (Jan Wieck) writes:
>     There are also disadvantages.

>         You can run out of space even if there  are  plenty  GB's
>         free  on  your  disks.   You  have  to create tablespaces
>         explicitly.

Not to mention the reverse: if I read this right, you have to suck
up your GB's long in advance of actually needing them.  That's OK
for a machine that's dedicated to Oracle ... not so OK for smaller
installations, playpens, etc.

I'm not convinced that there's anything fundamentally wrong with
doing storage allocation in Unix files the way we have been.

(At least not when we're sitting atop a well-done filesystem,
which may leave the Linux folk out in the cold ;-).)
        regards, tom lane


Re: Big 7.1 open items

From
Thomas Lockhart
Date:
> (At least not when we're sitting atop a well-done filesystem,
> which may leave the Linux folk out in the cold ;-).)

Those who live in HP houses should not throw stones :))
              - Thomas


Re: Big 7.1 open items

From
Tom Lane
Date:
JanWieck@t-online.de (Jan Wieck) writes:
> Tom Lane wrote:
>> It gets a little trickier if you want to be able to split
>> multi-gig tables across several tablespaces, though, since
>> you couldn't just append ".N" to the base table path in that
>> scenario.
>> 
>> I'd be interested to know what sort of facilities Oracle
>> provides for managing huge tables...

>     Oracle  tablespaces  are  a  collection of 1...n preallocated
>     files.   Each  table  then  is  bound  to  a  tablespace  and
>     allocates extents (chunks) from those files.

OK, to get back to the point here: so in Oracle, tables can't cross
tablespace boundaries, but a tablespace itself could span multiple
disks?

Not sure if I like that better or worse than equating a tablespace
with a directory (so, presumably, all the files within it live on
one filesystem) and then trying to make tables able to span
tablespaces.  We will need to do one or the other though, if we want
to have any significant improvement over the current state of affairs
for large tables.

One way is to play the flip-the-path-ordering game some more,
and access multiple-segment tables with pathnames like this:
.../TABLESPACE/RELATION        -- first or only segment.../TABLESPACE/N/RELATION    -- N'th extension segment

This isn't any harder for md.c to deal with than what we do now,
but by making the /N subdirectories be symlinks, the dbadmin could
easily arrange for extension segments to go on different filesystems.
Also, since /N subdirectory symlinks can be added as needed,
expanding available space by attaching more disks isn't hard.
(If the admin hasn't pre-made a /N symlink when it's needed,
I'd envision the backend just automatically creating a plain
subdirectory so that it can extend the table.)

A limitation is that the N'th extension segments of all the relations
in a given tablespace have to be in the same place, but I don't see
that as a major objection.  Worst case is you make a separate tablespace
for each of your multi-gig relations ... you're probably not going to
have a very large number of such relations, so this doesn't seem like
unmanageable admin complexity.

We'd still want to create some tools to help the dbadmin with slinging
all these symlinks around, of course.  But I think it's critical to keep
the low-level file access protocol simple and reliable, which really
means minimizing the amount of information the backend needs to know to
figure out which file to write a page in.  With something like the above
you only need to know the tablespace name (or more likely OID), the
relation OID (+name or not, depending on outcome of other argument),
and the offset in the table.  No worse than now from the software's
point of view.

Comments?
        regards, tom lane


Re: Big 7.1 open items

From
Thomas Lockhart
Date:
> ...  But I think it's critical to keep
> the low-level file access protocol simple and reliable, which really
> means minimizing the amount of information the backend needs to know 
> to figure out which file to write a page in.  With something like the 
> above you only need to know the tablespace name (or more likely OID), 
> the relation OID (+name or not, depending on outcome of other 
> argument), and the offset in the table.  No worse than now from the 
> software's point of view.
> Comments?

I'm probably missing the context a bit, but imho we should try hard to
stay away from symlinks as the general solution for anything.

Sorry for being behind here, but to make sure I'm on the right page:
o tablespaces decouple storage from logical tables
o a database lives in a default tablespace, unless specified
o by default, a table will live in the default tablespace
o (eventually) a table can be split across tablespaces

Some thoughts:
o the ability to split single tables across disks was essential for
scalability when disks were small. But with RAID, NAS, etc etc isn't
that a smaller issue now?
o "tablespaces" would implement our less-developed "with location"
feature, right? Splitting databases, whole indices and whole tables
across storage is the biggest win for this work since more users will
use the feature.
o location information needs to travel with individual tables anyway.


Re: Big 7.1 open items

From
Bruce Momjian
Date:
>     There are also disadvantages.
> 
>         You can run out of space even if there  are  plenty  GB's
>         free  on  your  disks.   You  have  to create tablespaces
>         explicitly.
> 
>         If you've choosen inadequate extent size parameters,  you
>         end  up with high fragmented tables (slowing down) or get
>         stuck with running against maxextents, where only a reorg
>         (export/import) helps.

Also, Tom Lane pointed out to me that file system read-ahead does not
help if your table is spread around in tablespaces.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
The Hermit Hacker
Date:
On Thu, 15 Jun 2000, Bruce Momjian wrote:

> > "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > > Now I like neither relname nor oid because it's not sufficient 
> > > for my purpose.
> > 
> > We should probably not do much of anything with this issue until
> > we have a clearer understanding of what we want to do about
> > tablespaces and schemas.
> 
> Here is an analysis of our options:
> 
>                           Work required             Disadvantages
> ----------------------------------------------------------------------------
> 
> Keep current system       no work                   rename/create no rollback
> 
> relname/oid but           less work                 new pg_class column,
> no rename change                                    filename not accurate on
>                                                     rename
> 
> relname/oid with          more work                 complex code
> rename change during      
> vacuum
> 
> oid filename              less work, but            confusing to admins
>                           need admin tools          

My vote is with Tom on this one ... oid only ... the admin should be able
to do a quick SELECT on a table to find out the OID->table mapping, and I
believe its already been pointed out that you cant' just restore one file
anyway, so it kinda negates the "server isn't running problem" ...





Re: Big 7.1 open items

From
The Hermit Hacker
Date:
On Thu, 15 Jun 2000, Tom Lane wrote:

> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > Please add my opinion for naming rule.
> 
> > relname/unique_id but    need some work        new pg_class column,    
> > no relname change.    for unique-id generation    filename not relname
> 
> Why is a unique ID better than --- or even different from ---
> using the relation's OID?  It seems pointless to me...

just to open up a whole new bucket of worms here, but ... if we do use OID
(which up until this thought I endorse 100%) ... do we not run a risk if
we run out of OIDs?  As far as I know, those are still a finite resource,
no? 

or, do we just assume that by the time that comes, everyone will be pretty
much using 64bit machines? :)




Re: Big 7.1 open items

From
Tom Lane
Date:
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> ...  But I think it's critical to keep
>> the low-level file access protocol simple and reliable, which really
>> means minimizing the amount of information the backend needs to know 
>> to figure out which file to write a page in.  With something like the 
>> above you only need to know the tablespace name (or more likely OID), 
>> the relation OID (+name or not, depending on outcome of other 
>> argument), and the offset in the table.  No worse than now from the 
>> software's point of view.
>> Comments?

> I'm probably missing the context a bit, but imho we should try hard to
> stay away from symlinks as the general solution for anything.

Why?
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
The Hermit Hacker <scrappy@hub.org> writes:
> just to open up a whole new bucket of worms here, but ... if we do use OID
> (which up until this thought I endorse 100%) ... do we not run a risk if
> we run out of OIDs?  As far as I know, those are still a finite resource,
> no? 

They are, and there is some risk involved, but OID collisions in the
system tables will cause you just as much headache.  There's not only
the pg_class row to think of, but the pg_attribute rows, etc etc.

If you did have an OID collision with an existing table you'd have to
keep trying until you got a set of OID assignments with no conflicts.
(Now that we have unique indexes on the system tables, this should
work properly, ie, you will hear about it if you have a conflict.)
I don't think the physical table names make this noticeably worse.
Of course we'd better be careful to create table files with O_EXCL,
so as not to tromp on existing files, but we do that already IIRC.

> or, do we just assume that by the time that comes, everyone will be pretty
> much using 64bit machines? :)

I think we are not too far away from being able to offer 64-bit OID as
a compile-time option (on machines where there is a 64-bit integer type
that is).  It's just a matter of someone putting it at the head of their
todo list.

Bottom line is I'm not real worried about this issue.

But having said all that, I am coming round to agree with Hiroshi's idea
anyway.  See upcoming message.
        regards, tom lane


Re: Big 7.1 open items

From
Don Baccus
Date:
At 04:27 PM 6/16/00 +0000, Thomas Lockhart wrote:

>Sorry for being behind here, but to make sure I'm on the right page:
>o tablespaces decouple storage from logical tables
>o a database lives in a default tablespace, unless specified
>o by default, a table will live in the default tablespace
>o (eventually) a table can be split across tablespaces

Or tablespaces across filesystems/mountpoints whatever.

>Some thoughts:
>o the ability to split single tables across disks was essential for
>scalability when disks were small. But with RAID, NAS, etc etc isn't
>that a smaller issue now?

Yes for size issues, I should think, especially if you have the 
money for a large RAID subsystem.  But for throughput performance,
control over which spindles particularly busy tables and indices
go on would still seem to be pretty relevant, when they're being
updated a lot.  In order to minimize seek times.

I really can't say how important this is in reality.  Oracle-world
folks still talk about this kind of optimization being important,
but I'm not personally running any kind of database-backed website
that's busy enough or contains enough storage to worry about it.

>o "tablespaces" would implement our less-developed "with location"
>feature, right? Splitting databases, whole indices and whole tables
>across storage is the biggest win for this work since more users will
>use the feature.
>o location information needs to travel with individual tables anyway.



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
Don Baccus
Date:
At 11:46 AM 6/16/00 -0400, Tom Lane wrote:

>OK, to get back to the point here: so in Oracle, tables can't cross
>tablespace boundaries,

Right, the construct AFAIK is "create table/index foo on tablespace ..."

> but a tablespace itself could span multiple
>disks?

Right.

>Not sure if I like that better or worse than equating a tablespace
>with a directory (so, presumably, all the files within it live on
>one filesystem) and then trying to make tables able to span
>tablespaces.  We will need to do one or the other though, if we want
>to have any significant improvement over the current state of affairs
>for large tables.

Oracle's way does a reasonable job of isolating the datamodel
from the details of the physical layout.

Take the OpenACS web toolkit, for instance.  We could take
each module's tables and indices and assign them appropriately
to various dataspaces, then provide a separate .sql files with
only "create tablespace" statements in there.

By modifying that one central file, the toolkit installation
could be customized to run anything from a small site (one
disk with everything on it, ala my own personal webserver at
birdnotes.net) or a very large site with many spindles, with
various index and table structures spread out widely hither
and thither.

Given that the OpenACS datamodel is nearly 10K lines long (including
many comments, of course), being able to customize an installation
to such a degree by modifying a single file filled with "create
tablespaces" would be very attractive.

>One way is to play the flip-the-path-ordering game some more,
>and access multiple-segment tables with pathnames like this:
>
>    .../TABLESPACE/RELATION        -- first or only segment
>    .../TABLESPACE/N/RELATION    -- N'th extension segment
>
>This isn't any harder for md.c to deal with than what we do now,
>but by making the /N subdirectories be symlinks, the dbadmin could
>easily arrange for extension segments to go on different filesystems.

I personally dislike depending on symlinks to move stuff around.
Among other things, a pg_dump/restore (and presumably future 
backup tools?) can't recreate the disk layout automatically.

>We'd still want to create some tools to help the dbadmin with slinging
>all these symlinks around, of course.

OK, if symlinks are simply an implementation detail hidden from the
dbadmin, and if the physical structure is kept in the db so it can
be rebuilt if necessary automatically, then I don't mind symlinks.

> But I think it's critical to keep
>the low-level file access protocol simple and reliable, which really
>means minimizing the amount of information the backend needs to know to
>figure out which file to write a page in.  With something like the above
>you only need to know the tablespace name (or more likely OID), the
>relation OID (+name or not, depending on outcome of other argument),
>and the offset in the table.  No worse than now from the software's
>point of view.

Make the code that creates and otherwise manipulates tablespaces
do the work, while keeping the low-level file access protocol simple.

Yes, this approach sounds very good to me.



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Don Baccus <dhogaza@pacifier.com> writes:
>> This isn't any harder for md.c to deal with than what we do now,
>> but by making the /N subdirectories be symlinks, the dbadmin could
>> easily arrange for extension segments to go on different filesystems.

> I personally dislike depending on symlinks to move stuff around.
> Among other things, a pg_dump/restore (and presumably future 
> backup tools?) can't recreate the disk layout automatically.

Good point, we'd need some way of saving/restoring the tablespace
structures.

>> We'd still want to create some tools to help the dbadmin with slinging
>> all these symlinks around, of course.

> OK, if symlinks are simply an implementation detail hidden from the
> dbadmin, and if the physical structure is kept in the db so it can
> be rebuilt if necessary automatically, then I don't mind symlinks.

I'm not sure about keeping it in the db --- creates a bit of a
chicken-and-egg problem doesn't it?  Maybe there needs to be a
"system database" that has nailed-down pathnames (no tablespaces
for you baby) and contains the critical installation-wide tables
like pg_database, pg_user, pg_tablespace.  A restore would have
to restore these tables first anyway.

> Make the code that creates and otherwise manipulates tablespaces
> do the work, while keeping the low-level file access protocol simple.

Right, that's the bottom line for me.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> >Some thoughts:
> >o the ability to split single tables across disks was essential for
> >scalability when disks were small. But with RAID, NAS, etc etc isn't
> >that a smaller issue now?
> 
> Yes for size issues, I should think, especially if you have the 
> money for a large RAID subsystem.  But for throughput performance,
> control over which spindles particularly busy tables and indices
> go on would still seem to be pretty relevant, when they're being
> updated a lot.  In order to minimize seek times.
> 
> I really can't say how important this is in reality.  Oracle-world
> folks still talk about this kind of optimization being important,
> but I'm not personally running any kind of database-backed website
> that's busy enough or contains enough storage to worry about it.

It is important when you have a few big tables that must be fast.  One
objection I have always had to the HP logical volume manager is that it
is difficult to know what drives are being assigned to each logical
volume.

Seems if they don't have RAID, we should allow such drive partitioning.


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Don Baccus
Date:
At 03:00 PM 6/16/00 -0400, Tom Lane wrote:

>> OK, if symlinks are simply an implementation detail hidden from the
>> dbadmin, and if the physical structure is kept in the db so it can
>> be rebuilt if necessary automatically, then I don't mind symlinks.
>
>I'm not sure about keeping it in the db --- creates a bit of a
>chicken-and-egg problem doesn't it? 

Not if the tablespace creates preceeds the tables stored in them.

> Maybe there needs to be a
>"system database" that has nailed-down pathnames (no tablespaces
>for you baby) and contains the critical installation-wide tables
>like pg_database, pg_user, pg_tablespace.  A restore would have
>to restore these tables first anyway.

Oh, I see.  Yes, when I've looked into this and have thought about
it I've assumed that there would always be a known starting point
which would contain the installation-wide tables.

From a practical point of view, I don't think that's really a
problem.

I've not looked into how Oracle does this, I assume it builds 
a system tablespace on one of the initial mount points you give
it when you install the thing.  The paths to the mount points
are stored in specific files known to Oracle, I think.  It's 
been over a year (not long enough!) since I've set up Oracle...




- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
"Ross J. Reedstrom"
Date:
On Fri, Jun 16, 2000 at 04:27:22PM +0000, Thomas Lockhart wrote:
> > ...  But I think it's critical to keep
> > the low-level file access protocol simple and reliable, which really
> > means minimizing the amount of information the backend needs to know 
> > to figure out which file to write a page in.  With something like the 
> > above you only need to know the tablespace name (or more likely OID), 
> > the relation OID (+name or not, depending on outcome of other 
> > argument), and the offset in the table.  No worse than now from the 
> > software's point of view.
> > Comments?

I think the backend needs a per table token that indicates how
to get at the physical bits of the file. Whether that's a filename
alone, filename with path, oid, key to a smgr hash table or something
else, it's opaque above the smgr routines.

Hmm, now I'm thinking, since the tablespace discussion has been reopened,
the way to go about coding all this is to reactivate the smgr code: how
about I leave the existing md smgr as is, and clone it, call it md2 or
something, and start messing with adding features there?


> 
> I'm probably missing the context a bit, but imho we should try hard to
> stay away from symlinks as the general solution for anything.
> 
> Sorry for being behind here, but to make sure I'm on the right page:
> o tablespaces decouple storage from logical tables
> o a database lives in a default tablespace, unless specified
> o by default, a table will live in the default tablespace
> o (eventually) a table can be split across tablespaces
> 
> Some thoughts:
> o the ability to split single tables across disks was essential for
> scalability when disks were small. But with RAID, NAS, etc etc isn't
> that a smaller issue now?
> o "tablespaces" would implement our less-developed "with location"
> feature, right? Splitting databases, whole indices and whole tables
> across storage is the biggest win for this work since more users will
> use the feature.
> o location information needs to travel with individual tables anyway.

I was juist thinking that that discussion needed some summation.

Some links to historic discussion: 

This one is Vadim saying WAL will need oids names:
http://www.postgresql.org/mhonarc/pgsql-hackers/1999-11/msg00809.html

A longer discussion kicked off by Don Baccus:
http://www.postgresql.org/mhonarc/pgsql-hackers/2000-01/msg00510.html

Tom suggesting OIDs to allow rollback:
http://www.postgresql.org/mhonarc/pgsql-hackers/2000-03/msg00119.html


Martin Neumann posted an question on dataspaces:

(can't find it in the offical archives:  looks like March 2000, 10-29 is
missing. here's my copy: don't beat on it!  n particular, since I threw
it together for local access, it's one _big_ index page)

http://cooker.ir.rice.edu/postgresql/msg20257.html
(in that thread is a post where I mention blindwrites and getting rid
of GetRawDatabaseInfo)

Martin later posted an RFD on tablespaces:

http://cooker.ir.rice.edu/postgresql/msg20490.html

Here's Horák Daniel with a patch for discussion, implementing dataspaces
on a per database level:

http://cooker.ir.rice.edu/postgresql/msg20498.html

Ross
-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005


Re: Big 7.1 open items

From
Kaare Rasmussen
Date:
> (At least not when we're sitting atop a well-done filesystem,
> which may leave the Linux folk out in the cold ;-).)

Exactly what fs of Linux are you talking about? I believe that for a database
server, ReiserFS would be a natural choice.

-- 
Kaare Rasmussen            --Linux, spil,--        Tlf:        3816 2582
Kaki Data                tshirts, merchandize      Fax:        3816 2582
Howitzvej 75               �ben 14.00-18.00        Email: kar@webline.dk
2000 Frederiksberg        L�rdag 11.00-17.00       Web:      www.suse.dk


Re: Big 7.1 open items

From
"Ross J. Reedstrom"
Date:
On Thu, Jun 15, 2000 at 07:53:52PM -0400, Tom Lane wrote:
> "Ross J. Reedstrom" <reedstrm@rice.edu> writes:
> > On Thu, Jun 15, 2000 at 03:11:52AM -0400, Tom Lane wrote:
> >> "Ross J. Reedstrom" <reedstrm@rice.edu> writes:
> >>>> Any strong objections to the mixed relname_oid solution?
> >> 
> >> Yes!
> 
> > The plan here was to let VACUUM handle renaming the file, since it
> > will already have all the necessary locks. This shortens the window
> > of confusion.  ALTER TABLE RENAME doesn't happen that often, really - 
> > the relname is there just for human consumption, then.
> 
> Yeah, I've seen tons of discussion of how if we do this, that, and
> the other thing, and be prepared to fix up some other things in case
> of crash recovery, we can make it work with filename == relname + OID
> (where relname tracks logical name, at least at some remove).
> 
> Probably.  Assuming nobody forgets anything.

I agree, it seems a major undertaking, at first glance. And second. Even
third. Especially for someone who hasn't 'earned his spurs' yet. as
it were.

> I'm just trying to point out that that's a huge amount of pretty
> delicate mechanism.  The amount of work required to make it trustworthy
> looks to me to dwarf the admin tools that Bruce is complaining about.
> And we only have a few people competent to do the work.  (With all
> due respect, Ross, if you weren't already aware of the implications
> for mdblindwrt, I have to wonder what else you missed.)

Ah, you knew that comment would come back to haunt me (I have a
tendency to think out loud, even if checking and coming back latter
would be better;-) In fact, there's no problem, and never was, since the
buffer->blind.relname is filled in via RelationGetPhysicalRelationName,
just like every other path that requires direct file access. I just
didn't remember that I had in fact checked it (it's been a couple months,
and I just got back from vacation ;-)

Actually, Once I re-checked it, the code looked very familiar. I had
spent time looking at the blind write code in the context of getting
rid of the only non-startup use of GetRawDatabaseInfo.

As to missing things: I'm leaning heavily on Bruce's previous
work for temp tables, to seperate the two uses of relname, via the
RelationGetRelationName and RelationGetPhysicalRelationName. There are
102 uses of the first in the current code (many in elog messages), and
only 11 of the second. If I'd had to do the original work of finding
every use of relname, and catagorizing it, I agree I'm not (yet) up to
it, but I have more confidence in Bruce's  (already tested) work.

> 
> Filename == OID is so simple, reliable, and straightforward by
> comparison that I think the decision is a no-brainer.
> 

Perhaps. Changing the label of the file on disk still requires finding
all the code that assumes it knows what that name is, and changing it.
Same work.

> If we could afford to sink unlimited time into this one issue then
> it might make sense to do it the hard way, but we have enough
> important stuff on our TODO list to keep us all busy for years ---
> I cannot believe that it's an effective use of our time to do this.
> 

The joys of Open Development. You've spent a fair amount of time trying
to convince _me_ not to waste my time. Thanks, but I'm pretty bull headed
sometimes. Since I've already done something of the work, take a look
at what I've got, and then tell me I'm wasting my time, o.k.?

> 
> > Hmm, what's all this with functions in catalog.c that are only called by
> > smgr/md.c? seems to me that anything having to do with physical storage
> > (like the path!) belongs in the smgr abstraction.
> 
> Yeah, there's a bunch of stuff that should have been implemented by
> adding new smgr entry points, but wasn't.  It should be pushed down.
> (I can't resist pointing out that one of those things is physical
> relation rename, which will go away and not *need* to be pushed down
> if we do it the way I want.)
> 

Oh, I agree completely. In fact, As I said to Hiroshi last time this came
up, I think of the field in pg_class an an opaque token, to be filled in
by the smgr, and only used by code further up to hand back to the smgr
routines. Same should be true of the buffer->blind struct.

Ross
-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005



RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> JanWieck@t-online.de (Jan Wieck) writes:
> >     There are also disadvantages.
> 
> >         You can run out of space even if there  are  plenty  GB's
> >         free  on  your  disks.   You  have  to create tablespaces
> >         explicitly.
> 
> Not to mention the reverse: if I read this right, you have to suck
> up your GB's long in advance of actually needing them.  That's OK
> for a machine that's dedicated to Oracle ... not so OK for smaller
> installations, playpens, etc.
>

I've had an anxiety about the way like Oracle's preallocation.
It had not been easy for me to estimate the extent size in
Oracle.  Maybe it would lose the simplicity of environment
settings which is one of the biggest advantage of PostgreSQL.
It seems that we should also provide not_preallocated DATAFILE
when many_tables_in_a_file storage manager is introduced.

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp 


Re: Big 7.1 open items

From
Tom Lane
Date:
"Ross J. Reedstrom" <reedstrm@rice.edu> writes:
> I think the backend needs a per table token that indicates how
> to get at the physical bits of the file. Whether that's a filename
> alone, filename with path, oid, key to a smgr hash table or something
> else, it's opaque above the smgr routines.

Except to the commands that provide the user interface for tablespaces
and so forth.  And there aren't all that many places that deal with
physical filenames anyway.  It would be a good idea to try to be a
little stricter about this, but I'm not sure you can make the separation
a whole lot cleaner than it is now ... with the exception of the obvious
bogosities like "rename table" being done above the smgr level.  (But,
as I said, I want to see that code go away, not just get moved into
smgr...)

> Hmm, now I'm thinking, since the tablespace discussion has been reopened,
> the way to go about coding all this is to reactivate the smgr code: how
> about I leave the existing md smgr as is, and clone it, call it md2 or
> something, and start messing with adding features there?

Um, well, you can't have it both ways.  If you're going to change/fix
the assumptions of code above the smgr, then you've got to update md
at the same time to match your new definition of the smgr interface.
Won't do much good to have a playpen smgr if the "standard" one is
broken.

One thing I have been thinking would be a good idea is to take the
relcache out of the bufmgr/smgr interfaces.  The relcache is a
higher-level concept and ought not be known to bufmgr or smgr; they
ought to work with some low-level data structure or token for relations.
We might be able to eliminate the whole concept of "blind write" if we
do that.  There are other problems with the relcache dependency: entries
in relcache can get blown away at inopportune times due to shared cache
inval, and it doesn't provide a good home for tokens for multiple
"versions" of a relation if we go with the fill-a-new-physical-file
approach to CLUSTER and so on.

Hmm, if you replace relcache in the smgr interfaces with pointers to
an smgr-maintained data structure, that might be the same thing that
you are alluding to above about an smgr hash table.

One thing *not* to do is add yet a third layer of data structure on
top of the ones already maintained in fd.c and md.c.  Whatever extra
data might be needed here should be added to md.c's tables, I think,
and then the tokens used in the smgr interface would be pointers into
that table.
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> It seems that we should also provide not_preallocated DATAFILE
> when many_tables_in_a_file storage manager is introduced.

Several people in this thread have been talking like a
single-physical-file storage manager is in our future, but I can't
recall anyone saying that they were going to do such a thing or even
presenting reasons why it'd be a good idea.

Seems to me that physical file per relation is considerably better for
our purposes.  It's easier to figure out what's going on for admin and
debug work, it means less lock contention among different backends
appending concurrently to different relations, and it gives the OS a
better shot at doing effective read-ahead on sequential scans.

So why all the enthusiasm for multi-tables-per-file?
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > It seems that we should also provide not_preallocated DATAFILE
> > when many_tables_in_a_file storage manager is introduced.
> 
> Several people in this thread have been talking like a
> single-physical-file storage manager is in our future, but I can't
> recall anyone saying that they were going to do such a thing or even
> presenting reasons why it'd be a good idea.
> 
> Seems to me that physical file per relation is considerably better for
> our purposes.  It's easier to figure out what's going on for admin and
> debug work, it means less lock contention among different backends
> appending concurrently to different relations, and it gives the OS a
> better shot at doing effective read-ahead on sequential scans.
> 
> So why all the enthusiasm for multi-tables-per-file?

No idea.  I thought Vadim mentioned it, but I am not sure anymore.  I
certainly like our current system.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Chris Bitmead
Date:
> > So why all the enthusiasm for multi-tables-per-file?

It allows you to use raw partitions which stop the OS double buffering
and wasting half of memory, as well as removing the overhead of indirect
blocks in the file system.


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
> > 
> > So why all the enthusiasm for multi-tables-per-file?
> 
> No idea.  I thought Vadim mentioned it, but I am not sure anymore.  I
> certainly like our current system.
> 

Oops,I'm not so enthusiastic for multi_tables_per_file smgr.
I believe that Ross and I have taken a practical way that doesn't
break current file_per_table smgr.

However it seems very natural to take multi_tables_per_file
smgr into account when we consider TABLESPACE concept.
Because TABLESPACE is an encapsulation,it should have
a possibility to handle multi_tables_per_file smgr IMHO.

Regards. 

Hiroshi Inoue
Inoue@tpf.co.jp 


RE: Big 7.1 open items

From
Kaare Rasmussen
Date:
> Not to mention the reverse: if I read this right, you have to suck
> up your GB's long in advance of actually needing them.  That's OK
> for a machine that's dedicated to Oracle ... not so OK for smaller
> installations, playpens, etc.

To me it looks like a way to make Oracle work on VMS machines. This is the way
files are allocated on Digital hardware.

-- 
Kaare Rasmussen            --Linux, spil,--        Tlf:        3816 2582
Kaki Data                tshirts, merchandize      Fax:        3816 2582
Howitzvej 75               �ben 14.00-18.00        Email: kar@webline.dk
2000 Frederiksberg        L�rdag 11.00-17.00       Web:      www.suse.dk


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> However it seems very natural to take multi_tables_per_file
> smgr into account when we consider TABLESPACE concept.
> Because TABLESPACE is an encapsulation,it should have
> a possibility to handle multi_tables_per_file smgr IMHO.

OK, I see: you're just saying that the tablespace stuff should be
designed in such a way that it would work with a non-file-per-table
smgr.  Agreed, that'd be a good check of a clean design, and someday
we might need it...
        regards, tom lane


Re: Big 7.1 open items

From
JanWieck@t-online.de (Jan Wieck)
Date:
Thomas Lockhart wrote:
> > (At least not when we're sitting atop a well-done filesystem,
> > which may leave the Linux folk out in the cold ;-).)
>
> Those who live in HP houses should not throw stones :))
   Huh?  Up to HPUX-9 they used to have BSD-FFS - even if it was   a 4.2 BSD one - no?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #




Re: Big 7.1 open items

From
JanWieck@t-online.de (Jan Wieck)
Date:
Tom Lane wrote:
> JanWieck@t-online.de (Jan Wieck) writes:
> > Tom Lane wrote:
> >> It gets a little trickier if you want to be able to split
> >> multi-gig tables across several tablespaces, though, since
> >> you couldn't just append ".N" to the base table path in that
> >> scenario.
> >>
> >> I'd be interested to know what sort of facilities Oracle
> >> provides for managing huge tables...
>
> >     Oracle  tablespaces  are  a  collection of 1...n preallocated
> >     files.   Each  table  then  is  bound  to  a  tablespace  and
> >     allocates extents (chunks) from those files.
>
> OK, to get back to the point here: so in Oracle, tables can't cross
> tablespace boundaries, but a tablespace itself could span multiple
> disks?
   They can. The path in
       ALTER TABLESPACE <tsname> ADD DATAFILE ...
   can point to any location the db system has access to.

>
> Not sure if I like that better or worse than equating a tablespace
> with a directory (so, presumably, all the files within it live on
> one filesystem) and then trying to make tables able to span
> tablespaces.  We will need to do one or the other though, if we want
> to have any significant improvement over the current state of affairs
> for large tables.
>
> One way is to play the flip-the-path-ordering game some more,
> and access multiple-segment tables with pathnames like this:
>
>    .../TABLESPACE/RELATION       -- first or only segment
>    .../TABLESPACE/N/RELATION     -- N'th extension segment
>
> [...]
   In most cases all objects in one database are bound to one or   two tablespaces (data and indices). So you do  an
estimation  of  the  size  required, create the tablespaces (and probably   all their extension files), then create the
schema and  load   it.  The  only reason not to do so is if your DB exceeds some   size where you have to fear not
beeingable to finish  online   backups  before  getting into Online-Relolog stuck. Has to do   the the online backup
procedureof Oracle.
 

> This isn't any harder for md.c to deal with than what we do now,
> but by making the /N subdirectories be symlinks, the dbadmin could
> easily arrange for extension segments to go on different filesystems.
> Also, since /N subdirectory symlinks can be added as needed,
> expanding available space by attaching more disks isn't hard.
> (If the admin hasn't pre-made a /N symlink when it's needed,
> I'd envision the backend just automatically creating a plain
> subdirectory so that it can extend the table.)
   So the admin allways have to leave enough  freespace  in  the   default  location to keep the DB running until he
cantake it   offline, move the autocreated files and create the  symlinks.   What a pain for 24/7 systems.
 

> We'd still want to create some tools to help the dbadmin with slinging
> all these symlinks around, of course.  But I think it's critical to keep
> the low-level file access protocol simple and reliable, which really
> means minimizing the amount of information the backend needs to know to
> figure out which file to write a page in.  With something like the above
> you only need to know the tablespace name (or more likely OID), the
> relation OID (+name or not, depending on outcome of other argument),
> and the offset in the table.  No worse than now from the software's
> point of view.
   Exactly  the  "low-level  file  access"  protocol  is  highly   complicated in Postgres. Because nearly  every
object needs   his  own file, we need to deal with virtual file descriptors.   With an Oracle-like tablespace concept
anda fixed  limit  of   total   tablespace   files  (this  time  OS  or  installation   specific), we could keep them
allopen all the time.  IMHO  a   big win.
 


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #




Re: Big 7.1 open items

From
JanWieck@t-online.de (Jan Wieck)
Date:
Tom Lane wrote:
> JanWieck@t-online.de (Jan Wieck) writes:
> >     There are also disadvantages.
>
> >         You can run out of space even if there  are  plenty  GB's
> >         free  on  your  disks.   You  have  to create tablespaces
> >         explicitly.
>
> Not to mention the reverse: if I read this right, you have to suck
> up your GB's long in advance of actually needing them.  That's OK
> for a machine that's dedicated to Oracle ... not so OK for smaller
> installations, playpens, etc.
   Right,  the design is perfect for a few databases with a more   or less stable size and schema (slow to medium
growth). The   problem  is, that production databases tend to fall into that   behaviour and that might be  a  reason
for so  many  people   asking for Oracle compatibility - they want to do development   in the high  flexible  Postgres
environment, while  running   their production server under Oracle :-(.
 

> I'm not convinced that there's anything fundamentally wrong with
> doing storage allocation in Unix files the way we have been.
>
> (At least not when we're sitting atop a well-done filesystem,
> which may leave the Linux folk out in the cold ;-).)
   I'm  with  you on that, even if I'm one of the Linux loosers.   The only point that really strikes me is that in
our system   you  might  end  up with a corrupted file system because some   inode changes didn't make it to the disk
beforea crash. Even   if  using  fsync() instead of fdatasync() (what we cannot use   at all and that's a pain from the
performancePoV).   In  the   Oracle world, that could only happen during
 
       ALTER TABLESPACE <tsname> ADD DATAFILE ...
   which  is  a  fairly seldom command, issued usually by the DB   admin (at  least  it  requires  admin  privileges)
and thus   ensures  the "admin is there and already paying attention". A   little detail not to underestimate IMHO.
 


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #




Re: Big 7.1 open items

From
JanWieck@t-online.de (Jan Wieck)
Date:
Don Baccus wrote:
> At 11:46 AM 6/16/00 -0400, Tom Lane wrote:
>
> I personally dislike depending on symlinks to move stuff around.
> Among other things, a pg_dump/restore (and presumably future
> backup tools?) can't recreate the disk layout automatically.
>
   Most impact from this one, IMHO.
   Not  that  Oracle tools are able to do it either. But I think   it's more trivial to recreate a 30+ tablespace
layouton  the   disks   than   to   recreate   all  symlinks  for  a  20,000+   tables/indices database like an SAP R/3
one.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #




Re: Big 7.1 open items

From
JanWieck@t-online.de (Jan Wieck)
Date:
Bruce Momjian wrote:
> >     There are also disadvantages.
> >
> >         You can run out of space even if there  are  plenty  GB's
> >         free  on  your  disks.   You  have  to create tablespaces
> >         explicitly.
> >
> >         If you've choosen inadequate extent size parameters,  you
> >         end  up with high fragmented tables (slowing down) or get
> >         stuck with running against maxextents, where only a reorg
> >         (export/import) helps.
>
> Also, Tom Lane pointed out to me that file system read-ahead does not
> help if your table is spread around in tablespaces.
   Not  with our HEAP concept. With the Oracle EXTENT concept it   does pretty good, because they  have  different
block/extent  sizes.   Usually  an  extent spans multiple blocks, so in the   case of sequential reads they read each
extent of  probably   hundreds  of  K sequential. And in the case of indexed reads,   they know the extent and offset
ofthe tuple  inside  of  the   extent,  so they know the exact location of the record inside   the tablespace to read.
 
   The big problem we allways had (why we need TOAST at all)  is   that  the logical blocksize (extent size) of a table
isbound   to your physical blocksize used in the shared cache. This  is   fixed  so  deeply  in the heap storage
architecture,that I'm   scared about it.
 


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #




Re: Big 7.1 open items

From
Giles Lean
Date:
> Thomas Lockhart wrote:
> > > (At least not when we're sitting atop a well-done filesystem,
> > > which may leave the Linux folk out in the cold ;-).)
> >
> > Those who live in HP houses should not throw stones :))
> 
>     Huh?  Up to HPUX-9 they used to have BSD-FFS - even if it was
>     a 4.2 BSD one - no?

It's still there, along with VxFS from Veritas.

Ciao,

Giles


Re: Big 7.1 open items

From
Bruce Momjian
Date:
OK, I have thought about tablespaces, and here is my proposal.  Maybe
there will some good ideas in my design.

My feeling is that intelligent use of directories and symlinks can allow
PostgreSQL to handle tablespaces and allow administrators to use
symlinks outside of PostgreSQL and have PostgreSQL honor those changes
in a reload.

Seems we have three tablespace needs:
locate database in separate disklocate tables in separate directory/symlinklocate secondary extents on different
drives

If we have a new CREATE DATABASE LOCATION command, we can say:
CREATE DATABASE LOCATION dbloc IN '/var/private/pgsql';CREATE DATABASE newdb IN dbloc;

The first command makes sure /var/private/pgsql exists and is write-able
by postgres.  It then creates a dbloc directory and a symlink:
mkdir /var/private/pgsql/dblocln -s /var/private/pgsql/dbloc data/base/dbloc

The CREATE DATABASE command creates data/base/dbloc/newdb and creates
the database there.  We would have to store the dbloc location in
pg_database.

To handle placing tables, we can use:
CREATE LOCATION tabloc IN '/var/private/pgsql';CREATE TABLE newtab ... IN tabloc;

The first command makes sure /var/private/pgsql exists and is write-able
by postgres.  It then creates a directory tabloc in /var/private/pgsql,
and does a symlink:
ln -s /var/private/pgsql/tabloc data/base/dbloc/newdb/tabloc

and creates the table in there.  These location names have to be stored
in pg_class.

The difference betweeen CREATE LOCATION and CREATE DATABASE LOCATION is
that the first one puts it in the current database, while the latter
puts the symlinks in data/base.  

(Can we remove data/base and just make it data/?)

I would also allow a simpler CREATE LOCATION tabloc2 which just creates
a directory in the database directory.  These can be moved later using
symlinks.  Of course, CREATE DATABASE LOCATION too.

I haven't figured out extent locations yet.  One idea is to allow
administrators to create symlinks for tables >1 gig, and to not remove
the symlinks when a table shrinks.   Only remove the file pointed to by
the table, but leave the symlink there so if the table grows again, it
can use the symlink.  lstat() would allow this.

Now on to preserving this information.  My ideas is that PostgreSQL
should never remove a directory or symlink in the data/base directory. 
Those represent locations made by the administrator.  So, pg_dump with a
-l option can go through the db directory and output CREATE LOCATION
commands for every database, so when reloaded, the locations will be
preserved, assuming the symlinks point to still-valid directories.

What this does allow is someone to create locations during table
population, but to keep them all on the same drive.  If they later move
things around on the disk using cp and symlinks, this will be preserved
by pg_dump.

My problem with many of the tablespace systems is that it requires two
changes.  One in the file system using symlinks, and another in the
database to point to the new entries, or it does not preserve them
across backups.

If someone does want to remove a location, they would have to remove all
tables in the directory, and the base directory and symlink can be
removed with DROP LOCATION.

My solution basically stores locations for databases and tables in the
database, but does _not_ store information about what locations exist or
if they are symlinks.  However, it does allow for preserving of this
information in dumps.

I feel this solution is very flexible. 

Comments?

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
JanWieck@t-online.de (Jan Wieck) writes:
> Thomas Lockhart wrote:
>> Those who live in HP houses should not throw stones :))

>     Huh?  Up to HPUX-9 they used to have BSD-FFS - even if it was
>     a 4.2 BSD one - no?

Yeah, the standard HPUX filesystem is still BSD ... and it still runs
rings around Linux extfs2 in my experience.  (I've been informed that
Linux has better filesystems than extfs2, but that seems to be what
the average Linux user is running.)  I have a realtime data collection
program that usually wants to write several thousand small files during
shutdown.  The shutdown typically takes about 3 minutes on an HP 715/75,
upwards of 10 minutes on a Linux box with nominally-faster hardware.

BTW, HP is trying to sell people on using a new journaling filesystem
that they claim outperforms BSD, but my few experiments with it
haven't encouraged me to pursue it.
        regards, tom lane


Re: Re: Big 7.1 open items

From
Tatsuo Ishii
Date:
> Oh. I was recalling SQL_TEXT as being a "subset" character set which
> contains only the characters (more or less) that are required for
> implementing the SQL92 query language and standard features.
> 
> Are you seeing it as being a "superset" character set which can
> represent all other character sets??

Yes, it's my understanding from the section 19.3.1 of Date's book
(fourth edition). Please correct me if I am wrong.

> And, how would you suggest we start tracking this discussion in a design
> document? I could put something into the developer's guide, or we could
> have a plain-text FAQ, or ??
> 
> I'd propose that we start accumulating a feature list, perhaps ordering
> it into categories like
> 
> o required/suggested by SQL9x
> o required/suggested by experience in the real world
> o sure would be nice to have
> o really bad idea ;)

Sounds good. Could I put "CREATE CHARACTER SET" as the first item of
the list and start a discussion for that?

I have a feeling that you have an idea to treat user defined charset
as a PostgreSQL new data type. So probably "CREATE CHARACTER SET"
could be traslated to our "CREATE TYPE" by the parer, right?
--
Tatsuo Ishii


Re: Big 7.1 open items

From
Michael Reifenberger
Date:
On Sun, 18 Jun 2000, Jan Wieck wrote:
...
>         ALTER TABLESPACE <tsname> ADD DATAFILE ...
> 
>     which  is  a  fairly seldom command, issued usually by the DB
>     admin (at  least  it  requires  admin  privileges)  and  thus
>     ensures  the "admin is there and already paying attention". A
>     little detail not to underestimate IMHO.
...
Esp. in the R/3 area this will become no longer be true the more commonly
commands like "AUTOEXTEND" and "RESIZE" are used (automated at worst).

Bye!
----
Michael Reifenberger
^.*Plaut.*$, IT, R/3 Basis, GPS



Re: Big 7.1 open items

From
Bruce Momjian
Date:
> I haven't figured out extent locations yet.  One idea is to allow
> administrators to create symlinks for tables >1 gig, and to not remove
> the symlinks when a table shrinks.   Only remove the file pointed to by
> the table, but leave the symlink there so if the table grows again, it
> can use the symlink.  lstat() would allow this.

OK, I have an extent idea.  It is:
CREATE LOCATION tabloc IN '/var/private/pgsql' EXTENT2
'/usr/pg'.

This creates an /extents directory in the location, with extents/2
symlinked to /usr/pg:
data/base/mydb/tablocdata/base/mydb/tabloc/extents/2

When extending a table, it looks for an extents/2 directory and uses
that if it exists. Same for extents3.  We could even get fancy and
round-robin through all the extents directories, looping around to the
beginning when we run out of them.  That sounds nice.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > I haven't figured out extent locations yet.  One idea is to allow
> > administrators to create symlinks for tables >1 gig, and to not remove
> > the symlinks when a table shrinks.   Only remove the file pointed to by
> > the table, but leave the symlink there so if the table grows again, it
> > can use the symlink.  lstat() would allow this.
> 
> OK, I have an extent idea.  It is:
> 
>     CREATE LOCATION tabloc IN '/var/private/pgsql' EXTENT2
> '/usr/pg'.

Even better:
CREATE LOCATION tabloc IN '/var/private/pgsql'     EXTENT '/usr/pg', '/usr1/pg'

This will create extent/2 and extent/3, and the system can rotate
extents between the primary storage area, and 2 and 3.

Also, CREATE INDEX will need a location specification added.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> ...  We could even get fancy and
> round-robin through all the extents directories, looping around to the
> beginning when we run out of them.  That sounds nice.

That sounds horrible.  There's no way to tell which extent directory
extent N goes into except by scanning the location directory to find
out how many extent subdirectories there are (so that you can compute
N modulo number-of-directories).  Do you want to pay that price on every
file open?

Worse, what happens when you add another extent directory?  You can't
find your old extents anymore, that's what, because they're not in the
right place (N modulo number-of-directories just changed).  Since the
extents are presumably on different volumes, you're talking about
physical file moves to get them where they should be.  You probably
can't add a new extent without shutting down the entire database while
you reshuffle files --- at the very least you'd need to get exclusive
locks on all the tables in that tablespace.

Also, you'll get filename conflicts from multiple extents of a single
table appearing in one of the recycled extent dirs.  You could work
around it by using the non-modulo'd N as part of the final file name,
but that just adds more complexity and makes the filename-generation
machinery that much more closely tied to this specific way of doing
things.

The right way to do this is that extent N goes into extents subdirectory
N, period.  If there's no such subdirectory, create one on-the-fly as a
plain subdirectory of the location directory.  The dbadmin can easily
create secondary extent symlinks *in advance of their being needed*.
Reorganizing later is much more painful since it requires moving
physical files, but I think that'd be true no matter what.  At least
we should see to it that adding more space in advance of needing it is
painless.

It's possible to do it that way (auto-create extent subdir if needed)
without tying the md.c machinery real closely to a specific filename
creation procedure: it's just the same sort of thing as install programs
customarily do.  "If you fail to create a file, try creating its
ancestor directory."  We'd have to think about whether it'd be a good
idea to allow auto-creation of more than one level of directory; offhand
it seems that needing to make more than one level is probably a sign of
an erroneous path, not need for another extent subdirectory.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
If we eliminate the round-robin idea, what did people think of the rest
of the ideas?

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > ...  We could even get fancy and
> > round-robin through all the extents directories, looping around to the
> > beginning when we run out of them.  That sounds nice.
> 
> That sounds horrible.  There's no way to tell which extent directory
> extent N goes into except by scanning the location directory to find
> out how many extent subdirectories there are (so that you can compute
> N modulo number-of-directories).  Do you want to pay that price on every
> file open?


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > Not to mention the reverse: if I read this right, you have to suck
> > up your GB's long in advance of actually needing them.  That's OK
> > for a machine that's dedicated to Oracle ... not so OK for smaller
> > installations, playpens, etc.
> 
> To me it looks like a way to make Oracle work on VMS machines. This is the way
> files are allocated on Digital hardware.

Agreed.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Don Baccus
Date:
At 06:50 PM 6/18/00 -0400, Bruce Momjian wrote:
>If we eliminate the round-robin idea, what did people think of the rest
>of the ideas?

Why invent new syntax when "create tablespace" is something a lot
of folks will recognize?

And why not use "create table ... using ... "?  In other words, 
Oracle-compatible for this construct?  Sure, Postgres doesn't
have to follow Oraclisms but picking an existing contruct means
at least SOME folks can import a datamodel without having to
edit it.

Does your proposal break the smgr abstraction, i.e. does it
preclude later efforts to (say) implement an (optional) 
raw-device storage manager?




- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> At 06:50 PM 6/18/00 -0400, Bruce Momjian wrote:
> >If we eliminate the round-robin idea, what did people think of the rest
> >of the ideas?
> 
> Why invent new syntax when "create tablespace" is something a lot
> of folks will recognize?
> 
> And why not use "create table ... using ... "?  In other words, 
> Oracle-compatible for this construct?  Sure, Postgres doesn't
> have to follow Oraclisms but picking an existing contruct means
> at least SOME folks can import a datamodel without having to
> edit it.

Sure, use another syntax.  My idea was to use symlinks, and allow their
moving using symlinks and preserve them during dump.

> 
> Does your proposal break the smgr abstraction, i.e. does it
> preclude later efforts to (say) implement an (optional) 
> raw-device storage manager?

Seeing very few want that done, I don't see it as an issue at this
point.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Don Baccus
Date:
At 08:08 PM 6/18/00 -0400, Bruce Momjian wrote:

>> Does your proposal break the smgr abstraction, i.e. does it
>> preclude later efforts to (say) implement an (optional) 
>> raw-device storage manager?
>
>Seeing very few want that done, I don't see it as an issue at this
>point.

Sorry, I disagree.  There's excuse for breaking existing abstractions
unless there's a compelling reason to do so.

My question should make it clear I was using a raw-device storage
manager as an example.  There are other possbilities, like a 
many-tables-per-file storage manager.



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> JanWieck@t-online.de (Jan Wieck) writes:
> > Thomas Lockhart wrote:
> >> Those who live in HP houses should not throw stones :))
> 
> >     Huh?  Up to HPUX-9 they used to have BSD-FFS - even if it was
> >     a 4.2 BSD one - no?
> 
> Yeah, the standard HPUX filesystem is still BSD ... and it still runs
> rings around Linux extfs2 in my experience.  (I've been informed that
> Linux has better filesystems than extfs2, but that seems to be what
> the average Linux user is running.)  I have a realtime data collection
> program that usually wants to write several thousand small files during
> shutdown.  The shutdown typically takes about 3 minutes on an HP 715/75,
> upwards of 10 minutes on a Linux box with nominally-faster hardware.
> 
> BTW, HP is trying to sell people on using a new journaling filesystem
> that they claim outperforms BSD, but my few experiments with it
> haven't encouraged me to pursue it.

You should really try the BSD4.4 FFS with soft updates.  It re-orders
disk flushes to greatly improve performance.  It really is great.  


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
"Ross J. Reedstrom"
Date:
On Sun, Jun 18, 2000 at 05:12:22PM -0700, Don Baccus wrote:
> At 08:08 PM 6/18/00 -0400, Bruce Momjian wrote:
> 
> >> Does your proposal break the smgr abstraction, i.e. does it
> >> preclude later efforts to (say) implement an (optional) 
> >> raw-device storage manager?
> >
> >Seeing very few want that done, I don't see it as an issue at this
> >point.
> 
> Sorry, I disagree.  There's excuse for breaking existing abstractions
> unless there's a compelling reason to do so.
> 
> My question should make it clear I was using a raw-device storage
> manager as an example.  There are other possbilities, like a 
> many-tables-per-file storage manager.
> 

Don, I see Bruce's proposal as implementation details within the sotrage
manager. In fact, we should probably implement the tablespace commands
with an extention of the smgr api. One different smgr I've been thinking
a little about is the persistent RAM smgr: I've heard there's some
new technologies coming up that may make large amounts cheaper, soon.
And there's always PostgreSQL for PalmOS, right? (Hey, IBM's got a Pocket
DB2, why shouldn't we?)

Ross
-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005



Re: Big 7.1 open items

From
Bruce Momjian
Date:
> At 08:08 PM 6/18/00 -0400, Bruce Momjian wrote:
> 
> >> Does your proposal break the smgr abstraction, i.e. does it
> >> preclude later efforts to (say) implement an (optional) 
> >> raw-device storage manager?
> >
> >Seeing very few want that done, I don't see it as an issue at this
> >point.
> 
> Sorry, I disagree.  There's excuse for breaking existing abstractions
> unless there's a compelling reason to do so.
> 
> My question should make it clear I was using a raw-device storage
> manager as an example.  There are other possbilities, like a 
> many-tables-per-file storage manager.

I agree it is nice to keep things as abstract as possible.  I just don't
know if the abstraction will cause added complexity.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
My basic proposal is that we optionally allow symlinks when creating
tablespace directories, and that we interrogate those symlinks during a
dump so administrators can move tablespaces around without having to
modify environment variables or system tables.

I also suggested creating an extent directory to hold extents, like
extent/2 and extent/3.  This will allow administration for smaller sites
to be simpler.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Don Baccus
Date:
At 11:13 PM 6/18/00 -0400, Bruce Momjian wrote:
>My basic proposal is that we optionally allow symlinks when creating
>tablespace directories, and that we interrogate those symlinks during a
>dump so administrators can move tablespaces around without having to
>modify environment variables or system tables.

If they can move them around from within the db, they'll have no need to
move them around from outside the db. 

I don't quite understand your devotion to using filesystem commands
outside the database to do database administration.



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I also suggested creating an extent directory to hold extents, like
> extent/2 and extent/3.  This will allow administration for smaller sites
> to be simpler.

I don't see the value in creating an extra level of directory --- seems
that just adds one more Unix directory-lookup cycle to each file open,
without any apparent return.  What's wrong with extent directory names
like extent2, extent3, etc?

Obviously the extent dirnames must be chosen so they can't conflict
with table filenames, but that's easily done.  For example, if table
files are named like 'OID_xxx' then 'extentN' will never conflict.
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
Don Baccus <dhogaza@pacifier.com> writes:
> If they can move them around from within the db, they'll have no need to
> move them around from outside the db. 
> I don't quite understand your devotion to using filesystem commands
> outside the database to do database administration.

Being *able* to use filesystem commands to see/fix what's going on is a
good thing, particularly from a development/debugging standpoint.  But
I agree we want to have within-the-system admin commands to do the same
things.
        regards, tom lane


Re: Big 7.1 open items

From
Don Baccus
Date:
At 12:28 AM 6/19/00 -0400, Tom Lane wrote:

>Being *able* to use filesystem commands to see/fix what's going on is a
>good thing, particularly from a development/debugging standpoint. 

Of course it's a crutch for development, but outside of development
circles few users will know how to use the OS in regard to the
database.

Assuming PG takes off.  Of course, if it remains the realm of the
dedicated hard-core hacker, I'm wrong.  

I have nothing against preserving the ability to use filesystem
commands if there's no significant costs inherent with this approach.
I'd view the breaking of smgr abstraction as a significant cost (though
I agree with Ross that it Bruce's proposal shouldn't require that, I
asked my question to flush Bruce out, if you will, because he's 
devoted to a particular outside-the-db management model).

> But
>I agree we want to have within-the-system admin commands to do the same
>things.

MUST have, I should think.



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Don Baccus <dhogaza@pacifier.com> writes:
> > If they can move them around from within the db, they'll have no need to
> > move them around from outside the db. 
> > I don't quite understand your devotion to using filesystem commands
> > outside the database to do database administration.
> 
> Being *able* to use filesystem commands to see/fix what's going on is a
> good thing, particularly from a development/debugging standpoint.  But
> I agree we want to have within-the-system admin commands to do the same
> things.

Yes, I like to have db commands to do it.  I just like to allow things
outside too, if possible.  It also prevents things from getting out of
sync because the database doesn't need to store the symlink location.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Don Baccus <dhogaza@pacifier.com> writes:
> I'd view the breaking of smgr abstraction as a significant cost

Actually, the "smgr abstraction" has *been* broken for a long time,
due to sloppy implementation of features like relation rename.
But I agree we should try to re-establish a clean separation.

>> But
>> I agree we want to have within-the-system admin commands to do the same
>> things.

> MUST have, I should think.

No argument from this quarter.  It seems to me that once a PG
installation has been set up, it ought to be possible to do routine
admin tasks remotely --- and that means no direct access to the
server's filesystem.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I also suggested creating an extent directory to hold extents, like
> > extent/2 and extent/3.  This will allow administration for smaller sites
> > to be simpler.
> 
> I don't see the value in creating an extra level of directory --- seems
> that just adds one more Unix directory-lookup cycle to each file open,
> without any apparent return.  What's wrong with extent directory names
> like extent2, extent3, etc?
> 
> Obviously the extent dirnames must be chosen so they can't conflict
> with table filenames, but that's easily done.  For example, if table
> files are named like 'OID_xxx' then 'extentN' will never conflict.

We could call them extent.2, extent-2, or Extent-2.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> At 12:28 AM 6/19/00 -0400, Tom Lane wrote:
> 
> >Being *able* to use filesystem commands to see/fix what's going on is a
> >good thing, particularly from a development/debugging standpoint. 
> 
> Of course it's a crutch for development, but outside of development
> circles few users will know how to use the OS in regard to the
> database.
> 
> Assuming PG takes off.  Of course, if it remains the realm of the
> dedicated hard-core hacker, I'm wrong.  
> 
> I have nothing against preserving the ability to use filesystem
> commands if there's no significant costs inherent with this approach.
> I'd view the breaking of smgr abstraction as a significant cost (though
> I agree with Ross that it Bruce's proposal shouldn't require that, I
> asked my question to flush Bruce out, if you will, because he's 
> devoted to a particular outside-the-db management model).

The fact is that symlink information is already stored in the file
system.  If we store symlink information in the database too, there
exists the ability for the two to get out of sync.  My point is that I
think we can _not_ store symlink information in the database, and query
the file system using lstat when required.


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
> 
> The fact is that symlink information is already stored in the file
> system.  If we store symlink information in the database too, there
> exists the ability for the two to get out of sync.  My point is that I
> think we can _not_ store symlink information in the database, and query
> the file system using lstat when required.
>

Hmm,this seems pretty confusing to me.
I don't understand the necessity of symlink.
Directory tree,symlink,hard link ... are OS's standard.
But I don't think they are fit for dbms management.

PostgreSQL is a database system of cource. So
couldn't it handle more flexible structure than OS's
directory tree for itself ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp 



Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > -----Original Message-----
> > From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
> > 
> > The fact is that symlink information is already stored in the file
> > system.  If we store symlink information in the database too, there
> > exists the ability for the two to get out of sync.  My point is that I
> > think we can _not_ store symlink information in the database, and query
> > the file system using lstat when required.
> >
> 
> Hmm,this seems pretty confusing to me.
> I don't understand the necessity of symlink.
> Directory tree,symlink,hard link ... are OS's standard.
> But I don't think they are fit for dbms management.
> 
> PostgreSQL is a database system of cource. So
> couldn't it handle more flexible structure than OS's
> directory tree for itself ?

Yes, but is anyone suggesting a solution that does not work with
symlinks?  If not, why not do it that way?

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
>
> > > -----Original Message-----
> > > From: Bruce Momjian [mailto:pgman@candle.pha.pa.us]
> > >
> > > The fact is that symlink information is already stored in the file
> > > system.  If we store symlink information in the database too, there
> > > exists the ability for the two to get out of sync.  My point is that I
> > > think we can _not_ store symlink information in the database,
> and query
> > > the file system using lstat when required.
> > >
> > Hmm,this seems pretty confusing to me.
> > I don't understand the necessity of symlink.
> > Directory tree,symlink,hard link ... are OS's standard.
> > But I don't think they are fit for dbms management.
> >
> > PostgreSQL is a database system of cource. So
> > couldn't it handle more flexible structure than OS's
> > directory tree for itself ?
>
> Yes, but is anyone suggesting a solution that does not work with
> symlinks?  If not, why not do it that way?
>

Maybe other solutions have been proposed already because
there have been so many opinions and proposals.

I've felt TABLE(DATA)SPACE discussion has always been
divergent.  IMHO,one of the main cause is that various factors
have been discussed at once.  Shouldn't we make step by step
consensus in TABLE(DATA)SPACE discussion ?

IMHO,the first step is to decide the syntax of CREATE TABLE
command not to define TABLE(DATA)SPACE.

Comments ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp



Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > Yes, but is anyone suggesting a solution that does not work with
> > symlinks?  If not, why not do it that way?
> >
> 
> Maybe other solutions have been proposed already because
> there have been so many opinions and proposals.
> 
> I've felt TABLE(DATA)SPACE discussion has always been
> divergent.  IMHO,one of the main cause is that various factors
> have been discussed at once.  Shouldn't we make step by step
> consensus in TABLE(DATA)SPACE discussion ?
> 
> IMHO,the first step is to decide the syntax of CREATE TABLE
> command not to define TABLE(DATA)SPACE.
> 
> Comments ?

Agreed.  Seems we have several issues:
filename contentstablespace implementationtablespace directory layouttablespace commands and syntax

Filename syntax seems to have resolved to
tablespace/tablename_oid_version or something like that.  I think a
clean solution to keep symlink names in sync with rename is to use hard
links during rename, and during vacuum, if the link count is greater
than one, we can scan the directory and remove old files matching the
oid.

I hope we can implement tablespaces using symlinks that can be dump, but
the symlink location does not have to be stored in the database.

Seems we are going to use Extent-2/Extent-3 to store extents under each
tablespace.

It also seems we will be using the Oracle tablespace syntax where
appropriate.

Comments?
--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
"Philip J. Warner"
Date:
At 09:40 20/06/00 -0400, Bruce Momjian wrote:
>
> [lots of stuff about symlinks]
>

It just occurred to me that the symlinks concerns may be short-circuitable,
if the following are true:

1. most of the desirability is for external 'management' and debugging etc
on 'reasonably' static database designs.

2. metadata changes (specifically renaming tables) occur infrequently.

3. there is no reason why they are desirable *technically* within the
implementations being discussed.

If these are true, then why not create a utility (eg. pg_update_symlinks)
that creates the relevant symlinks. It does not matter if they are
outdated, from an integrity point of view, and for the most part they can
be automatically maintained. Internally, postgresql can totally ignore them.

Have I missed something?


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|                                |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> At 09:40 20/06/00 -0400, Bruce Momjian wrote:
> >
> > [lots of stuff about symlinks]
> >
> 
> It just occurred to me that the symlinks concerns may be short-circuitable,
> if the following are true:
> 
> 1. most of the desirability is for external 'management' and debugging etc
> on 'reasonably' static database designs.
> 
> 2. metadata changes (specifically renaming tables) occur infrequently.
> 
> 3. there is no reason why they are desirable *technically* within the
> implementations being discussed.
> 
> If these are true, then why not create a utility (eg. pg_update_symlinks)
> that creates the relevant symlinks. It does not matter if they are
> outdated, from an integrity point of view, and for the most part they can
> be automatically maintained. Internally, postgresql can totally ignore them.
> 
> Have I missed something?

I am a little confused.  Are you suggesting that the entire symlink
thing can be done outside the database?  Yes, that is true if we don't
store the symlink locations in the database.  Of course, the database
has to be down to do this.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Agreed.  Seems we have several issues:

>     filename contents
>     tablespace implementation
>     tablespace directory layout
>     tablespace commands and syntax

I think we've agreed that the filename must depend on tablespace,
file version, and file segment number in some fashion --- plus
the table name/OID of course.  Although there's no real consensus
about exactly how to construct the name, agreeing on the components
is still a positive step.

A couple of other areas of contention were:
revising smgr interface to be cleanerexactly what to store in pg_class

I don't think there's any quibble about the idea of cleaning up smgr,
but we don't have a complete proposal on the table yet either.

As for the pg_class issue, I still favor storing(a) OID of tablespace --- not for file access, but so that
associatedtablespace-table entry can be looked up    by tablespace management operations(b) pathname of file as a
columnof type "name", including           a %d to be replaced by segment #
 

I think Peter was holding out for storing purely numeric tablespace OID
and table version in pg_class and having a hardwired mapping to pathname
somewhere in smgr.  However, I think that doing it that way gains only
micro-efficiency compared to passing a "name" around, while using the
name approach buys us flexibility that's needed for at least some of
the variants under discussion.  Given that the exact filename contents
are still so contentious, I think it'd be a bad idea to pick an
implementation that doesn't allow some leeway as to what the filename
will be.  A name also has the advantage that it is a single item that
can be used to identify the table to smgr, which will help in cleaning
up the smgr interface.

As for tablespace layout/implementation, the only real proposal I've
heard is that there be a subdirectory of the database directory for each
tablespace, and that that have a subdirectory for each segment (extent)
of its tables --- where any of these subdirectories could be symlinks
off to a different filesystem.  Some unhappiness was raised about
depending on symlinks for this function, but I didn't hear one single
concrete reason not to do it, nor an alternative design.  Unless someone
comes up with a counterproposal, I think that that's what the actual
access mechanism will look like.  We still need to talk about what we
want to store in the SQL-level representation of a tablespace, and what
sort of tablespace management tools/commands are needed.  (Although
"try to make it look like Oracle" seems to be pretty much the consensus
for the command level, not all of us know exactly what that means...)

Comments?  Anything else that we do have consensus on?
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
"Philip J. Warner" <pjw@rhyme.com.au> writes:
> If these are true, then why not create a utility (eg. pg_update_symlinks)
> that creates the relevant symlinks. It does not matter if they are
> outdated, from an integrity point of view, and for the most part they can
> be automatically maintained. Internally, postgresql can totally ignore them.

What?

I think you are confusing a couple of different things.  IIRC, at one
time when we were just thinking about ALTER TABLE RENAME, there was
a suggestion that the "real" table files be named by table OID, and
that there be symlinks to those files named by logical table name as
a crutch (:-)) for admins who wanted to know which table file was which.
That could be handled as you've sketched above, but I think the whole
proposal has fallen by the wayside anyway.

The current discussion of symlinks is focusing on using directory
symlinks, not file symlinks, to represent/implement tablespace layout.
        regards, tom lane


Re: Big 7.1 open items

From
"Philip J. Warner"
Date:
At 10:35 20/06/00 -0400, Bruce Momjian wrote:
>> 
>> If these are true, then why not create a utility (eg. pg_update_symlinks)
>> that creates the relevant symlinks. It does not matter if they are
>> outdated, from an integrity point of view, and for the most part they can
>> be automatically maintained. Internally, postgresql can totally ignore
them.
>>
>I am a little confused.  Are you suggesting that the entire symlink
>thing can be done outside the database?  Yes, that is true if we don't
>store the symlink locations in the database.  Of course, the database
>has to be down to do this.

The idea was to have postgresql, internally, totally ignore symlinks - use
OID or whatever is technically best for file names. Then create a
utility/command to make human-centric symlinks in a known location. The
symlinks *could* be updated automatically by postgres, if possible, but
would never be used internally. Things like vacuum could report out of date
symlinks, and maybe fix them (but probably not).

It may sound crude, but the only reason for the symlinks is for humans to
'see what is going on', and in most cases they wont be very volatile.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|                                |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Re: Big 7.1 open items

From
"Philip J. Warner"
Date:
At 10:45 20/06/00 -0400, Tom Lane wrote:
>
>What?
>
...
>
>The current discussion of symlinks is focusing on using directory
>symlinks, not file symlinks, to represent/implement tablespace layout.
>

Ooops. I'll pull my head in again.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|                                |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Re: Re: Big 7.1 open items

From
Thomas Lockhart
Date:
> > Oh. I was recalling SQL_TEXT as being a "subset" character set which
> > contains only the characters (more or less) that are required for
> > implementing the SQL92 query language and standard features.
> > Are you seeing it as being a "superset" character set which can
> > represent all other character sets??
> Yes, it's my understanding from the section 19.3.1 of Date's book
> (fourth edition). Please correct me if I am wrong.

Yuck. That is what is says, all right :(

Date says that SQL_TEXT is required to have two things:
1) all characters used in the SQL language itself (which is what I
recalled)

2) Every other character from every character set in the installation.

afaict (2) pretty much kills extensibility if we interpret that
literally. I'd like to research it a bit more before we accept it as a
requirement.

> > I'd propose that we start accumulating a feature list, perhaps ordering
> > it into categories like
> >
> > o required/suggested by SQL9x
> > o required/suggested by experience in the real world
> > o sure would be nice to have
> > o really bad idea ;)
> 
> Sounds good. Could I put "CREATE CHARACTER SET" as the first item of
> the list and start a discussion for that?
> 
> I have a feeling that you have an idea to treat user defined charset
> as a PostgreSQL new data type. So probably "CREATE CHARACTER SET"
> could be traslated to our "CREATE TYPE" by the parer, right?

Yes. Though the SQL_TEXT issue may completely kill this. And lead to a
requirement that we have a full-unicode backend :((

I'm hoping that there is a less-intrusive way to do this. What do other
database systems have for this? I assume most do not have much...
                 - Thomas


Character sets (Re: Re: Big 7.1 open items)

From
Peter Eisentraut
Date:
Thomas Lockhart writes:

> One issue: I can see (or imagine ;) how we can use the Postgres type
> system to manage multiple character sets.

But how are you going to tell a genuine "type" from a character set? And
you might have to have three types for each charset. There'd be a lot of
redundancy and confusion regarding the input and output functions and
other pg_type attributes. No doubt there's something to be learned from
the type system, but character sets have different properties -- like
characters(!), collation rules, encoding "translations" and what not.
There is no doubt also need for different error handling. So I think that
just dumping every character set into pg_type is not a good idea. That's
almost equivalent to having separate types for char(6), char(7), etc.

Instead, I'd suggest that character sets become separate objects. A
character entity would carry around its character set in its header
somehow. Consider a string concatenation function, being invoked with two
arguments of the same exotic character set. Using the type system only
you'd have to either provide a function signature for all combinations of
characters sets or you'd have to cast them up to SQL_TEXT, concatenate
them and cast them back to the original charset. A smarter concatentation
function instead might notice that both arguments are of the same
character set and simply paste them together right there.


> But allowing arbitrary character sets in, say, table names forces us
> to cope with allowing a mix of character sets in a single column of a
> system table.

The priority is probably the data people store, not the way they get to
name their tables.

> Would it be acceptable to have a "default database character set"
> which is allowed to creep into the pg_xxx tables?

I think we could go with making all system table char columns Unicode, but
of course they are really of the "name" type, which is another issue
completely.


> We should itemize all of these issues so we can keep track of what is
> necessary, possible, and/or "easy".

Here are a couple of "items" I keep wondering about:

* To what extend would we be able to use the operating systems locale
facilities? Besides the fact that some systems are deficient or broken one
way or another, POSIX really doesn't provide much besides "given two
strings, which one is greater", and then only on a per-process basis.
We'd really need more that, see also LIKE indexing issues, and indexing in
general.

* Client support: A lot of language environments provide pretty smooth
Unicode support these days, e.g., Java, Perl 5.6, and I think that C99 has
also made some strides. So while "we can store stuff in any character set
you want" is great, it's really no good if it doesn't work transparently
with the client interfaces. At least something to keep in mind.


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Bruce Momjian writes:

> If we have a new CREATE DATABASE LOCATION command, we can say:
> 
>     CREATE DATABASE LOCATION dbloc IN '/var/private/pgsql';
>     CREATE DATABASE newdb IN dbloc;

We kind of have this already, with CREATE DATABASE foo WITH LOCATION =
'bar'; but of course with environment variable kludgery. But it's a start.

>     mkdir /var/private/pgsql/dbloc
>     ln -s /var/private/pgsql/dbloc data/base/dbloc

I think the problem with this was that you'd have to do an extra lookup
into, say, pg_location to resolve this. Some people are talking about
blind writes, this is not really blind.

>     CREATE LOCATION tabloc IN '/var/private/pgsql';
>     CREATE TABLE newtab ... IN tabloc;

Okay, so we'd have "table spaces" and "database spaces". Seems like one
"space" ought to be enough. I was thinking that the database "space" would
serve as a default "space" for tables created within it but you could
still create tables in other "spaces" than were the database really is. In
fact, the database wouldn't show up at all in the file names anymore,
which may or may not be a good thing.

I think Tom suggested something more or less like this:

$PGDATA/base/tablespace/segment/table

(leaving the details of "table" aside for now). pg_class would get a
column storing the table space somehow, say an oid reference to
pg_location. There would have to be a default tablespace that's created by
initdb and it's indicated by oid 0. So if you create a simple little table
"foo" it ends up in

$PGDATA/base/0/0/foo

That is pretty manageable. Now to create a table space you do

CREATE LOCATION "name" AT '/some/where';

which would make an entry in pg_location and, similar to how you
suggested, create a symlink from

$PGDATA/base/newoid -> /some/where

Then when you create a new table at that new location this gets simply
noted in pg_class with an oid reference, the rest works completely
transparently and no lookup outside of pg_class required. The system would
create the segment 0 subdirectory automatically.

When tables get segmented the system would simply create subdirectories 1,
2, 3, etc. as needed, just as it created the 0 as need, no extra code.

pg_dump doesn't need to use lstat or whatever at all because the locations
are catalogued. Administrators don't even need to know about the linking
business, they just make sure the target directory exists.

Two more items to ponder:

* per-location transaction logs

* pg_upgrade


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Re: Big 7.1 open items

From
Randall Parker
Date:
Thomas,

A few (hopefully relevant) comments regarding character sets, code pages, 
I18N, and all that:

1) I've seen databases (DB2 if memory serves) that allowed the client 
side to declare itself to the database back-end engine as being in a 
particular code page. For instance, one could have a CP850 Latin-1 client 
and an ISO 8859-1 database. The database engine did appropriate 
translations in both directions.  
2) Mixing code pages in a single column and then having the database 
engine support it is not trivial. Either each CHAR/VARCHAR would have to 
have some code page settable per row (eg either as a separate column or 
as something like mycolumnname.encoding).   Even if you could handle all that you'd still be faced with the issue 
is collating sequence. Each individual code page will have a collating 
sequence. But how do you collate across code pages? There'd be letters 
that were only in a single code page. Plus, it gets messy because with, 
for instance, a simple umlauted a that occurs in CP850, CP1252, and ISO 
8859-1 (and likely in other code pages as well). That letter is really 
the same letter in all those code pages and should treated as such when 
sorting. 

3) I think it is more important for a database to support lots of 
languages in the stored data than in the field names and table names. If 
a programmer has to deal with A-Za-z for naming identifiers and that 
perseon is Korean or Japanese then that is certain is an imposition on 
them. But its a far far bigger imposition if that programmer can't build 
a database that will store the letters of his national language and sort 
and index and search them in convenient ways. 

4) The real solution to the multiple code page dilemma is Unicode.   Yes, its more space. But the can of worms of
dealingwith multiple 
 
code pages in a column is really no fun and the result is not great. 
BTDTHTTS.

5) The problem with enforcing   I've built a database in DB2 where particular columns in it contained 
data from many different code pages (each row had a code page field as 
well as a text field). For some applications that is okay if that field 
is not going to be part of an index.   However, if a database is going to be defined as being in a particular 
code page, and if the database engine is going to reject characters that 
are not recognized as part of that code page then you can't play the sort 
of game I just described _unless_ there is a different datatype that is 
similar to CHAR/VARCHAR but for which the RDBMS does not enforce code 
page legality on each character. Otherwise you choose some code page for 
a column, you go merrily stuffing in all sorts of rows in all sorts of 
code pages, and then along come some character that is of a value that is 
not a value for some other character in the code page that the RDBMS 
thinks it is. 

Anyway, I've done lots of I18N database stuff and hopefully a few of my 
comments will be useful to the assembled brethren <g>.

In news:<3948E4D7.A3B722E9@alumni.caltech.edu>, 
lockhart@alumni.caltech.edu says...
> One issue: I can see (or imagine ;) how we can use the Postgres type
> system to manage multiple character sets. But allowing arbitrary
> character sets in, say, table names forces us to cope with allowing a
> mix of character sets in a single column of a system table. afaik this
> general capability is not mandated by SQL9x (the SQL_TEXT character set
> is used for all system resources??). Would it be acceptable to have a
> "default database character set" which is allowed to creep into the
> pg_xxx tables? Even that seems to be a difficult thing to accomplish at
> the moment (we'd need to get some of the text manipulation functions
> from the catalogs, not from hardcoded references as we do now).
> 


Re: Big 7.1 open items

From
Randall Parker
Date:
[This followup was posted to comp.databases.postgresql.hackers and a copy 
was sent to the cited author.]

A few thoughts:

1) There may be reasons why someone might not want to use RAID.   For instance, suppose one wants to put different
tableson different 
 
drives so that the seeks for one table doesn't move the drive heads away 
from the disk area for another table.  Also, suppose someone wants to use a particular drive for a particular 
purpose (eg certain indexes) because it is faster at seeking vs another 
drive that is faster at sustained transfer rates.  Also, someone may want to span a drive across multiple SCSI 
controllers. Most RAID arrays I'm aware of are per SCSI controller.   I think it is fair to say that there will always
beinstances where 
 
people want to have more control over where stuff goes because they are 
willing to put the effort into more subtle tuning games. Well, there 
ought to be a way.

2) Some OSs do not support symlinks. The ability to list a bunch of 
devices for where things will go would be of value.  Also, if you aren't putting your data on a real file system (say
on
 
raw partitions instead) you are going to need a way to specify that 
anyway.

In news:<394A556A.4EAC8B9A@alumni.caltech.edu>, 
lockhart@alumni.caltech.edu says...
> o the ability to split single tables across disks was essential for
> scalability when disks were small. But with RAID, NAS, etc etc isn't
> that a smaller issue now?
> o "tablespaces" would implement our less-developed "with location"
> feature, right? Splitting databases, whole indices and whole tables
> across storage is the biggest win for this work since more users will
> use the feature.
> o location information needs to travel with individual tables anyway.




Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> Bruce Momjian writes:
> 
> > If we have a new CREATE DATABASE LOCATION command, we can say:
> > 
> >     CREATE DATABASE LOCATION dbloc IN '/var/private/pgsql';
> >     CREATE DATABASE newdb IN dbloc;
> 
> We kind of have this already, with CREATE DATABASE foo WITH LOCATION =
> 'bar'; but of course with environment variable kludgery. But it's a start.

Yes, I didn't like the environment variable stuff.  In fact, I would
like to not mention the symlink location anywhere in the database, so it
can be changed without changing it in the database.

> 
> >     mkdir /var/private/pgsql/dbloc
> >     ln -s /var/private/pgsql/dbloc data/base/dbloc
> 
> I think the problem with this was that you'd have to do an extra lookup
> into, say, pg_location to resolve this. Some people are talking about
> blind writes, this is not really blind.

I was think of storing the relfilename as dbloc/mytab32332.


> 
> >     CREATE LOCATION tabloc IN '/var/private/pgsql';
> >     CREATE TABLE newtab ... IN tabloc;
> 
> Okay, so we'd have "table spaces" and "database spaces". Seems like one
> "space" ought to be enough. I was thinking that the database "space" would
> serve as a default "space" for tables created within it but you could
> still create tables in other "spaces" than were the database really is. In
> fact, the database wouldn't show up at all in the file names anymore,
> which may or may not be a good thing.
> 
> I think Tom suggested something more or less like this:
> 
> $PGDATA/base/tablespace/segment/table

So you mix tables from different database in the same tablespace?  Seems
better to keep them in separate directories for efficiency and clarity.

We could use tablespace/dbname/table so that a tablespace would have
a directory for each database that uses the tablespace.
> 
> (leaving the details of "table" aside for now). pg_class would get a
> column storing the table space somehow, say an oid reference to
> pg_location. There would have to be a default tablespace that's created by
> initdb and it's indicated by oid 0. So if you create a simple little table
> "foo" it ends up in
> 
> $PGDATA/base/0/0/foo
> 

Seems better to use the top directory for 0, and have extents in
subdirectories like Extent-2, etc.  Easier for administrators and new
people.

However, one problem is that tables created in a database without a
location are put under pgsql directory.  You would have to symlink the
actual database directory.  Maybe that is why I had separate database
locations.  I realize that is bad.

> That is pretty manageable. Now to create a table space you do
> 
> CREATE LOCATION "name" AT '/some/where';
> 
> which would make an entry in pg_location and, similar to how you
> suggested, create a symlink from
> 
> $PGDATA/base/newoid -> /some/where
> 
> Then when you create a new table at that new location this gets simply
> noted in pg_class with an oid reference, the rest works completely
> transparently and no lookup outside of pg_class required. The system would
> create the segment 0 subdirectory automatically.

> 
> When tables get segmented the system would simply create subdirectories 1,
> 2, 3, etc. as needed, just as it created the 0 as need, no extra code.
> 
> pg_dump doesn't need to use lstat or whatever at all because the locations
> are catalogued. Administrators don't even need to know about the linking
> business, they just make sure the target directory exists.

What I was suggesting is not to catalog the symlink locations, but to
use lstat when dumping, so that admins can move files around using
symlinks and not have to udpate the database.


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Agreed.  Seems we have several issues:
> 
> >     filename contents
> >     tablespace implementation
> >     tablespace directory layout
> >     tablespace commands and syntax
>

[snip]
> 
> Comments?  Anything else that we do have consensus on?
>

Before the details of tablespace implementation,

1) How to change(extend) the syntax of CREATE TABLE   We only add table(data)space name with some   keyword ? i.e Do we
considertablespace as an  abstraction ? 
 

To confirm our mutual understanding.

2) Is tablespace defined per PostgreSQL's database ?
3) Is default tablespace defined per database/user or    for all ?

AFAIK in Oracle,2) global, 3) per user. 

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp 


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Peter Eisentraut
>
> Bruce Momjian writes:
>
> > If we have a new CREATE DATABASE LOCATION command, we can say:
> >
> >     CREATE DATABASE LOCATION dbloc IN '/var/private/pgsql';
> >     CREATE DATABASE newdb IN dbloc;
>
> We kind of have this already, with CREATE DATABASE foo WITH LOCATION =
> 'bar'; but of course with environment variable kludgery. But it's a start.
>
> >     mkdir /var/private/pgsql/dbloc
> >     ln -s /var/private/pgsql/dbloc data/base/dbloc
>
> I think the problem with this was that you'd have to do an extra lookup
> into, say, pg_location to resolve this. Some people are talking about
> blind writes, this is not really blind.
>
> >     CREATE LOCATION tabloc IN '/var/private/pgsql';
> >     CREATE TABLE newtab ... IN tabloc;
>
> Okay, so we'd have "table spaces" and "database spaces". Seems like one
> "space" ought to be enough.

Does your "database space" correspond to current PostgreSQL's database ?
And is it different from SCHEMA ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp



RE: Big 7.1 open items

From
"Philip J. Warner"
Date:
At 05:59 21/06/00 +0900, Hiroshi Inoue wrote:
>
>Before the details of tablespace implementation,
>
>1) How to change(extend) the syntax of CREATE TABLE
>    We only add table(data)space name with some
>    keyword ? i.e Do we consider tablespace as an
>   abstraction ? 
>

It may be worth considering leaving the CREATE TABLE statement alone.
Dec/RDB uses a new statement entirely to define where a table goes. It's
actually a *very* complex statement, but the key syntax is:

CREATE STORAGE MAP <map-name> FOR <table-name>   [PLACEMENT VIA INDEX <index-name>]   STORE [COLUMNS ([col-name,])]
[IN<area-name>    | RANDOMLY ACROSS <area-list>]
 
;

where <area-name> is the name of a Dec/RDB STORAGE AREA, which is basically
a file that contains one or more tables/indices etc. There are options to
specify area choice by column value, fullness, how to store BLOBs etc etc.

I realize that this is way too complex for a first pass, but it gives an
idea of where you *might* want to go, and hence, possibly, a reason for
starting out with something like:

CREATE STORAGE MAP <map-name> for <table-name> STORE IN <area-name>;


P.S. I really hope this is more cogent than my last message.

----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|                                |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Re: Big 7.1 open items

From
Chris Bitmead
Date:
Tom Lane wrote:
> Some unhappiness was raised about
> depending on symlinks for this function, but I didn't hear one single
> concrete reason not to do it, nor an alternative design. 

Are symlinks portable?


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Tom Lane wrote:
> > Some unhappiness was raised about
> > depending on symlinks for this function, but I didn't hear one single
> > concrete reason not to do it, nor an alternative design. 
> 
> Are symlinks portable?

Sure, and if the system loading it can not create the required symlinks
because the directories don't exist, it can just skip the symlink step.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> > -----Original Message-----
> > From: Peter Eisentraut
> >
> > Bruce Momjian writes:
> >
> > > If we have a new CREATE DATABASE LOCATION command, we can say:
> > >
> > >     CREATE DATABASE LOCATION dbloc IN '/var/private/pgsql';
> > >     CREATE DATABASE newdb IN dbloc;
> >
> > We kind of have this already, with CREATE DATABASE foo WITH LOCATION =
> > 'bar'; but of course with environment variable kludgery. But it's a start.
> >
> > >     mkdir /var/private/pgsql/dbloc
> > >     ln -s /var/private/pgsql/dbloc data/base/dbloc
> >
> > I think the problem with this was that you'd have to do an extra lookup
> > into, say, pg_location to resolve this. Some people are talking about
> > blind writes, this is not really blind.
> >
> > >     CREATE LOCATION tabloc IN '/var/private/pgsql';
> > >     CREATE TABLE newtab ... IN tabloc;
> >
> > Okay, so we'd have "table spaces" and "database spaces". Seems like one
> > "space" ought to be enough.
> 
> Does your "database space" correspond to current PostgreSQL's database ?
> And is it different from SCHEMA ?

OK, seems I have things a little confused.  My whole idea of database
locations vs. normal locations is flawed.  Here is my new proposal.

First, I believe there should be locations define per database, not
global locations.

I recommend 
CREATE TABLESPACE tabloc USING '/var/private/pgsql';CREATE TABLE newtab ... IN tabloc;

and this does:
mkdir /var/private/pgsql/dbnamemkdir /var/private/pgsql/dbname/tabloc    ln -s /var/private/pgsql/dbname/tabloc
data/base/tabloc

I recommend making a dbname in each directory, then putting the
location inside there.

This allows the same directory to be used for tablespaces by several
databases, and allows databases created in locations without making
special per-database locations.

I can give a more specific proposal if people wish.

Comments?

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I recommend making a dbname in each directory, then putting the
> location inside there.

This still seems backwards to me.  Why is it better than tablespace
directory inside database directory?

One significant problem with it is that there's no longer (AFAICS)
a "default" per-database directory that corresponds to the current
working directory of backends running in that database.  Thus,
for example, it's not immediately clear where temporary files and
backend core-dump files will end up.  Also, you've just added an
essential extra level (if not two) to the pathnames that backends will
use to address files.

There is a great deal to be said for..../database/tablespace/filename
where .../database/ is the working directory of a backend running in
that database, so that the relative pathname used by that backend to
get to a table is just tablespace/filename.  I fail to see any advantage
in reversing the pathname order.  If you see one, enlighten me.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I recommend making a dbname in each directory, then putting the
> > location inside there.
> 
> This still seems backwards to me.  Why is it better than tablespace
> directory inside database directory?

Yes, that is what I want too.

> 
> One significant problem with it is that there's no longer (AFAICS)
> a "default" per-database directory that corresponds to the current
> working directory of backends running in that database.  Thus,
> for example, it's not immediately clear where temporary files and
> backend core-dump files will end up.  Also, you've just added an
> essential extra level (if not two) to the pathnames that backends will
> use to address files.
> 
> There is a great deal to be said for
>     ..../database/tablespace/filename
> where .../database/ is the working directory of a backend running in
> that database, so that the relative pathname used by that backend to
> get to a table is just tablespace/filename.  I fail to see any advantage
> in reversing the pathname order.  If you see one, enlighten me.

Yes, agreed.  I was thinking this:
CREATE TABLESPACE loc USING '/var/pgsql'

does:
ln -s /var/pgsql/dbname/loc data/base/dbname/loc 

In this way, the database has a view of its main directory, plus a /loc
subdirectory for the tablespace.  In the other location, we have
/var/pgsql/dbname/loc because this allows different databases to use:
CREATE TABLESPACE loc USING '/var/pgsql'

and they do not collide with each other in /var/pgsql.  It puts /loc
inside the dbname that created it.  It also allows:
CREATE DATABASE loc IN '/var/pgsql'

to work because this does:
ln -s /var/pgsql/dbname data/base/dbname

Seems we should create the dbname and loc directories for the users
automatically in the synlink target to keep things clean.  It prevents
them from accidentally having two databases point to the same directory.

Comments?

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Chris Bitmead
Date:
Bruce Momjian wrote:
> 
> > Tom Lane wrote:
> > > Some unhappiness was raised about
> > > depending on symlinks for this function, but I didn't hear one single
> > > concrete reason not to do it, nor an alternative design.
> >
> > Are symlinks portable?
> 
> Sure, and if the system loading it can not create the required symlinks
> because the directories don't exist, it can just skip the symlink step.

What I meant is, would you still be able to create tablespaces on
systems without symlinks? That would seem to be a desirable feature.


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I recommend making a dbname in each directory, then putting the
> > location inside there.
> 
> This still seems backwards to me.  Why is it better than tablespace
> directory inside database directory?
> 
> One significant problem with it is that there's no longer (AFAICS)
> a "default" per-database directory that corresponds to the current
> working directory of backends running in that database.  Thus,
> for example, it's not immediately clear where temporary files and
> backend core-dump files will end up.  Also, you've just added an
> essential extra level (if not two) to the pathnames that backends will
> use to address files.
> 
> There is a great deal to be said for
>     ..../database/tablespace/filename

OK,I seem to have gotten the answer for the question  Is tablespace defined per PostgreSQL's database ?

You and Bruce  1) tablespace is per database
Peter seems to have the following idea(?? not sure)  2) database = tablespace
My opinion  3) database and tablespace are relatively irrelevant.      I assume PostgreSQL's database would correspond
    to the concept of SCHEMA.
 

It seems we are different from the first.
Shoudln't we reach an agreement on it in the first place ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp



Re: Big 7.1 open items

From
Tom Lane
Date:
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au> writes:
> What I meant is, would you still be able to create tablespaces on
> systems without symlinks? That would seem to be a desirable feature.

All else being equal, it'd be nice.  Since all else is not equal,
exactly how much sweat are we willing to expend on supporting that
feature on such systems --- to the exclusion of other features we
might expend the same sweat on, with more widely useful results?

Bear in mind that everything will still *work* just fine on such a
platform, you just don't have a way to spread the database across
multiple filesystems.  That's only an issue if the platform has a
fairly Unixy notion of filesystems ... but no symlinks.

A few messages back someone was opining that we were wasting our time
thinking about tablespaces at all, because any modern platform can
create disk-spanning filesystems for itself, so applications don't have
to worry.  I don't buy that argument in general, but I'm quite willing
to quote it for the *very* few systems that are Unixy enough to run
Postgres in the first place, but not quite Unixy enough to have
symlinks.

You gotta draw the line somewhere at what you will support, and
this particular line seems to me to be entirely reasonable and
justifiable.  YMMV...
        regards, tom lane


Re: Big 7.1 open items

From
Thomas Lockhart
Date:
> Yes, I didn't like the environment variable stuff.  In fact, I would
> like to not mention the symlink location anywhere in the database, so 
> it can be changed without changing it in the database.

Well, as y'all have noticed, I think there are strong reasons to use
environment variables to manage locations, and that symlinks are a
potential portability and robustness problem.

An additional point which has relevance to this whole discussion:

In the future we may allow system resource such as tables to carry names
which use multi-byte encodings. afaik these encodings are not allowed to
be used for physical file names, and even if they were the utility of
using standard operating system utilities like ls goes way down.

istm that from a portability and evolutionary standpoint OID-only file
names (or at least file names *not* based on relation/class names) is a
requirement.

Comments?
                  - Thomas


Re: Big 7.1 open items

From
Don Baccus
Date:
At 12:27 PM 6/21/00 +1000, Chris Bitmead wrote:
>Tom Lane wrote:
>> Some unhappiness was raised about
>> depending on symlinks for this function, but I didn't hear one single
>> concrete reason not to do it, nor an alternative design. 
>
>Are symlinks portable?

In today's world?  Yeah, I think so.

My only unhappiness has hinged around the possibility that a new
storage scheme might temp folks to toss aside the sgmr abstraction,
or weaken it.

It doesn't appear that this will happen. 

Given an adequate sgmr abstraction, it doesn't really matter what
low-level model is adopted in some sense (i.e. other models might
become available, the implemented model might get replaced, etc -
without breaking backends).

Obviously we'll all be using the default model for some time, maybe
forever, but if mistakes are made maintaining the smgr abstraction
means that replacements are possible.  Or kinky substitutes like
working with DAFS.



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


RE: Big 7.1 open items

From
Don Baccus
Date:
At 11:22 AM 6/21/00 +1000, Philip J. Warner wrote:

>It may be worth considering leaving the CREATE TABLE statement alone.
>Dec/RDB uses a new statement entirely to define where a table goes...

It's worth considering, but on the other hand Oracle users greatly
outnumber Compaq/RDB users these days...

If there's no SQL92 guidance for implementing a feature, I'm pretty much in
favor of tracking Oracle, whose SQL dialect is rapidly becoming a
de-facto standard. 

I'm not saying I like the fact, Oracle's a pain in the ass.  But when
adopting existing syntax, might as well adopt that of the crushing
borg.



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
>> There is a great deal to be said for
>> ..../database/tablespace/filename

> OK,I seem to have gotten the answer for the question
>    Is tablespace defined per PostgreSQL's database ?

Not necessarily --- the tablespace subdirectories could be symlinks
pointing to the same place (assuming you use OIDs or something to keep
the table filenames unique even across databases).  This is just an
implementation mechanism; it doesn't foreclose the policy decision
whether tablespaces are database-local or installation-wide.

(OTOH, pathnames like tablespace/database would pretty much force
tablespaces to be installation-wide whether you wanted it that way
or not.)

> My opinion
>    3) database and tablespace are relatively irrelevant.
>        I assume PostgreSQL's database would correspond 
>        to the concept of SCHEMA.

My inclindation is that tablespaces should be installation-wide, but
I'm not completely sold on it.  In any case I could see wanting a
permissions mechanism that would only allow some databases to have
tables in a particular tablespace.

We do need to think more about how traditional Postgres databases
fit together with SCHEMA.  Maybe we wouldn't even need multiple
databases per installation if we had SCHEMA done right.
        regards, tom lane


Re: Big 7.1 open items

From
"Ross J. Reedstrom"
Date:
On Wed, Jun 21, 2000 at 01:23:57AM -0400, Tom Lane wrote:
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> 
> > My opinion
> >    3) database and tablespace are relatively irrelevant.
> >        I assume PostgreSQL's database would correspond 
> >        to the concept of SCHEMA.
> 
> My inclindation is that tablespaces should be installation-wide, but
> I'm not completely sold on it.  In any case I could see wanting a
> permissions mechanism that would only allow some databases to have
> tables in a particular tablespace.
> 
> We do need to think more about how traditional Postgres databases
> fit together with SCHEMA.  Maybe we wouldn't even need multiple
> databases per installation if we had SCHEMA done right.
> 

The important point I think is that tablespaces are about physical
storage/namespace, and SCHEMA are about logical namespace: it would make
sense for tables from multiple schema to live in the same tablespace,
as well as tables from one schema to be stored in multiple tablespaces.

Ross
-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005


Re: Character sets (Re: Re: Big 7.1 open items)

From
Tatsuo Ishii
Date:
> But how are you going to tell a genuine "type" from a character set? And
> you might have to have three types for each charset. There'd be a lot of
> redundancy and confusion regarding the input and output functions and
> other pg_type attributes. No doubt there's something to be learned from
> the type system, but character sets have different properties -- like
> characters(!), collation rules, encoding "translations" and what not.
> There is no doubt also need for different error handling. So I think that
> just dumping every character set into pg_type is not a good idea. That's
> almost equivalent to having separate types for char(6), char(7), etc.
> 
> Instead, I'd suggest that character sets become separate objects. A
> character entity would carry around its character set in its header
> somehow. Consider a string concatenation function, being invoked with two
> arguments of the same exotic character set. Using the type system only
> you'd have to either provide a function signature for all combinations of
> characters sets or you'd have to cast them up to SQL_TEXT, concatenate
> them and cast them back to the original charset. A smarter concatentation
> function instead might notice that both arguments are of the same
> character set and simply paste them together right there.

Intersting idea. But what about collations? SQL allows to assign a
collation different from the default one to a character set on the
fly. Should we make collations as separate obejcts as well?

> Here are a couple of "items" I keep wondering about:
> 
> * To what extend would we be able to use the operating systems locale
> facilities? Besides the fact that some systems are deficient or broken one
> way or another, POSIX really doesn't provide much besides "given two
> strings, which one is greater", and then only on a per-process basis.
> We'd really need more that, see also LIKE indexing issues, and indexing in
> general.

Correct. I'd suggest completely getting ride of OS's locale.

> * Client support: A lot of language environments provide pretty smooth
> Unicode support these days, e.g., Java, Perl 5.6, and I think that C99 has
> also made some strides. So while "we can store stuff in any character set
> you want" is great, it's really no good if it doesn't work transparently
> with the client interfaces. At least something to keep in mind.

Do you suggest that we should convert everyting into Unicode and store
them into DB?
--
Tatsuo Ishii


SQL_TEXT (Re: Re: Big 7.1 open items)

From
Tatsuo Ishii
Date:
> Yuck. That is what is says, all right :(
> 
> Date says that SQL_TEXT is required to have two things:
> 1) all characters used in the SQL language itself (which is what I
> recalled)
> 
> 2) Every other character from every character set in the installation.

Doesn't it say "charcter repertory", rather than character set? I
think it would be possible to let our SQL_TEXT support every character
repertories in the world, if we use Unicode or Mule internal code for
that.
--
Tatsuo Ishii


Re: Big 7.1 open items

From
Chris Bitmead
Date:
"Ross J. Reedstrom" wrote:

> The important point I think is that tablespaces are about physical
> storage/namespace, and SCHEMA are about logical namespace: it would make
> sense for tables from multiple schema to live in the same tablespace,
> as well as tables from one schema to be stored in multiple tablespaces.

If we accept that argument (which sounds good) then wouldn't we have...

data/base/db1/table1 -> ../../../tablespace/ts1/db1.table1
data/base/db1/table2 -> ../../../tablespace/ts1/db1.table2
data/tablespace/ts1/db1.table1
data/tablespace/ts1/db1.table2

In other words there is a directory for databases, and a directory for
tablespaces. Database tables are symlinked to the appropriate
tablespace. So there is multiple databases per tablespace and multiple
tablespaces per database.


RE: Big 7.1 open items

From
"Philip J. Warner"
Date:
At 22:12 20/06/00 -0700, Don Baccus wrote:
>At 11:22 AM 6/21/00 +1000, Philip J. Warner wrote:
>
>>It may be worth considering leaving the CREATE TABLE statement alone.
>>Dec/RDB uses a new statement entirely to define where a table goes...
>
>It's worth considering, but on the other hand Oracle users greatly
>outnumber Compaq/RDB users these days...

It's actually Oracle/Rdb, but I call it Dec/Rdb to distinguish it from
'Oracle/Oracle'. It was acquired by Oracle, supposedly because Oracle
wanted their optimizer, management and tuning tools (although that was only
hearsay). They *say* that they plan to merge the two products.

What I was trying to suggest was that the CREATE TABLE statement will get
very overloaded, and it might be worth avoiding having to support two
storage management syntaxes if/when it becomes desirable to create a
'storage' statement of some kind.


>
>I'm not saying I like the fact, Oracle's a pain in the ass.  But when
>adopting existing syntax, might as well adopt that of the crushing
>borg.
>

Only if it is a good thing, or part of a real standard. Philosophically,
where possible I would prefer to see statement that are *in* the SQL
standard (ie. CREATE TABLE) to be left as unencumbered as possible.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|                                |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: pgsql-hackers-owner@hub.org
> [mailto:pgsql-hackers-owner@hub.org]On Behalf Of Chris Bitmead
>
> "Ross J. Reedstrom" wrote:
>
> > The important point I think is that tablespaces are about physical
> > storage/namespace, and SCHEMA are about logical namespace: it would make
> > sense for tables from multiple schema to live in the same tablespace,
> > as well as tables from one schema to be stored in multiple tablespaces.
>
> If we accept that argument (which sounds good) then wouldn't we have...
>
> data/base/db1/table1 -> ../../../tablespace/ts1/db1.table1
> data/base/db1/table2 -> ../../../tablespace/ts1/db1.table2
> data/tablespace/ts1/db1.table1
> data/tablespace/ts1/db1.table2
>

Hmm,is above symlinking business really preferable just because
it is possible ?  Why do we have to be dependent upon directory
tree representation when we handle db structure ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp



Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > Sure, and if the system loading it can not create the required symlinks
> > because the directories don't exist, it can just skip the symlink step.
> 
> What I meant is, would you still be able to create tablespaces on
> systems without symlinks? That would seem to be a desirable feature.

You could create tablespaces, but you could not point them at different
drives.  The issue is that we don't store the symlink location in the
database, just the tablespace name.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Yes, agreed.  I was thinking this:
>     CREATE TABLESPACE loc USING '/var/pgsql'
> does:
>     ln -s /var/pgsql/dbname/loc data/base/dbname/loc 
> In this way, the database has a view of its main directory, plus a /loc
> subdirectory for the tablespace.  In the other location, we have
> /var/pgsql/dbname/loc because this allows different databases to use:
>     CREATE TABLESPACE loc USING '/var/pgsql'
> and they do not collide with each other in /var/pgsql.

But they don't collide anyway, because the dbname is already unique.
Isn't the extra subdirectory a waste?

Because table files will have installation-wide unique names, there's
no really good reason to have either level of subdirectory; you could
just makeCREATE TABLESPACE loc USING '/var/pgsql'
doln -s /var/pgsql data/base/dbname/loc 
and it'd still work even if multiple DBs were using the same tablespace.

However, forcing creation of a subdirectory does give you the chance to
make sure the subdir is owned by postgres and has the right permissions,
so there's something to be said for that.  It might be reasonable to domkdir /var/pgsql/dbnamechmod 700
/var/pgsql/dbnameln-s /var/pgsql/dbname data/base/dbname/loc 
 
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> At 12:27 PM 6/21/00 +1000, Chris Bitmead wrote:
> >Tom Lane wrote:
> >> Some unhappiness was raised about
> >> depending on symlinks for this function, but I didn't hear one single
> >> concrete reason not to do it, nor an alternative design. 
> >
> >Are symlinks portable?
> 
> In today's world?  Yeah, I think so.
> 
> My only unhappiness has hinged around the possibility that a new
> storage scheme might temp folks to toss aside the sgmr abstraction,
> or weaken it.
> 
> It doesn't appear that this will happen. 
> 
> Given an adequate sgmr abstraction, it doesn't really matter what
> low-level model is adopted in some sense (i.e. other models might
> become available, the implemented model might get replaced, etc -
> without breaking backends).
> 
> Obviously we'll all be using the default model for some time, maybe
> forever, but if mistakes are made maintaining the smgr abstraction
> means that replacements are possible.  Or kinky substitutes like
> working with DAFS.

The symlink solution where the actual symlink location is not stored
in the database is certainly abstract.  We store that info in the file
system, which is where it belongs.  We only query the symlink location
when we need it for database location dumping.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > Yes, I didn't like the environment variable stuff.  In fact, I would
> > like to not mention the symlink location anywhere in the database, so 
> > it can be changed without changing it in the database.
> 
> Well, as y'all have noticed, I think there are strong reasons to use
> environment variables to manage locations, and that symlinks are a
> potential portability and robustness problem.

Sorry, disagree.  Environment variables are a pain to administer, and
quite counter-intuitive.

I also don't see any portability or robustness problems.  Can you be
more specific?

> An additional point which has relevance to this whole discussion:
> 
> In the future we may allow system resource such as tables to carry names
> which use multi-byte encodings. afaik these encodings are not allowed to
> be used for physical file names, and even if they were the utility of
> using standard operating system utilities like ls goes way down.

That is really a different issues of file names.  Multi-byte table names
can be made to hold just the oid.  We have complete control over that
because the file name will be in pg_class.

> istm that from a portability and evolutionary standpoint OID-only file
> names (or at least file names *not* based on relation/class names) is a
> requirement.

Maybe a requirement at some point for some installations, but I hope not
a general requirement.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> >> There is a great deal to be said for
> >> ..../database/tablespace/filename
> 
> > OK,I seem to have gotten the answer for the question
> >    Is tablespace defined per PostgreSQL's database ?
> 
> Not necessarily --- the tablespace subdirectories could be symlinks
> pointing to the same place (assuming you use OIDs or something to keep
> the table filenames unique even across databases).  This is just an
> implementation mechanism; it doesn't foreclose the policy decision
> whether tablespaces are database-local or installation-wide.

Seems we are better just auto-creating a directory that matches the
dbname.

> 
> (OTOH, pathnames like tablespace/database would pretty much force
> tablespaces to be installation-wide whether you wanted it that way
> or not.)


> 
> > My opinion
> >    3) database and tablespace are relatively irrelevant.
> >        I assume PostgreSQL's database would correspond 
> >        to the concept of SCHEMA.
> 
> My inclindation is that tablespaces should be installation-wide, but
> I'm not completely sold on it.  In any case I could see wanting a
> permissions mechanism that would only allow some databases to have
> tables in a particular tablespace.

On idea is to allow tablespaces defined in template1 to be propogated to
newly created directories, with the symlinks adjusted so they use the
proper dbname in the symlink.


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> The important point I think is that tablespaces are about physical
> storage/namespace, and SCHEMA are about logical namespace: it would make
> sense for tables from multiple schema to live in the same tablespace,
> as well as tables from one schema to be stored in multiple tablespaces.
> 

It seems mixing the physical layout and the logical SCHEMA would have
problems because people have different reasons for using each feature.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> At 22:12 20/06/00 -0700, Don Baccus wrote:
> >At 11:22 AM 6/21/00 +1000, Philip J. Warner wrote:
> >
> >>It may be worth considering leaving the CREATE TABLE statement alone.
> >>Dec/RDB uses a new statement entirely to define where a table goes...
> >
> >It's worth considering, but on the other hand Oracle users greatly
> >outnumber Compaq/RDB users these days...
> 
> It's actually Oracle/Rdb, but I call it Dec/Rdb to distinguish it from
> 'Oracle/Oracle'. It was acquired by Oracle, supposedly because Oracle
> wanted their optimizer, management and tuning tools (although that was only
> hearsay). They *say* that they plan to merge the two products.
> 
> What I was trying to suggest was that the CREATE TABLE statement will get
> very overloaded, and it might be worth avoiding having to support two
> storage management syntaxes if/when it becomes desirable to create a
> 'storage' statement of some kind.
> 

Seems adding tablespace to CREATE TABLE/INDEX/DATABASE is pretty simple.
Doing it as a separate command seems cumbersome.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> Well, as y'all have noticed, I think there are strong reasons to use
> environment variables to manage locations, and that symlinks are a
> potential portability and robustness problem.

Reasons?  Evidence?

> An additional point which has relevance to this whole discussion:
> In the future we may allow system resource such as tables to carry names
> which use multi-byte encodings. afaik these encodings are not allowed to
> be used for physical file names, and even if they were the utility of
> using standard operating system utilities like ls goes way down.

Good point, although in one sense a string is a string --- as long as
we don't allow embedded nulls in server-side encodings, we could use
anything that Postgres thought was a name in a filename, and the OS
should take it.  But if your local ls doesn't show it the way you see
in Postgres, the usefulness of having the tablename in the filename
goes way down.

> istm that from a portability and evolutionary standpoint OID-only file
> names (or at least file names *not* based on relation/class names) is a
> requirement.

No argument from me ;-).  I've been looking for compromise positions
but I still think that pure numeric filenames are the cleanest solution.

There's something else that should be taken into account: for WAL, the
log will need to record the table file that each insert/delete/update
operation affects.  To do that with the smgr-token-is-a-pathname
approach I was suggesting yesterday, I think you have to record the
database name and pathname in each WAL log entry.  That's 64 bytes/log
entry which is a *lot*.  If we bit the bullet and restricted ourselves
to numeric filenames then the log would need just four numeric values:database OIDtablespace OIDrelation OIDrelation
versionnumber
 
(this set of 4 values would also be an smgr file reference token).
16 bytes/log entry looks much better than 64.

At the moment I can recall the following opinions:

Pure OID filenames: Thomas, Tom, Marc, Peter E.

OID+relname filenames: Bruce

Vadim was in the pure-OID camp a few months ago, but I won't presume
to list him there now since he hasn't been involved in this most
recent round of discussions.  I'm not sure where anyone else stands...
but at least in terms of the core group it's pretty clear where the
majority opinion is.
        regards, tom lane


Re: Big 7.1 open items

From
Thomas Lockhart
Date:
> Sorry, disagree.  Environment variables are a pain to administer, and
> quite counter-intuitive.

Well, I guess we disagree. But until we have a complete proposed
solution, we should leave environment variables on the table, since they
*do* allow some decoupling of logical and physical storage, and *do*
give the administrator some control over resources *that the admin would
not otherwise have*.

> > istm that from a portability and evolutionary standpoint OID-only 
> > file names (or at least file names *not* based on relation/class 
> > names) is a requirement.
> Maybe a requirement at some point for some installations, but I hope 
> not a general requirement.

If a table name can have characters which are not legal for file names,
then how would you propose to support it? If we are doing a
restructuring of the storage scheme, this should be taken into account.

lockhart=# create table "one/two" (i int);
ERROR:  cannot create one/two

Why not? It demonstrates an unfortunate linkage between file systems and
database resources.
                    - Thomas


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Yes, agreed.  I was thinking this:
> >     CREATE TABLESPACE loc USING '/var/pgsql'
> > does:
> >     ln -s /var/pgsql/dbname/loc data/base/dbname/loc 
> > In this way, the database has a view of its main directory, plus a /loc
> > subdirectory for the tablespace.  In the other location, we have
> > /var/pgsql/dbname/loc because this allows different databases to use:
> >     CREATE TABLESPACE loc USING '/var/pgsql'
> > and they do not collide with each other in /var/pgsql.
> 
> But they don't collide anyway, because the dbname is already unique.
> Isn't the extra subdirectory a waste?

Not really.  Yes, we could put them all in the same directory, but why
bother.  Probably easier to put them in unique directories per database.
Cuts down on directory searches to open file, and allows 'du' to return
meaningful numbers per database.  If you don't do that, you can't really
tell what files belong to which databases.

> 
> Because table files will have installation-wide unique names, there's
> no really good reason to have either level of subdirectory; you could
> just make
>     CREATE TABLESPACE loc USING '/var/pgsql'
> do
>     ln -s /var/pgsql data/base/dbname/loc 
> and it'd still work even if multiple DBs were using the same tablespace.
> 
> However, forcing creation of a subdirectory does give you the chance to
> make sure the subdir is owned by postgres and has the right permissions,
> so there's something to be said for that.  It might be reasonable to do
>     mkdir /var/pgsql/dbname
>     chmod 700 /var/pgsql/dbname
>     ln -s /var/pgsql/dbname data/base/dbname/loc 

Yes, that is true.  My idea is that they may want to create loc1 and
loc2 which initially point to the same location, but later may be moved.
For example, one tablespace for tables, another for indexes.  They may
initially point to the same directory, but later be split.  Seems we
need to keep the actual tablespace information relivant by using
different directories on the other end too.


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Lamar Owen
Date:
Tom Lane wrote:
> Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
> > Well, as y'all have noticed, I think there are strong reasons to use
> > environment variables to manage locations, and that symlinks are a
> > potential portability and robustness problem.
> Reasons?  Evidence?

Does Win32 do symlinks these days?  I know Win32 does envvars, and Win32
is currently a supported platform.

I'm not thrilled with either solution -- envvars have their problems
just as surely as symlinks do.
> At the moment I can recall the following opinions:
> Pure OID filenames: Thomas, Tom, Marc, Peter E.

FWIW, count me here.  I have tried administering my system using the
filenames -- and have been bitten.  Better admin tools in the PostgreSQL
package beat using standard filesystem tools -- the PostgreSQL tools can
be WAL-aware, transaction-aware, and can provide consistent results. 
Filesystem tools never will be able to provide consistent results for a
database system that must remain up 24x7, as many if not most PostgreSQL
installations must.
> OID+relname filenames: Bruce

Sorry Bruce -- I understand and am sympathetic to your position, and, at
one time, I agreed with it.  But not any more.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> FWIW, count me here.  I have tried administering my system using the
> filenames -- and have been bitten.  Better admin tools in the PostgreSQL
> package beat using standard filesystem tools -- the PostgreSQL tools can
> be WAL-aware, transaction-aware, and can provide consistent results. 
> Filesystem tools never will be able to provide consistent results for a
> database system that must remain up 24x7, as many if not most PostgreSQL
> installations must.
>  
> > OID+relname filenames: Bruce
> 
> Sorry Bruce -- I understand and am sympathetic to your position, and, at
> one time, I agreed with it.  But not any more.

I thought the most recent proposal was to just throw ~16 chars of the
file name on the end of the file name, and that should not be used for
anything except visibility.  WAL would not need to store that.  It could
just grab the file name that matches the oid/sequence number.

If people don't want table names in the file name, I totally understand,
and we can move on without them.  I have made the best case I can for
their inclusion, but if people are not convinced, then maybe I was
wrong.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Yes, that is true.  My idea is that they may want to create loc1 and
> loc2 which initially point to the same location, but later may be moved.
> For example, one tablespace for tables, another for indexes.  They may
> initially point to the same directory, but later be split.

Well, that opens up a completely different issue, which is what about
moving tables from one tablespace to another?

I think the way you appear to be implying above (shut down the server
so that you can rearrange subdirectories by hand) is the wrong way to
go about it.  For one thing, lots of people don't want to shut down
their servers completely for that long, but it's difficult to avoid
doing so if you want to move files by filesystem commands.  For another
thing, the above approach requires guessing in advance --- maybe long
in advance --- how you are going to want to repartition your database
when it gets too big for your existing storage.

The right way to address this problem is to invent a "move table to
new tablespace" command.  This'd be pretty trivial to implement based
on a file-versioning approach: the new version of the pg_class tuple
has a new tablespace identifier in it.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Yes, that is true.  My idea is that they may want to create loc1 and
> > loc2 which initially point to the same location, but later may be moved.
> > For example, one tablespace for tables, another for indexes.  They may
> > initially point to the same directory, but later be split.
> 
> Well, that opens up a completely different issue, which is what about
> moving tables from one tablespace to another?

Are you suggesting that doing dbname/locname is somehow harder to do
that?  If you are, I don't understand why.

The general issue of moving tables between tablespaces can be done from
in the database.  I don't think it is reasonable to shut down the db to
do that.  However, I can see moving tablespaces to different symlinked
locations may require a shutdown.

> 
> I think the way you appear to be implying above (shut down the server
> so that you can rearrange subdirectories by hand) is the wrong way to
> go about it.  For one thing, lots of people don't want to shut down
> their servers completely for that long, but it's difficult to avoid
> doing so if you want to move files by filesystem commands.  For another
> thing, the above approach requires guessing in advance --- maybe long
> in advance --- how you are going to want to repartition your database
> when it gets too big for your existing storage.
> 
> The right way to address this problem is to invent a "move table to
> new tablespace" command.  This'd be pretty trivial to implement based
> on a file-versioning approach: the new version of the pg_class tuple
> has a new tablespace identifier in it.

Agreed.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Sorry Bruce -- I understand and am sympathetic to your position, and, at
>> one time, I agreed with it.  But not any more.

> I thought the most recent proposal was to just throw ~16 chars of the
> file name on the end of the file name, and that should not be used for
> anything except visibility.  WAL would not need to store that.  It could
> just grab the file name that matches the oid/sequence number.

But that's extra complexity in WAL, plus extra complexity in renaming
tables (if you want the filename to track the logical table name, which
I expect you would), plus extra complexity in smgr and bufmgr and other
places.

I think people are coming around to the notion that it's better to keep
these low-level operations simple, even if we need to expend more work
on high-level admin tools as a result.

But we do need to remember to expend that effort on tools!  Let's not
drop the ball on that, folks.
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Well, that opens up a completely different issue, which is what about
>> moving tables from one tablespace to another?

> Are you suggesting that doing dbname/locname is somehow harder to do
> that?  If you are, I don't understand why.

It doesn't make it harder, but it still seems pointless to have the
extra directory level.  Bear in mind that if we go with all-OID
filenames then you're not going to be looking at "loc1" and "loc2"
anyway, but at "5938171" and "8583727".  It's not much of a convenience
to the admin to see that, so we might as well save a level of directory
lookup.

> The general issue of moving tables between tablespaces can be done from
> in the database.  I don't think it is reasonable to shut down the db to
> do that.  However, I can see moving tablespaces to different symlinked
> locations may require a shutdown.

Only if you insist on doing it outside the database using filesystem
tools.  Another way is to create a new tablespace in the desired new
location, then move the tables one-by-one to that new tablespace.

I suppose either one might be preferable depending on your access
patterns --- locking your most critical tables while they're being moved
might be as bad as a total shutdown.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> Well, that opens up a completely different issue, which is what about
> >> moving tables from one tablespace to another?
> 
> > Are you suggesting that doing dbname/locname is somehow harder to do
> > that?  If you are, I don't understand why.
> 
> It doesn't make it harder, but it still seems pointless to have the
> extra directory level.  Bear in mind that if we go with all-OID
> filenames then you're not going to be looking at "loc1" and "loc2"
> anyway, but at "5938171" and "8583727".  It's not much of a convenience
> to the admin to see that, so we might as well save a level of directory
> lookup.

Just seems easier to have stuff segregates into separate per-db
directories for clarity.  Also, as directories get bigger, finding a
specific file in there becomes harder.  Putting 10 databases all in the
same directory seems bad in this regard.

> 
> > The general issue of moving tables between tablespaces can be done from
> > in the database.  I don't think it is reasonable to shut down the db to
> > do that.  However, I can see moving tablespaces to different symlinked
> > locations may require a shutdown.
> 
> Only if you insist on doing it outside the database using filesystem
> tools.  Another way is to create a new tablespace in the desired new
> location, then move the tables one-by-one to that new tablespace.
> 
> I suppose either one might be preferable depending on your access
> patterns --- locking your most critical tables while they're being moved
> might be as bad as a total shutdown.

Seems we are better having the directory be a symlink so we don't have
symlink overhead for every file open.  Also, symlinks when removed just
remove symlink and not the file.  I don't think we want to be using
symlinks for tables if we can avoid it.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>>>> Are you suggesting that doing dbname/locname is somehow harder to do
>>>> that?  If you are, I don't understand why.
>> 
>> It doesn't make it harder, but it still seems pointless to have the
>> extra directory level.  Bear in mind that if we go with all-OID
>> filenames then you're not going to be looking at "loc1" and "loc2"
>> anyway, but at "5938171" and "8583727".  It's not much of a convenience
>> to the admin to see that, so we might as well save a level of directory
>> lookup.

> Just seems easier to have stuff segregates into separate per-db
> directories for clarity.  Also, as directories get bigger, finding a
> specific file in there becomes harder.  Putting 10 databases all in the
> same directory seems bad in this regard.

Huh?  I wasn't arguing against making a db-specific directory below the
tablespace point.  I was arguing against making *another* directory
below that one.

> I don't think we want to be using
> symlinks for tables if we can avoid it.

Agreed, but where did that come from?  None of these proposals mentioned
symlinks for anything but directories, AFAIR.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > Just seems easier to have stuff segregates into separate per-db
> > directories for clarity.  Also, as directories get bigger, finding a
> > specific file in there becomes harder.  Putting 10 databases all in the
> > same directory seems bad in this regard.
> 
> Huh?  I wasn't arguing against making a db-specific directory below the
> tablespace point.  I was arguing against making *another* directory
> below that one.

I was suggesting:
ln -s /var/pgsql/dbname/loc data/base/dbname/loc

I thought you were suggesting:
ln -s /var/pgsql/dbname data/base/dbname/loc

With this system:
ln -s /var/pgsql/dbname data/base/dbname/loc1ln -s /var/pgsql/dbname data/base/dbname/loc2

go into the same directory, which makes it impossible to move loc1
easily using the file system.  Seems cheap to add the extra directory.

> > I don't think we want to be using
> > symlinks for tables if we can avoid it.
> 
> Agreed, but where did that come from?  None of these proposals mentioned
> symlinks for anything but directories, AFAIR.

I thought you mentioned it.  Sorry.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Tom Lane writes:

> I think Peter was holding out for storing purely numeric tablespace OID
> and table version in pg_class and having a hardwired mapping to pathname
> somewhere in smgr.  However, I think that doing it that way gains only
> micro-efficiency compared to passing a "name" around, while using the
> name approach buys us flexibility that's needed for at least some of
> the variants under discussion.

But that name can only be a dozen or so characters, contain no slash or
other funny characters, etc. That's really poor. Then the alternative is
to have an internal name and an external canonical name. Then you have two
names to worry about. Also consider that when you store both the table
space oid and the internal name in pg_class you create redundant data.
What if you rename the table space? Do you leave the internal name out of
sync? Then what good is the internal name? I'm just concerned that we are
creating at the table space level problems similar to that we're trying to
get rid of at the relation and database level.


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> Tom Lane writes:
> 
> > I think Peter was holding out for storing purely numeric tablespace OID
> > and table version in pg_class and having a hardwired mapping to pathname
> > somewhere in smgr.  However, I think that doing it that way gains only
> > micro-efficiency compared to passing a "name" around, while using the
> > name approach buys us flexibility that's needed for at least some of
> > the variants under discussion.
> 
> But that name can only be a dozen or so characters, contain no slash or
> other funny characters, etc. That's really poor. Then the alternative is
> to have an internal name and an external canonical name. Then you have two
> names to worry about. Also consider that when you store both the table
> space oid and the internal name in pg_class you create redundant data.
> What if you rename the table space? Do you leave the internal name out of
> sync? Then what good is the internal name? I'm just concerned that we are
> creating at the table space level problems similar to that we're trying to
> get rid of at the relation and database level.

Agreed.  Having table spaces stored by directories named by oid just
seems very complicated for no reason.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> But that name can only be a dozen or so characters, contain no slash or
>> other funny characters, etc. That's really poor. Then the alternative is
>> to have an internal name and an external canonical name. Then you have two
>> names to worry about. Also consider that when you store both the table
>> space oid and the internal name in pg_class you create redundant data.
>> What if you rename the table space? Do you leave the internal name out of
>> sync? Then what good is the internal name? I'm just concerned that we are
>> creating at the table space level problems similar to that we're trying to
>> get rid of at the relation and database level.

> Agreed.  Having table spaces stored by directories named by oid just
> seems very complicated for no reason.

Huh?  He just gave you two very good reasons: avoid Unix-derived
limitations on the naming of tablespaces (and tables), and avoid
problems with renaming tablespaces.

I'm pretty much firmly back in the "OID and nothing but" camp.
Or perhaps I should say "OID, file version, and nothing but",
since we still need a version number to do CLUSTER etc.
        regards, tom lane


Re: Big 7.1 open items

From
"Randall Parker"
Date:
<font default="FACE" face="Arial"><font default="SIZE" pointsize="10" size="2">Lamar,<br /><br /> See:<br /><br /><font
color="0000ff"><u><fontpointsize="9" size="1">http://support.microsoft.com/support/kb/articles/Q205/5/24.ASP<br /><font
color="000000"default="COLOR"></font></font></u><br /> IMO, its a bad idea to require the use of symlinks in order to
beable to put different tablespaces on different drives. For a discussion on how DB2 supports tablespaces see my
messageentitled:<br /> "tablespace managed by system vs managed by database"<br /><br /> I think one of the reasons one
needsa fairly complex syntax for creating table spaces is that different devices have different hardware
characteristicsand one might want to tell the RDBMS to treat them differently for that reason. You can see how DB2
allowsyou to do that if you read that message I posted about it. <font color="000000" default="COLOR"><font
default="SIZE"pointsize="10" size="2"><br /><br /> On Wed, 21 Jun 2000 11:48:19 -0400, Lamar Owen wrote:<br /><br />
>DoesWin32 do symlinks these days? I know Win32 does envvars, and Win32<br /> >is currently a supported
platform.<br/></font></font></font></font></font> 

Re: Big 7.1 open items

From
"Randall Parker"
Date:
Tom,

DB2 supports an ALTER TABLESPACE command that allows one to add new containers to an existing tablespace. IMO, that's
farmore supportive of 24x7 usage.
 

On Wed, 21 Jun 2000 12:10:15 -0400, Tom Lane wrote:

>The right way to address this problem is to invent a "move table to
>new tablespace" command.  This'd be pretty trivial to implement based
>on a file-versioning approach: the new version of the pg_class tuple
>has a new tablespace identifier in it.




RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> If we bit the bullet and restricted ourselves to numeric filenames then
> the log would need just four numeric values:
>     database OID
>     tablespace OID

Is someone going to implement it for 7.1?

>     relation OID
>     relation version number

I believe that we can avoid versions using WAL...

> (this set of 4 values would also be an smgr file reference token).
> 16 bytes/log entry looks much better than 64.
> 
> At the moment I can recall the following opinions:
> 
> Pure OID filenames: Thomas, Tom, Marc, Peter E.

+ me.

But what about LOCATIONs? I object using environment and think that
locations
must be stored in pg_control..?

Vadim


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
>
> No argument from me ;-).  I've been looking for compromise positions
> but I still think that pure numeric filenames are the cleanest solution.
>
> There's something else that should be taken into account: for WAL, the
> log will need to record the table file that each insert/delete/update
> operation affects.  To do that with the smgr-token-is-a-pathname
> approach I was suggesting yesterday, I think you have to record the
> database name and pathname in each WAL log entry.  That's 64 bytes/log
> entry which is a *lot*.  If we bit the bullet and restricted ourselves
> to numeric filenames then the log would need just four numeric values:
>     database OID
>     tablespace OID

I strongly object to keep tablespace OID for smgr file reference token
though we have to keep it for another purpose of cource. I've mentioned
many times tablespace(where to store) info should be distinguished from
*where it is stored* info. Generally tablespace isn't sufficiently
restrictive
for this purpose. e.g. there was an idea about round-robin. e.g. Oracle's
tablespace could have pluaral files... etc.
IMHO,it is misleading to use tablespace OID as (a part of) reference token.

>     relation OID
>     relation version number
> (this set of 4 values would also be an smgr file reference token).
> 16 bytes/log entry looks much better than 64.
>

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp



RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Mikheev, Vadim [mailto:vmikheev@SECTORBASE.COM]
> 
> > If we bit the bullet and restricted ourselves to numeric filenames then
> > the log would need just four numeric values:
> >     database OID
> >     tablespace OID
> 
> Is someone going to implement it for 7.1?
> 
> >     relation OID
> >     relation version number
> 
> I believe that we can avoid versions using WAL...
>

How to re-construct tables in place ?
Is the following right ?
1) save the content of current table to somewhere
2) shrink the table and related indexes
3) reload the saved(+some filtering) content

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> > >     relation version number
> > 
> > I believe that we can avoid versions using WAL...
> >
> 
> How to re-construct tables in place ?
> Is the following right ?
> 1) save the content of current table to somewhere
> 2) shrink the table and related indexes
> 3) reload the saved(+some filtering) content

Or - create tmp file and load with new content; log "intent to relink table
file";
relink table file; log "file is relinked".

Vadim



RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Mikheev, Vadim [mailto:vmikheev@SECTORBASE.COM]
> 
> > > >     relation version number
> > > 
> > > I believe that we can avoid versions using WAL...
> > >
> > 
> > How to re-construct tables in place ?
> > Is the following right ?
> > 1) save the content of current table to somewhere
> > 2) shrink the table and related indexes
> > 3) reload the saved(+some filtering) content
> 
> Or - create tmp file and load with new content; log "intent to 
> relink table
> file";
> relink table file; log "file is relinked".
>

It seems to me that whole content of the table should be
logged before relinking or shrinking.
Is my understanding right ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp 


Re: Big 7.1 open items

From
Chris Bitmead
Date:
Bruce Momjian wrote:

> The symlink solution where the actual symlink location is not stored
> in the database is certainly abstract.  We store that info in the file
> system, which is where it belongs.  We only query the symlink location
> when we need it for database location dumping.

how would that work? would pg_dump dump the tablespace locations or not?


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> > Or - create tmp file and load with new content;
> > log "intent to relink table file";
> > relink table file; log "file is relinked".
> 
> It seems to me that whole content of the table should be
> logged before relinking or shrinking.

Why not just fsync tmp files?

Vadim


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> At the moment I can recall the following opinions:
> 
> Pure OID filenames: Thomas, Tom, Marc, Peter E.
> 
> OID+relname filenames: Bruce
>

Please add my opinion to the list.

Unique-id filename: Hiroshi(Unqiue-id is irrelevant to OID/relname).

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Mikheev, Vadim [mailto:vmikheev@SECTORBASE.COM]
> 
> > > Or - create tmp file and load with new content;
> > > log "intent to relink table file";
> > > relink table file; log "file is relinked".
> > 
> > It seems to me that whole content of the table should be
> > logged before relinking or shrinking.
> 
> Why not just fsync tmp files?
>

Probably I've misunderstood *relink*.
If *relink* different from *rename* ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> > > > Or - create tmp file and load with new content;
> > > > log "intent to relink table file";
> > > > relink table file; log "file is relinked".
> > > 
> > > It seems to me that whole content of the table should be
> > > logged before relinking or shrinking.
> > 
> > Why not just fsync tmp files?
> >
> 
> Probably I've misunderstood *relink*.
> If *relink* different from *rename* ?

I ment something like this - link(table file, tmp2 file); fsync(tmp2 file);
unlink(table file); link(tmp file, table file); fsync(table file);
unlink(tmp file). We can do additional logging (with log flush) of these
steps
if required, postpone on-recovery redo of operations till last relink log
record/
end of log/transaction abort etc etc etc.

Vadim


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Bruce Momjian wrote:
> 
> > The symlink solution where the actual symlink location is not stored
> > in the database is certainly abstract.  We store that info in the file
> > system, which is where it belongs.  We only query the symlink location
> > when we need it for database location dumping.
> 
> how would that work? would pg_dump dump the tablespace locations or not?
> 

pg_dump would recreate a CREATE TABLESPACE command:
printf("CREATE TABLESPACE %s USING %s", loc, symloc);

where symloc would be SELECT symloc(loc) and return the value into a
variable that is used by pg_dump.  The backend would do the lstat() and
return the value to the client.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
"Randall Parker" <randall@nls.net> writes:
> DB2 supports an ALTER TABLESPACE command that allows one to add new
> containers to an existing tablespace. IMO, that's far more supportive
> of 24x7 usage.

Er, what do they mean by "container", and why is it better?
        regards, tom lane


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Mikheev, Vadim [mailto:vmikheev@SECTORBASE.COM]
> 
> > > > > Or - create tmp file and load with new content;
> > > > > log "intent to relink table file";
> > > > > relink table file; log "file is relinked".
> > > > 
> > > > It seems to me that whole content of the table should be
> > > > logged before relinking or shrinking.
> > > 
> > > Why not just fsync tmp files?
> > >
> > 
> > Probably I've misunderstood *relink*.
> > If *relink* different from *rename* ?
> 
> I ment something like this - link(table file, tmp2 file); 
> fsync(tmp2 file);
> unlink(table file); link(tmp file, table file); fsync(table file);
> unlink(tmp file).

I see,old file would be rolled back from tmp2 file on abort.
This would work on most platforms.
But cygwin port has a flaw that files could not be unlinked
if they are open. So *relink* may fail in some cases(including
rollback cases).

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


Re: Big 7.1 open items

From
Tom Lane
Date:
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:
>> relation OID
>> relation version number

> I believe that we can avoid versions using WAL...

I don't think so.  You're basically saying that1. create file 'new'2. delete file 'old'3. rename 'new' to 'old'
is safe as long as you have a redo log to ensure that the rename
happens even if you crash between steps 2 and 3.  But crash is not
the only hazard.  What if step 3 just plain fails?  Redo won't help.

I'm having a hard time inventing really plausible examples, but a
slightly implausible example is that someone chmod's the containing
directory -w between steps 2 and 3.  (Maybe it's not so implausible
if you assume a crash after step 2 ... someone might have left the
directory nonwritable while restoring the system.)

If we use file version numbers, then the *only* thing needed to
make a valid transition between one set of files and another is
a commit of the update of pg_class that shows the new version number
in the rel's pg_class tuple.  The worst that can happen to you in
a crash or other failure is that you are unable to get rid of the
set of files that you don't want anymore.  That might waste disk
space but it doesn't leave the database corrupted.

> But what about LOCATIONs? I object using environment and think that
> locations must be stored in pg_control..?

I don't like environment variables for this either; it's just way too
easy to start the postmaster with wrong environment.  It still seems
to me that relying on subdirectory symlinks is a good way to go.
pg_control is not so good --- if it gets corrupted, how do you recover?
symlinks can be recreated by hand if necessary, but...
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> Please add my opinion to the list.
> Unique-id filename: Hiroshi
>  (Unqiue-id is irrelevant to OID/relname).

"Unique ID" is more or less equivalent to "OID + version number",
right?

I was trying earlier to convince myself that a single unique-ID value
would be better than OID+version for the smgr interface, because it'd
certainly be easier to pass around.  I failed to convince myself though,
and the thing that bothered me was this.  Suppose you are trying to
recover a corrupted database manually, and the only information you have
about which table is which is a somewhat out-of-date listing of OIDs
versus table names.  (Maybe it's out of date because you got it from
your last backup tape.)  If the files are named OID+version you're not
going to have much trouble seeing which is which, even if some of the
versions are higher than what was on the tape.  But if version-updated
tables are given entirely new unique IDs, you've got no hope at all of
telling which one corresponds to what you had in the listing.  Maybe
you can tell by looking through the physical file contents, but
certainly this way is more fragile from the point of view of data
recovery.
        regards, tom lane


Re: Big 7.1 open items

From
Chris Bitmead
Date:
Bruce Momjian wrote:
> 
> > Bruce Momjian wrote:
> >
> > > The symlink solution where the actual symlink location is not stored
> > > in the database is certainly abstract.  We store that info in the file
> > > system, which is where it belongs.  We only query the symlink location
> > > when we need it for database location dumping.
> >
> > how would that work? would pg_dump dump the tablespace locations or not?
> >
> 
> pg_dump would recreate a CREATE TABLESPACE command:
> 
>         printf("CREATE TABLESPACE %s USING %s", loc, symloc);
> 
> where symloc would be SELECT symloc(loc) and return the value into a
> variable that is used by pg_dump.  The backend would do the lstat() and
> return the value to the client.

I'm wondering if pg_dump should store the location of the tablespace. If
your machine dies, you get a new machine to re-create the database, you
may not want the tablespace in the same spot. And text-editing a
gigabyte file would be extremely painful.


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > where symloc would be SELECT symloc(loc) and return the value into a
> > variable that is used by pg_dump.  The backend would do the lstat() and
> > return the value to the client.
> 
> I'm wondering if pg_dump should store the location of the tablespace. If
> your machine dies, you get a new machine to re-create the database, you
> may not want the tablespace in the same spot. And text-editing a
> gigabyte file would be extremely painful.

If the symlink create fails in CREATE TABLESPACE, it just creates an
ordinary directory.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> I strongly object to keep tablespace OID for smgr file reference token
> though we have to keep it for another purpose of cource. I've mentioned
> many times tablespace(where to store) info should be distinguished from
> *where it is stored* info.

Sure.  But this proposal assumes that we're relying on symlinks to
carry the information about physical locations corresponding to
tablespace OIDs.  The backend just needs to know enough to access a
relation file at a relative pathname liketablespaceOID/relationOID
(ignoring version and segment numbers for now).  Under the hood,
a symlink for tablespaceOID gets the work done.

Certainly this is not a perfect mechanism.  But it is simple, it
is reliable, it is portable to most of the platforms we care about
(yeah, I know we have a Win port, but you wouldn't ever recommend
someone to run a *serious* database on it would you?), and in general
I think the bang-for-the-buck ratio is enormous.  I do not want to
have to deal with explicit tablespace bookkeeping in the backend,
but that seems like what we'd have to do in order to improve on
symlinks.
        regards, tom lane


Re: Big 7.1 open items

From
"Randall Parker"
Date:
Tom,

A "container" can be a file or a device or a directory. Here again are examples that I already posted in 
another thread:

In the first example there are 3 devices specified as containers. In the second example 3 directories are 
specified as containers (DB2 therefore makes its own file names in it - and uses OIDs to do it I think). In 
the third example 2 files are the 2 containers. In the fourth example 6 devices on 3 nodes are the 
containers.
  CREATE TABLESPACE PAYROLL     MANAGED BY DATABASE     USING (DEVICE'/dev/rhdisk6' 10000,        DEVICE '/dev/rhdisk7'
10000,       DEVICE '/dev/rhdisk8' 10000)     OVERHEAD 24.1     TRANSFERRATE 0.9
 

  CREATE TABLESPACE ACCOUNTING     MANAGED BY SYSTEM     USING ('d:\acc_tbsp', 'e:\acc_tbsp', 'f:\acc_tbsp')
EXTENTSIZE64     PREFETCHSIZE 32
 


  CREATE TEMPORARY TABLESPACE TEMPSPACE2     MANAGED BY DATABASE     USING (FILE '/tmp/tempspace2.f1' 50000,
FILE'/tmp/tempspace2.f2' 50000)     EXTENTSIZE 256
 

  CREATE TABLESPACE PLANS     MANAGED BY DATABASE     USING (DEVICE '/dev/rhdisk0' 10000, DEVICE '/dev/rn1hd01' 40000)
ONNODE 1     USING (DEVICE '/dev/rhdisk0' 10000, DEVICE '/dev/rn3hd03' 40000) ON NODE 3     USING (DEVICE
'/dev/rhdisk0'10000, DEVICE '/dev/rn5hd05' 40000) ON NODE 5
 

On Wed, 21 Jun 2000 23:03:03 -0400, Tom Lane wrote:

>"Randall Parker" <randall@nls.net> writes:
>> DB2 supports an ALTER TABLESPACE command that allows one to add new
>> containers to an existing tablespace. IMO, that's far more supportive
>> of 24x7 usage.
>
>Er, what do they mean by "container", and why is it better?
>
>            regards, tom lane





Re: Big 7.1 open items

From
Don Baccus
Date:
At 01:43 PM 6/22/00 +1000, Chris Bitmead wrote:

>I'm wondering if pg_dump should store the location of the tablespace. If
>your machine dies, you get a new machine to re-create the database, you
>may not want the tablespace in the same spot. And text-editing a
>gigabyte file would be extremely painful.

So you don't dump your create tablespace statements, recognizing that on
a new machine (due to upgrades or crashing) you might assign them to
different directories/mount points/whatever.  That's the reason for
wanting to hide physical allocation in tablespaces ... the rest of
your datamodel doesn't need to know.

Or you do dump your tablespaces, and knowing the paths assigned
to various ones set up your new machine accordingly.



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > I strongly object to keep tablespace OID for smgr file reference token
> > though we have to keep it for another purpose of cource. I've mentioned
> > many times tablespace(where to store) info should be distinguished from
> > *where it is stored* info.
> 
> Sure.  But this proposal assumes that we're relying on symlinks to
> carry the information about physical locations corresponding to
> tablespace OIDs.  The backend just needs to know enough to access a
> relation file at a relative pathname like
>     tablespaceOID/relationOID
> (ignoring version and segment numbers for now).  Under the hood,
> a symlink for tablespaceOID gets the work done.
>

I think tablespaceOID is an easy substitution for the purpose.
I don't like to depend on poor directory tree structure in dbms
either.. 
> Certainly this is not a perfect mechanism.  But it is simple, it
> is reliable, it is portable to most of the platforms we care about
> (yeah, I know we have a Win port, but you wouldn't ever recommend
> someone to run a *serious* database on it would you?), and in general
> I think the bang-for-the-buck ratio is enormous.  I do not want to
> have to deal with explicit tablespace bookkeeping in the backend,
> but that seems like what we'd have to do in order to improve on
> symlinks.
>

I've already mentioned about it 10 times or so but unfortunately
I see no one on my side yet. 
OK,I've given up the discussion about it.  I don't want to waste
my time any more.

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


Re: Big 7.1 open items

From
Don Baccus
Date:
At 12:03 AM 6/22/00 -0400, Bruce Momjian wrote:

>If the symlink create fails in CREATE TABLESPACE, it just creates an
>ordinary directory.

Silent surprises - the earmark of truly professional software ...



- Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert
Serviceand other goodies at http://donb.photo.net.
 


Re: Big 7.1 open items

From
"Philip J. Warner"
Date:
At 13:43 22/06/00 +1000, Chris Bitmead wrote:
>Bruce Momjian wrote:
>
>I'm wondering if pg_dump should store the location of the tablespace. If
>your machine dies, you get a new machine to re-create the database, you
>may not want the tablespace in the same spot. And text-editing a
>gigabyte file would be extremely painful.
>

This is a very good point; the way Dec/RDB gets around it is to allow the
'pg_restore' command to override storage settings when restoring a backup
file.


----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|                                |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Re: Big 7.1 open items

From
"Philip J. Warner"
Date:
At 23:27 21/06/00 -0400, Tom Lane wrote:
>"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
>> Please add my opinion to the list.
>> Unique-id filename: Hiroshi
>>  (Unqiue-id is irrelevant to OID/relname).
>
>I was trying earlier to convince myself that a single unique-ID value
>would be better than OID+version for the smgr interface, because it'd
>certainly be easier to pass around.  I failed to convince myself though,
>and the thing that bothered me was this.  Suppose you are trying to
>recover a corrupted database manually, and the only information you have
>about which table is which is a somewhat out-of-date listing of OIDs
>versus table names.

This worries me a little; in the Dec/RDB world it is a very long time since
database backups were done by copying the files. There is a database
backup/restore utility which runs while the database is on-line and makes
sure a valid snapshot is taken. Backing up storage areas (table spapces)
can be done separately by the same utility, and again, it records enough
information to ensure integrity. Maybe the thing to do is write a pg_backup
utility, which in a first pass could, presumably, be synonymous with pg_dump?

Am I missing something here? Is there a problem with backing up using
'pg_dump | gzip'?


>  (Maybe it's out of date because you got it from
>your last backup tape.)  If the files are named OID+version you're not
>going to have much trouble seeing which is which, even if some of the
>versions are higher than what was on the tape.

Unfortunately here you hit severe RI problems, unless you use a 'proper'
database backup.


>  But if version-updated
>tables are given entirely new unique IDs, you've got no hope at all of
>telling which one corresponds to what you had in the listing.  Maybe
>you can tell by looking through the physical file contents, but
>certainly this way is more fragile from the point of view of data
>recovery.

In the Dec/RDB world, one has to very occasionally restore from files (this
only happens if multiple prior database backups and after-image journals
are corrupt). In this case, there is a utility for examining and changing
storage area file information. This is probably way over the top for
PostgreSQL.

[Aside: FWIW, the Dec/RDB storage area files are named by DBAs to be
something meaningful to the DBA (eg. EMPLOYEE_ACHIVE), and can contain one
of more tables etc. The files are never renamed or moved by the database
without an instruction from the DBA. The 'storage manager' manages the
datafiles internally. Usually, tables are allocated in chunks of multiples
of some file-based buffer size, and the file grows as needed. This allows
for disk read-ahead to be useful, while storing multiple tables in one
file. As stated in a previous message, tables can also be split across
storage areas]

Once again, I hope I have not missed a fundamental fact...

----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|                                |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


Re: Big 7.1 open items

From
Tom Lane
Date:
Chris Bitmead <chrisb@nimrod.itg.telstra.com.au> writes:
> I'm wondering if pg_dump should store the location of the tablespace. If
> your machine dies, you get a new machine to re-create the database, you
> may not want the tablespace in the same spot. And text-editing a
> gigabyte file would be extremely painful.

Might make sense to store the tablespace setup separately from the bulk
of the data, but certainly you want some way to dump that info in a
restorable form.

I've been thinking lately that the pg_dump shove-it-all-in-one-file
approach doesn't scale anyway.  We ought to start thinking about ways
to make the standard dump method store schema separately from bulk
data, for example.  That's offtopic for this thread but ought to be
on the TODO list someplace...
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
"Philip J. Warner" <pjw@rhyme.com.au> writes:
>> ... the thing that bothered me was this.  Suppose you are trying to
>> recover a corrupted database manually, and the only information you have
>> about which table is which is a somewhat out-of-date listing of OIDs
>> versus table names.

> This worries me a little; in the Dec/RDB world it is a very long time since
> database backups were done by copying the files. There is a database
> backup/restore utility which runs while the database is on-line and makes
> sure a valid snapshot is taken. Backing up storage areas (table spapces)
> can be done separately by the same utility, and again, it records enough
> information to ensure integrity. Maybe the thing to do is write a pg_backup
> utility, which in a first pass could, presumably, be synonymous with pg_dump?

pg_dump already does the consistent-snapshot trick (it just has to run
inside a single transaction).

> Am I missing something here? Is there a problem with backing up using
> 'pg_dump | gzip'?

None, as long as your ambition extends no further than restoring your
data to where it was at your last pg_dump.  I was thinking about the
all-too-common-in-the-real-world scenario where you're hoping to recover
some data more recent than your last backup from the fractured shards
of your database...
        regards, tom lane


Re: Big 7.1 open items

From
"Philip J. Warner"
Date:
At 03:17 22/06/00 -0400, Tom Lane wrote:
>
>> This worries me a little; in the Dec/RDB world it is a very long time since
>> database backups were done by copying the files. There is a database
>> backup/restore utility which runs while the database is on-line and makes
>> sure a valid snapshot is taken. Backing up storage areas (table spapces)
>> can be done separately by the same utility, and again, it records enough
>> information to ensure integrity. Maybe the thing to do is write a pg_backup
>> utility, which in a first pass could, presumably, be synonymous with
pg_dump?
>
>pg_dump already does the consistent-snapshot trick (it just has to run
>inside a single transaction).
>
>> Am I missing something here? Is there a problem with backing up using
>> 'pg_dump | gzip'?
>
>None, as long as your ambition extends no further than restoring your
>data to where it was at your last pg_dump.  I was thinking about the
>all-too-common-in-the-real-world scenario where you're hoping to recover
>some data more recent than your last backup from the fractured shards
>of your database...
>

pg_dump is a good basis for any pg_backup utility; perhaps as you indicated
elsewhere, more carefull formatting of the dump files would make
table-based restoration possible. In another response, I also suggested
allowing overrides of placement information in a restore operation- the
simplest approach would be an 'ignore-storage-parameters' flag. Does this
sound reasonable? If so, then discussion of file-id based on OID needs not
be too concerned about how db restoration is done.





----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|                                |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Peter Eisentraut [mailto:e99re41@DoCS.UU.SE]
> 
> > My opinion
> >    3) database and tablespace are relatively irrelevant.
> >        I assume PostgreSQL's database would correspond 
> >        to the concept of SCHEMA.
> 
> A database corresponds to a catalog and a schema corresponds to nothing
> yet.
>

Oh I see your point. However I've thought that current PostgreSQL's
database is an imcomplete SCHEMA and still feel so in reality.
Catalog per database has been nothing but needless for me from
the first.

Regards.  

Hiroshi Inoue
Inoue@tpf.co.jp


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > Please add my opinion to the list.
> > Unique-id filename: Hiroshi
> >  (Unqiue-id is irrelevant to OID/relname).
> 
> "Unique ID" is more or less equivalent to "OID + version number",
> right?
>

Hmm,no one seems to be on my side at this point also.
OK,I change my mind as follows.
  OID except cygwin,unique-id on cygwin

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


Re: Big 7.1 open items

From
Giles Lean
Date:
> I ment something like this - link(table file, tmp2 file); fsync(tmp2 file);
> unlink(table file); link(tmp file, table file); fsync(table file);
> unlink(tmp file).

I don't see the purpose of the fsync() calls here: link() and unlink()
effect file system metadata, which with normal Unix (but not Linux)
filesystem semantics is written synchronously.

fsync() on a file forces outstanding data to disk; it doesn't effect
the preceding or subsequent link() and unlink() calls unless
McKusick's soft updates are in use.

If the intent is to make sure the files are in particular states
before each of the link() and unlink() calls (i.e. soft updates or
similar functionality are in use) then more than fsync() is required,
since the files can still be updated after the fsync() and before
link() or unlink().

On Linux I understand that a fsync() on a directory will force
metadata updates to that directory to be committed, but that doesn't
seem to be what this code is trying to do either?

Regards,

Giles





Re: Big 7.1 open items

From
Bruce Momjian
Date:
> At 01:43 PM 6/22/00 +1000, Chris Bitmead wrote:
> 
> >I'm wondering if pg_dump should store the location of the tablespace. If
> >your machine dies, you get a new machine to re-create the database, you
> >may not want the tablespace in the same spot. And text-editing a
> >gigabyte file would be extremely painful.
> 
> So you don't dump your create tablespace statements, recognizing that on
> a new machine (due to upgrades or crashing) you might assign them to
> different directories/mount points/whatever.  That's the reason for
> wanting to hide physical allocation in tablespaces ... the rest of
> your datamodel doesn't need to know.
> 
> Or you do dump your tablespaces, and knowing the paths assigned
> to various ones set up your new machine accordingly.

I imagine we will have a -l flag to pg_dump to dump tablespace
locations.  If they exist on the new machine, we use them.  If not, we
create just directories with no symlinks.


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
>> A database corresponds to a catalog and a schema corresponds to nothing
>> yet.

> Oh I see your point. However I've thought that current PostgreSQL's
> database is an imcomplete SCHEMA and still feel so in reality.
> Catalog per database has been nothing but needless for me from
> the first.

It may be needless for you, but not for everybody ;-).

In my mind the point of the "database" concept is to provide a domain
within which custom datatypes and functions are available.  Schemas
will control the visibility of tables, but SQL92 hasn't thought about
controlling visibility of datatypes or functions.  So I think we will
still want "database" = "span of applicability of system catalogs"
and multiple databases allowed per installation, even though there may
be schemas subdividing the database(s).
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> OK,I change my mind as follows.
>    OID except cygwin,unique-id on cygwin

We don't really want to do that, do we?  That's a huge difference in
behavior to have in just one port --- especially a port that none of
the primary developers use (AFAIK anyway).  The cygwin port's normal
state of existence will be "broken", surely, if we go that way.

Besides which, OID alone doesn't give us a possibility of file
versioning, and as I commented to Vadim I think we will want that,
WAL or no WAL.  So it seems to me the two viable choices are
unique-id or OID+version-number.  Either way, the file-naming behavior
should be the same across all platforms.
        regards, tom lane


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> > I believe that we can avoid versions using WAL...
> 
> I don't think so.  You're basically saying that
>     1. create file 'new'
>     2. delete file 'old'
>     3. rename 'new' to 'old'
> is safe as long as you have a redo log to ensure that the rename
> happens even if you crash between steps 2 and 3.  But crash is not
> the only hazard.  What if step 3 just plain fails?  Redo won't help.

Ok, ok. Let's use *unique* file name for each table version.
But after thinking, seems that I agreed with Hiroshi about using
*some unique id* for file names instead of oid+version: we could use
just DB' OID + this unique ID in log records to find table file - just
8 bytes.

So, add me to Hiroshi' camp... if Hiroshi is ready to implement new file
naming -:)

> > But what about LOCATIONs? I object using environment and think that
> > locations must be stored in pg_control..?
> 
> I don't like environment variables for this either; it's just way too
> easy to start the postmaster with wrong environment.  It still seems
> to me that relying on subdirectory symlinks is a good way to go.

I always thought so.

> pg_control is not so good --- if it gets corrupted, how do 
> you recover?

Impossible to recover anyway - pg_control keeps last checkpoint pointer,
required for recovery. That's why Oracle recommends (requires?) at least
two copies of control file (and log too).
But what if log gets corrupted? Or file system (lost symlinks etc)?
One will have to use backup...

Vadim


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> pg_dump is a good basis for any pg_backup utility; perhaps as you indicated
> elsewhere, more carefull formatting of the dump files would make
> table-based restoration possible. In another response, I also suggested
> allowing overrides of placement information in a restore operation- the
> simplest approach would be an 'ignore-storage-parameters' flag. Does this
> sound reasonable? If so, then discussion of file-id based on OID needs not
> be too concerned about how db restoration is done.

My idea was to make dumping of tablespace locations/symlinks optional. 
By trying to control it on the load end, you have to basically have some
way of telling the backend to ignore the symlinks on load.  Right now,
pg_dump just creates SQL commands and COPY commands.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
The Hermit Hacker
Date:
On Wed, 21 Jun 2000, Don Baccus wrote:

> At 01:43 PM 6/22/00 +1000, Chris Bitmead wrote:
> 
> >I'm wondering if pg_dump should store the location of the tablespace. If
> >your machine dies, you get a new machine to re-create the database, you
> >may not want the tablespace in the same spot. And text-editing a
> >gigabyte file would be extremely painful.
> 
> So you don't dump your create tablespace statements, recognizing that on
> a new machine (due to upgrades or crashing) you might assign them to
> different directories/mount points/whatever.  That's the reason for
> wanting to hide physical allocation in tablespaces ... the rest of
> your datamodel doesn't need to know.
> 
> Or you do dump your tablespaces, and knowing the paths assigned
> to various ones set up your new machine accordingly.

Or, modify pg_dump so that it auto-dumps to two files, one for schema, one
for data.  then its easier to modify the schema on a large database if
tablespaces change ...




Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Tom Lane writes:

> In my mind the point of the "database" concept is to provide a domain
> within which custom datatypes and functions are available.

Quoth SQL99:

"A user-defined type is a schema object"

"An SQL-invoked routine is an element of an SQL-schema"

I have yet to see anything in SQL that's a per-catalog object. Some things
are global, like users, but everything else is per-schema.

The way I see it is that schemas are required to be a logical hierarchy,
whereas implementations may see catalogs as a physical division (as indeed
this implementation does).

> So I think we will still want "database" = "span of applicability of
> system catalogs"

Yes, because the system catalogs would live in a schema of their own.


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Peter Eisentraut
>
> Tom Lane writes:
>
> > In my mind the point of the "database" concept is to provide a domain
> > within which custom datatypes and functions are available.
>

AFAIK few users understand it and many users have wondered
why we couldn't issue cross "database" queries.

> Quoth SQL99:
>
> "A user-defined type is a schema object"
>
> "An SQL-invoked routine is an element of an SQL-schema"
>
> I have yet to see anything in SQL that's a per-catalog object. Some things
> are global, like users, but everything else is per-schema.
>

So why is system catalog needed per "database" ?

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp



Re: Big 7.1 open items

From
Chris Bitmead
Date:
The Hermit Hacker wrote:

> Or, modify pg_dump so that it auto-dumps to two files, one for schema, one
> for data.  then its easier to modify the schema on a large database if
> tablespaces change ...

That's a pretty good idea as an option. But I'd say keep the schema
separate from the tablespace locations. And if you're going down that
path why not create a directory automatically and dump each table into a
separate file. On occasion I've had to restore one table by hand-editing
the pg_dump, and that's a real pain.


Re: Big 7.1 open items

From
Philip Warner
Date:
At 09:55 23/06/00 +1000, Chris Bitmead wrote:
>The Hermit Hacker wrote:
>
>> Or, modify pg_dump so that it auto-dumps to two files, one for schema, one
>> for data.  then its easier to modify the schema on a large database if
>> tablespaces change ...
>
>That's a pretty good idea as an option. But I'd say keep the schema
>separate from the tablespace locations. And if you're going down that
>path why not create a directory automatically and dump each table into a
>separate file. On occasion I've had to restore one table by hand-editing
>the pg_dump, and that's a real pain.
>

Have a look at my message entitled:

Proposal: More flexible backup/restore via pg_dump

It's supposed to address these issues.



----------------------------------------------------------------
Philip Warner                    |     __---_____
Albatross Consulting Pty. Ltd.   |----/       -  \
(A.C.N. 008 659 498)             |          /(@)   ______---_
Tel: (+61) 0500 83 82 81         |                 _________  \
Fax: (+61) 0500 83 82 82         |                 ___________ |
Http://www.rhyme.com.au          |                /           \|                                |    --________--
PGP key available upon request,  |  /
and from pgp5.ai.mit.edu:11371   |/


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Mikheev, Vadim [mailto:vmikheev@SECTORBASE.COM]
> 
> > > I believe that we can avoid versions using WAL...
> > 
> > I don't think so.  You're basically saying that
> >     1. create file 'new'
> >     2. delete file 'old'
> >     3. rename 'new' to 'old'
> > is safe as long as you have a redo log to ensure that the rename
> > happens even if you crash between steps 2 and 3.  But crash is not
> > the only hazard.  What if step 3 just plain fails?  Redo won't help.
> 
> Ok, ok. Let's use *unique* file name for each table version.
> But after thinking, seems that I agreed with Hiroshi about using
> *some unique id* for file names instead of oid+version: we could use
> just DB' OID + this unique ID in log records to find table file - just
> 8 bytes.
> 
> So, add me to Hiroshi' camp... if Hiroshi is ready to implement new file
> naming -:)
>

I've thought e.g. newfileid() like newoid() using pg_variable.
Other smarter ways ? 

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp


Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Bruce Momjian writes:

> Here is the list I have gotten of open 7.1 items:

>     new location for config files

I'm on that task now, more or less by accident but I might as well get it
done. I'm reorganizing all the file name handling code for pg_hba.conf,
pg_indent.conf, pg_control, etc. so they have consistent accessor
routines. The DataDir global variable will disappear, you'll have to use
GetDataDir().

-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> Bruce Momjian writes:
> 
> > Here is the list I have gotten of open 7.1 items:
> 
> >     new location for config files
> 
> I'm on that task now, more or less by accident but I might as well get it
> done. I'm reorganizing all the file name handling code for pg_hba.conf,
> pg_indent.conf, pg_control, etc. so they have consistent accessor
> routines. The DataDir global variable will disappear, you'll have to use
> GetDataDir().
> 

Can we get agreement to remove our secondary password files, and make
something that makes more sense?


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Bruce Momjian writes:

> Can we get agreement to remove our secondary password files, and make
> something that makes more sense?

How about this: Normally secondary password files look like

username:ABS5SGh1EL6bk

We could add the option of making them look like

username:+

which means "look into pg_shadow". That would be fully backward
compatible, allows the use of alter user with password, and avoids
creating any extra system tables (that would need to be dumped to plain
text). And the coding looks very simple.

-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> Bruce Momjian writes:
> 
> > Can we get agreement to remove our secondary password files, and make
> > something that makes more sense?
> 
> How about this: Normally secondary password files look like
> 
> username:ABS5SGh1EL6bk
> 
> We could add the option of making them look like
> 
> username:+
> 
> which means "look into pg_shadow". That would be fully backward
> compatible, allows the use of alter user with password, and avoids
> creating any extra system tables (that would need to be dumped to plain
> text). And the coding looks very simple.

Yes, perfect. In fact, how about:

> username

as doing that.  Any username with no colon uses pg_shadow.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: SQL_TEXT (Re: Re: Big 7.1 open items)

From
Thomas Lockhart
Date:
> > Date says that SQL_TEXT is required to have two things:
> > 1) all characters used in the SQL language itself (which is what I
> > recalled)
> > 2) Every other character from every character set in the 
> > installation.
> Doesn't it say "charcter repertory", rather than character set? I
> think it would be possible to let our SQL_TEXT support every character
> repertories in the world, if we use Unicode or Mule internal code for
> that.

I think that "character set" and "character repertoire" are synonymous
(at least I am interpreting them that way). SQL99 makes a slight
distinction, in that "repertoire" is a "set" in a specific context of
application.

I'm starting to look at the SQL99 doc. I am going to try to read the doc
as if SQL_TEXT is a placeholder for "any allowed character set", not
"all character sets simultaneously" and see if that works. 

Since there are a wide range of encodings to choose from, and since most
character sets can not be translated to another random character set,
having SQL_TEXT usefully require all sets present simultaneously seems a
bit of a stretch.

I'm also not going to try to understand the complete doc before having a
trial solution; we can extend/modify/redefine/throw away the trial
solution as we understand the spec better.

While I'm thinking about it: afaict, if we have the ability to load
multiple character sets simultaneously, we will want to have *one* of
those mapped in as the "default character set" for an installation or
database. So we might want to statically link that one in, while the
others get loaded dynamically.
                   - Thomas


Re: SQL_TEXT (Re: Re: Big 7.1 open items)

From
Tatsuo Ishii
Date:
> I think that "character set" and "character repertoire" are synonymous
> (at least I am interpreting them that way). 

Is it? I think that a "character set" consists of a "character
repertoire" and a "form of use".

> SQL99 makes a slight
> distinction, in that "repertoire" is a "set" in a specific context of
> application.

I don't understand this probably due to my English ability. Can you
tell me where I can get SQL99 on line doc so that I could study it
more?

> While I'm thinking about it: afaict, if we have the ability to load
> multiple character sets simultaneously, we will want to have *one* of
> those mapped in as the "default character set" for an installation or
> database. So we might want to statically link that one in, while the
> others get loaded dynamically.

Right. While I am not sure we could statically link it, there could be
"default character set" in a installation or database. Also we could
have another "default character set" for NATIONAL CHARACTER. It seems
that those "default character set" are actually same acording to the
standard?
--
Tatsuo Ishii


Re: SQL_TEXT (Re: Re: Big 7.1 open items)

From
Thomas Lockhart
Date:
> I think that a "character set" consists of a "character
> repertoire" and a "form of use".
> > SQL99 makes a slight distinction, in that "repertoire" is a "set" in 
> > a specific context of application.
> I don't understand this probably due to my English ability.

I'm pretty sure that it is due to convoluted standards ;)

> Can you tell me where I can get SQL99 on line doc so that I could 
> study it more?

From Peter E:

ftp://jerry.ece.umassd.edu/isowg3/x3h2/Standards/ansi-iso-9075-[12345]-1999.txt

(a set of 5 files) which are also available in PDF at the same site.
               - Thomas


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> BTW we are about to take in tablespace concept. You would
> need another information(the name of the symlink to a directory
> ,would be = tablespaceOID) for WAL logging.

Do we need *both* database & tablespace to find table file ?!
Imho, database shouldn't be used...

?

Vadim


Re: Big 7.1 open items

From
Tom Lane
Date:
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:
> Do we need *both* database & tablespace to find table file ?!
> Imho, database shouldn't be used...

That'd work fine for me, but I think Bruce was arguing for paths that
included the database name.  We'd end up with paths that go something
like..../data/tablespaces/TABLESPACEOID/RELATIONOID
(plus some kind of decoration for segment and version), so you'd have
a hard time telling which files in a tablespace belong to which
database.  Doesn't bother me a whole lot, personally --- if one wants
to know that one could just as well assign separate tablespaces to
different databases.  They're only subdirectories anyway...
        regards, tom lane


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> > Do we need *both* database & tablespace to find table file ?!
> > Imho, database shouldn't be used...
> 
> That'd work fine for me, but I think Bruce was arguing for paths that
> included the database name.  We'd end up with paths that go something
> like
>     ..../data/tablespaces/TABLESPACEOID/RELATIONOID
> (plus some kind of decoration for segment and version), so you'd have
> a hard time telling which files in a tablespace belong to which
> database.  Doesn't bother me a whole lot, personally --- if one wants

We could create /data/databases/DATABASEOID/ and create soft-links to
table-files. This way different tables of the same database could be in
different tablespaces. /data/database path would be used in production
and /data/tablespace path would be used in recovery.

Vadim


Re: Big 7.1 open items

From
Tom Lane
Date:
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:
> We could create /data/databases/DATABASEOID/ and create soft-links to
> table-files. This way different tables of the same database could be in
> different tablespaces. /data/database path would be used in production
> and /data/tablespace path would be used in recovery.

Why would you want to do it that way?  Having a different access path
for recovery than for normal operation strikes me as just asking for
trouble ;-)

The symlinks wouldn't do any good for what Bruce had in mind anyway
(IIRC, he wanted to get useful per-database numbers from "du").
        regards, tom lane


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> > We could create /data/databases/DATABASEOID/ and create 
> > soft-links to table-files. This way different tables of
> > the same database could be in different tablespaces.
> > /data/database path would be used in production
> > and /data/tablespace path would be used in recovery.
> 
> Why would you want to do it that way?  Having a different access path
> for recovery than for normal operation strikes me as just asking for
> trouble ;-)

I just think that *databases* (schemas) must be used for *logical* groupping
of tables, not for *physical* one. "Where to store table" is tablespace'
related kind of things!

> The symlinks wouldn't do any good for what Bruce had in mind anyway
> (IIRC, he wanted to get useful per-database numbers from "du").

Imho, ability to put different tables/indices (of the same database)
to different tablespaces (disks) is much more useful then ability to
use du/ls for administration purposes -:)

Also, I think that we *must* go away from OS' driven disk space
allocation anyway. Currently, the way we extend table files breaks WAL
rule (nothing must go to disk untill logged). + we have to move tuples
from end of file to top to shrink relation - not perfect way to reuse
empty space. +... +... +...

Vadim


Re: Big 7.1 open items

From
Hiroshi Inoue
Date:
Tom Lane wrote:

>
> The symlinks wouldn't do any good for what Bruce had in mind anyway
> (IIRC, he wanted to get useful per-database numbers from "du").

Our database design seems to be in the opposite direction
if it is restricted for the convenience of command calls.

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp




Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Tom Lane wrote:
> 
> >
> > The symlinks wouldn't do any good for what Bruce had in mind anyway
> > (IIRC, he wanted to get useful per-database numbers from "du").
> 
> Our database design seems to be in the opposite direction
> if it is restricted for the convenience of command calls.

Well, I don't see any reason not to use tablespace/database rather than
just tablespace.  Seems having fewer files in each directory will be a
little faster, and if we can make administration easier, why not?

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Hiroshi Inoue
Date:
Bruce Momjian wrote:

> > Tom Lane wrote:
> >
> > >
> > > The symlinks wouldn't do any good for what Bruce had in mind anyway
> > > (IIRC, he wanted to get useful per-database numbers from "du").
> >
> > Our database design seems to be in the opposite direction
> > if it is restricted for the convenience of command calls.
>
> Well, I don't see any reason not to use tablespace/database rather than
> just tablespace.  Seems having fewer files in each directory will be a
> little faster, and if we can make administration easier, why not?
>

I  only objected to emphasize the advantage in getting useful per-database
numbers from "du".  It's only misleading in dbms design.

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp



RE: Big 7.1 open items

From
Peter Eisentraut
Date:
Mikheev, Vadim writes:

> Do we need *both* database & tablespace to find table file ?!
> Imho, database shouldn't be used...

Then the system tables from different databases would collide.


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> > > The symlinks wouldn't do any good for what Bruce had in 
> > > mind anyway (IIRC, he wanted to get useful per-database
> > > numbers from "du").
> > 
> > Our database design seems to be in the opposite direction
> > if it is restricted for the convenience of command calls.
> 
> Well, I don't see any reason not to use tablespace/database 
> rather than just tablespace. Seems having fewer files in each directory

Once again - ability to use different tablespaces (disks) for tables/indices
in the same schema. Schemas must not dictate where to store objects <-
bad design.

> will be a little faster, and if we can make administration easier,
> why not?

Because you'll not be able use du/ls once we'll implement new smgr anyway.

And, btw, - for what are we going implement tablespaces? Just to have
fewer files in each dir ?!

Vadim


Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> > > > The symlinks wouldn't do any good for what Bruce had in 
> > > > mind anyway (IIRC, he wanted to get useful per-database
> > > > numbers from "du").
> > > 
> > > Our database design seems to be in the opposite direction
> > > if it is restricted for the convenience of command calls.
> > 
> > Well, I don't see any reason not to use tablespace/database 
> > rather than just tablespace. Seems having fewer files in each directory
> 
> Once again - ability to use different tablespaces (disks) for tables/indices
> in the same schema. Schemas must not dictate where to store objects <-
> bad design.

I am suggesting this symlink:
ln -s data/base/testdb/myspace /var/myspace/testdb

rather than:
ln -s data/base/testdb/myspace /var/myspace

Tablespaces still sit inside database directories, it is just that it
points to a subdirectory of myspace, rather than myspace itself.

Am I missing something?

> 
> > will be a little faster, and if we can make administration easier,
> > why not?
> 
> Because you'll not be able use du/ls once we'll implement new smgr anyway.

At least du will work.  I doubt we will be putting tables from different
databases in the same file.

> 
> And, btw, - for what are we going implement tablespaces? Just to have
> fewer files in each dir ?!

No, I thought it was to split files across drives.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> > > Well, I don't see any reason not to use tablespace/database 
> > > rather than just tablespace. Seems having fewer files in 
> > > each directory
> > 
> > Once again - ability to use different tablespaces (disks) 
> > for tables/indices in the same schema. Schemas must not dictate
> > where to store objects <- bad design.
> 
> I am suggesting this symlink:
> 
>     ln -s data/base/testdb/myspace /var/myspace/testdb
> 
> rather than:
> 
>     ln -s data/base/testdb/myspace /var/myspace
> 
> Tablespaces still sit inside database directories, it is just that it
> points to a subdirectory of myspace, rather than myspace itself.
^^^^^^^^^^^

Didn't you mean 

ln -s /var/myspace/testdb data/base/testdb/myspace

?

I thought that you don't like symlinks from data/base/... This is
how I understood Tom' words:

> The symlinks wouldn't do any good for what Bruce had in mind anyway
> (IIRC, he wanted to get useful per-database numbers from "du").

Vadim


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> > I am suggesting this symlink:
> > 
> >     ln -s data/base/testdb/myspace /var/myspace/testdb
> > 
> > rather than:
> > 
> >     ln -s data/base/testdb/myspace /var/myspace
> > 
> > Tablespaces still sit inside database directories, it is just that it
> > points to a subdirectory of myspace, rather than myspace itself.
> ^^^^^^^^^^^

Sorry, I should have said a symlink sits in data/base and points to the
tablespace.  My issue is having it point to tablespace/database and not
just tablespace/.

> 
> Didn't you mean 
> 
> ln -s /var/myspace/testdb data/base/testdb/myspace
> 

No, sorry for the confusion.


> ?
> 
> I thought that you don't like symlinks from data/base/... This is
> how I understood Tom' words:
> 
> > The symlinks wouldn't do any good for what Bruce had in mind anyway
> > (IIRC, he wanted to get useful per-database numbers from "du").

No, I want symlinks in data/base.  What I wanted was to have
per-database directories in every tablespace so I can do a 'du' in the
tablespace directory to see how much of each database is the tablespace.
It also nicely partitions the files, makes the directories smaller, and
prevents any possible file name conflict.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> > Do we need *both* database & tablespace to find table file ?!
> > Imho, database shouldn't be used...
> 
> Then the system tables from different databases would collide.

Actually, if we're going to use unique-ids for file names
then we have to know how to get system file names anyway.
Hm, OID+VERSION would make our life easier... Hiroshi?

Vadim


Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> > > Do we need *both* database & tablespace to find table file ?!
> > > Imho, database shouldn't be used...
> > 
> > Then the system tables from different databases would collide.
> 
> Actually, if we're going to use unique-ids for file names
> then we have to know how to get system file names anyway.
> Hm, OID+VERSION would make our life easier... Hiroshi?

I assume we were going to have a pg_class.relversion to do that, but
that is per-database because pg_class is per-database.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> > > Then the system tables from different databases would collide.
> > 
> > Actually, if we're going to use unique-ids for file names
> > then we have to know how to get system file names anyway.
> > Hm, OID+VERSION would make our life easier... Hiroshi?
> 
> I assume we were going to have a pg_class.relversion to do that, but                                  ^^^^^^^^
PG_CLASS_OID.VERSION_ID...

Just a clarification -:)

> that is per-database because pg_class is per-database.

Vadim


Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> > > > Then the system tables from different databases would collide.
> > > 
> > > Actually, if we're going to use unique-ids for file names
> > > then we have to know how to get system file names anyway.
> > > Hm, OID+VERSION would make our life easier... Hiroshi?
> > 
> > I assume we were going to have a pg_class.relversion to do that, but
>                                    ^^^^^^^^
> PG_CLASS_OID.VERSION_ID...
> 
> Just a clarification -:)

I actually meant I thought we were going to have a pg_class column
called relversion that held the currently active version for that
relation.

Yes, the file name will be pg_class_oid.version_id.

Is that OK?

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


RE: Big 7.1 open items

From
"Mikheev, Vadim"
Date:
> I actually meant I thought we were going to have a pg_class column
> called relversion that held the currently active version for that
> relation.
> 
> Yes, the file name will be pg_class_oid.version_id.
> 
> Is that OK?

We recently discussed pure *unique-id* file names...

Vadim


Re: Big 7.1 open items

From
Bruce Momjian
Date:
[ Charset ISO-8859-1 unsupported, converting... ]
> > I actually meant I thought we were going to have a pg_class column
> > called relversion that held the currently active version for that
> > relation.
> > 
> > Yes, the file name will be pg_class_oid.version_id.
> > 
> > Is that OK?
> 
> We recently discussed pure *unique-id* file names...
> 

Well, that would allow us to mix database files in the same directory,
if we wanted to do that.  My opinion it is better to keep databases in
separate directories in each tablespace for clarity and performance
reasons.


--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Peter Eisentraut <peter_e@gmx.net> writes:
> Mikheev, Vadim writes:
>> Do we need *both* database & tablespace to find table file ?!
>> Imho, database shouldn't be used...

> Then the system tables from different databases would collide.

I've been assuming that we would create a separate tablespace for
each database, which would be the location of that database's
system tables.  It's probably also the default tablespace for user
tables created in that database, though it wouldn't have to be.

There should also be a known tablespace for the installation-wide tables
(pg_shadow et al).

With this approach tablespace+relation would indeed be a sufficient
identifier.  We could even eliminate the knowledge that certain
tables are installation-wide from the bufmgr and below (currently
that knowledge is hardwired in places that I'd rather didn't know
about it...)
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Peter Eisentraut <peter_e@gmx.net> writes:
> > Mikheev, Vadim writes:
> >> Do we need *both* database & tablespace to find table file ?!
> >> Imho, database shouldn't be used...
> 
> > Then the system tables from different databases would collide.
> 
> I've been assuming that we would create a separate tablespace for
> each database, which would be the location of that database's
> system tables.  It's probably also the default tablespace for user
> tables created in that database, though it wouldn't have to be.
> 
> There should also be a known tablespace for the installation-wide tables
> (pg_shadow et al).
> 
> With this approach tablespace+relation would indeed be a sufficient
> identifier.  We could even eliminate the knowledge that certain
> tables are installation-wide from the bufmgr and below (currently
> that knowledge is hardwired in places that I'd rather didn't know
> about it...)

Well, if we did that, I can see a good reason to not use per-database
directories in the tablepspace.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Well, that would allow us to mix database files in the same directory,
> if we wanted to do that.  My opinion it is better to keep databases in
> separate directories in each tablespace for clarity and performance
> reasons.

One reason not to do that is that we'd still have to special-case
the system-wide relations.  If it's just tablespace and OID in the
path, then the system-wide rels look just the same as any other rel
as far as the low-level stuff is concerned.  That would be nice.

My feeling about the "clarity and performance" issue is that if a
dbadmin wants to keep track of database contents separately, he can
put different databases' tables into different tablespaces to start
with.  If he puts several tables into one tablespace, he's saying
he doesn't care about distinguishing their space usage.  There's
no reason for us to force an additional level of directory lookup
to be done whether the admin wants it or not.
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Well, that would allow us to mix database files in the same directory,
> > if we wanted to do that.  My opinion it is better to keep databases in
> > separate directories in each tablespace for clarity and performance
> > reasons.
> 
> One reason not to do that is that we'd still have to special-case
> the system-wide relations.  If it's just tablespace and OID in the
> path, then the system-wide rels look just the same as any other rel
> as far as the low-level stuff is concerned.  That would be nice.

Yes, good point about pg_shadow.  They don't have databases.  How do we
get multiple pg_class tables in the same directory?  Is the
pg_class.relversion file a number like 1,2,3,4, or does it come out of
some global counter like oid.  If so, we could put them in the same
directory.

Should we be concerned about performance when 10-20 database are sitting
in the same directory?  I am thinking about open() and other calls that
scan the directory.  Certainly shorter file names will help.

> 
> My feeling about the "clarity and performance" issue is that if a
> dbadmin wants to keep track of database contents separately, he can
> put different databases' tables into different tablespaces to start
> with.  If he puts several tables into one tablespace, he's saying
> he doesn't care about distinguishing their space usage.  There's
> no reason for us to force an additional level of directory lookup
> to be done whether the admin wants it or not.

OK.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Yes, good point about pg_shadow.  They don't have databases.  How do we
> get multiple pg_class tables in the same directory?  Is the
> pg_class.relversion file a number like 1,2,3,4, or does it come out of
> some global counter like oid.  If so, we could put them in the same
> directory.

I think we could get away with insisting that each database store its
pg_class and friends in a separate tablespace (physically distinct
directory) from any other database.  That gets around the OID conflict.

It's still an open question whether OID+version is better than
unique-ID for naming files that belong to different versions of the
same relation.  I can see arguments on both sides.
        regards, tom lane


Re: Big 7.1 open items

From
Hiroshi Inoue
Date:
Tom Lane wrote:

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Yes, good point about pg_shadow.  They don't have databases.  How do we
> > get multiple pg_class tables in the same directory?  Is the
> > pg_class.relversion file a number like 1,2,3,4, or does it come out of
> > some global counter like oid.  If so, we could put them in the same
> > directory.
>
> I think we could get away with insisting that each database store its
> pg_class and friends in a separate tablespace (physically distinct
> directory) from any other database.  That gets around the OID conflict.
>
> It's still an open question whether OID+version is better than
> unique-ID for naming files that belong to different versions of the
> same relation.  I can see arguments on both sides.
>

I don't stick to unique-ID. My main point has always been the
transactional control of file allocation change.
However *VERSION(_ID)* may be misleading because it couldn't
mean the version of pg_class tuples.

Regards.

Hiroshi Inoue
Inoue@tpf.co.jp




RE: Big 7.1 open items

From
Peter Mount
Date:
> Yes, the file name will be pg_class_oid.version_id.

What about segmented files (ie: those over 1Gb)?

--
Peter Mount
Enterprise Support
Maidstone Borough Council
Any views stated are my own, and not those of Maidstone Borough Council



Re: Big 7.1 open items

From
Tom Lane
Date:
Peter Mount <petermount@it.maidstone.gov.uk> writes:
>> Yes, the file name will be pg_class_oid.version_id.

> What about segmented files (ie: those over 1Gb)?

Separate issue.  Putting the segment number into the filename is
a bad idea because it doesn't give you any way to spread multiple
segments of a big table across filesystems.  What's currently being
discussed is paths that look like
something/SEGNO/RELATIONOID.VERSIONID

This lets you control space allocation by making the SEGNO
subdirectories be symlinks to various places.
        regards, tom lane


Re: Big 7.1 open items

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> If we put multiple database tables in the same directory, have we
> considered how to drop databases?  Right now we do rm -rf:

rm -rf will no longer work in a tablespaces environment anyway.
(Even if you kept symlinks underneath the DB directory, rm -rf
wouldn't follow them.)

DROP DATABASE will have to be implemented honestly: run through
pg_class and do a regular DROP on each user table.

Once you've got rid of the user tables, rm -rf should suffice to
get rid of the "home tablespace" as I've been calling it, with
all the system tables therein.

Now that you mention it, this is another reason why system tables for
each database have to live in a separate tablespace directory: there's
no other good way to do that final stage of DROP DATABASE.  The
DROP-each-table approach doesn't work for system tables (somewhere along
about the point where you drop pg_attribute, DROP TABLE itself would
stop working ;-)).

However I do see a bit of a problem here: since DROP DATABASE is
ordinarily executed by a backend that's running in a different database,
how's it going to read pg_class of the target database?  Perhaps it will
be necessary to fire up a sub-backend that runs in the target DB for
long enough to kill all the user tables.  Looking messy...
        regards, tom lane


Re: Big 7.1 open items

From
Bruce Momjian
Date:
> However I do see a bit of a problem here: since DROP DATABASE is
> ordinarily executed by a backend that's running in a different database,
> how's it going to read pg_class of the target database?  Perhaps it will
> be necessary to fire up a sub-backend that runs in the target DB for
> long enough to kill all the user tables.  Looking messy...

That was my feeling.  Imagine another issue.  If you see a file, how do
you figure out what database it belong to?  You would have to cycle
through the pg_class relations for every database.  Seems such reverse
lookups would not be impossible.  Not sure if it will ever be required.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


RE: Big 7.1 open items

From
"Hiroshi Inoue"
Date:
> -----Original Message-----
> From: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> 
> However I do see a bit of a problem here: since DROP DATABASE is
> ordinarily executed by a backend that's running in a different database,
> how's it going to read pg_class of the target database?  Perhaps it will
> be necessary to fire up a sub-backend that runs in the target DB for
> long enough to kill all the user tables.  Looking messy...
>

Why do we have to have system tables per *database* ?
Is there anything wrong with global system tables ?
And how about adding dbid to pg_class,pg_proc etc ?

Regards.

Hiroshi Inoue


Re: Big 7.1 open items

From
Tom Lane
Date:
"Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> Why do we have to have system tables per *database* ?
> Is there anything wrong with global system tables ?
> And how about adding dbid to pg_class,pg_proc etc ?

We could, but I think I'd vote against it on two grounds:

1. Reliability.  If something corrupts pg_class, do you want to
lose your whole installation, or just one database?

2. Increased locking overhead/loss of concurrency.  Currently, there
is very little lock contention between backends running in different
databases.  A shared pg_class will be a single point of locking (as
well as a single point of failure) for the whole installation.

It would solve the DROP DATABASE problem kind of nicely, but really
it'd just be downgrading DROP DATABASE to a DROP SCHEMA operation...
        regards, tom lane


Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Bruce Momjian writes:

> How do we get multiple pg_class tables in the same directory?

pg_class.DBOID


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden




Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Tom Lane writes:

> I've been assuming that we would create a separate tablespace for
> each database, which would be the location of that database's
> system tables.

Then I can't put more than one database into a table space? But I can put
more than one table space into a database? I think that's the wrong
hierarchy. More specifically, I think it's wrong that there is a hierarchy
here at all. Table spaces and databases don't have to know about each
other in any predefined way.


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Big 7.1 open items

From
Hiroshi Inoue
Date:
Tom Lane wrote:

> "Hiroshi Inoue" <Inoue@tpf.co.jp> writes:
> > Why do we have to have system tables per *database* ?
> > Is there anything wrong with global system tables ?
> > And how about adding dbid to pg_class,pg_proc etc ?
>
> We could, but I think I'd vote against it on two grounds:
>
> 1. Reliability.  If something corrupts pg_class, do you want to
> lose your whole installation, or just one database?
>
> 2. Increased locking overhead/loss of concurrency.  Currently, there
> is very little lock contention between backends running in different
> databases.  A shared pg_class will be a single point of locking (as
> well as a single point of failure) for the whole installation.

Isn't current design of PG's *database* for dropdb using "rm -rf"
rather than for above 1.2. ?
If we couldn't rely on our db itself and our locking mechanism is
poor,we could start different postmasters for different *database*s.


> It would solve the DROP DATABASE problem kind of nicely, but really
> it'd just be downgrading DROP DATABASE to a DROP SCHEMA operation...
>

What is our *DATABASE* ?
Is it clear to all people ?
At least it's a vague concept for me.
Could you please tell me what kind of objects are our *DATABASE*
objects but could not be schema objects ?

Regards.

Hiroshi Inoue




Re: Big 7.1 open items

From
Tom Lane
Date:
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> I've been assuming that we would create a separate tablespace for
>> each database, which would be the location of that database's
>> system tables.

> Then I can't put more than one database into a table space? But I can put
> more than one table space into a database?

You can put *user* tables from more than one database into a table space.
The restriction is just on *system* tables.

Admittedly this is a tradeoff.  We could avoid it along the lines you
suggest (name table files like DBOID.RELOID.VERSION instead of just
RELOID.VERSION) but is it really worth it?  Vadim's concerned about
every byte that has to go into the WAL log, and I think he's got a
good point.

> I think that's the wrong
> hierarchy. More specifically, I think it's wrong that there is a hierarchy
> here at all. Table spaces and databases don't have to know about each
> other in any predefined way.

They don't, at least not at the smgr level.  In my view of how this
should work, the smgr *only* knows about tablespaces and tables.
Databases are a higher-level construct.
        regards, tom lane


Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Tom Lane writes:

> You can put *user* tables from more than one database into a table space.
> The restriction is just on *system* tables.

I think my understanding as a user would be that a table space represents
a storage location. If I want to put a table/object/entire database on a
fancy disk somewhere I create a table space for it there. But if I want to
store all my stuff under /usr/local/pgsql/data then I wouldn't expect to
have to create more than one table space. So the table spaces become at
that point affected by the logical hierarchy: I must make sure to have
enough table spaces to have many databases.

More specifically, what would the user interface to this look like?
Clearly there has to be some sort of CREATE TABLESPACE command. Now does
CREATE DATABASE imply a CREATE TABLESPACE? I think not. Do you have to
create a table space before creating each database? I think not.

> We could avoid it along the lines you suggest (name table files like
> DBOID.RELOID.VERSION instead of just RELOID.VERSION) but is it really
> worth it?

I only intended that for pg_class and other bootstrap-sort-of tables,
maybe all system tables. Normal heap files could look like RELOID.VERSION,
whereas system tables would look like "name.DBOID". Clearly there's no
market for renaming system tables or dropping any of their columns. We're
obviously going to have to treat pg_class special anyway.

> Vadim's concerned about every byte that has to go into the WAL log,
> and I think he's got a good point.

True. But if you only do it for the system tables then it might take less
space than keeping track of lots of table spaces that are unneeded. :-)


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden




Re: Big 7.1 open items

From
Tom Lane
Date:
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> You can put *user* tables from more than one database into a table space.
>> The restriction is just on *system* tables.

> More specifically, what would the user interface to this look like?
> Clearly there has to be some sort of CREATE TABLESPACE command. Now does
> CREATE DATABASE imply a CREATE TABLESPACE? I think not. Do you have to
> create a table space before creating each database? I think not.

I would say that CREATE DATABASE just implicitly creates a new
tablespace that's physically located right under the toplevel data
directory of the installation, no symlink.  What's wrong with that?
You need not keep anything except the system tables of the DB there
if you don't want to.  In practice, for someone who doesn't need to
worry about tablespaces (because they put the installation on a disk
with enough room for their purposes), the whole thing acts exactly
the same as it does now.

>> We could avoid it along the lines you suggest (name table files like
>> DBOID.RELOID.VERSION instead of just RELOID.VERSION) but is it really
>> worth it?

> I only intended that for pg_class and other bootstrap-sort-of tables,
> maybe all system tables. Normal heap files could look like RELOID.VERSION,
> whereas system tables would look like "name.DBOID".

That would imply that the very bottom levels of the system know all
about which tables are system tables and which are not (and, if you
are really going to insist on the "name" part of that, that they
know what name goes with each system-table OID).  I'd prefer to avoid
that.  The less the smgr knows about the upper levels of the system,
the better.

> Clearly there's no market for renaming system tables or dropping any
> of their columns.

No, but there is a market for compacting indexes on system relations,
and I haven't heard a good proposal for doing index compaction in place.
So we need versioning for system indexes.

>> Vadim's concerned about every byte that has to go into the WAL log,
>> and I think he's got a good point.

> True. But if you only do it for the system tables then it might take less
> space than keeping track of lots of table spaces that are unneeded. :-)

Again, WAL should not need to distinguish system and user tables.

And as for the keeping track, the tablespace OID will simply replace the
database OID in the log and in the smgr interfaces.  There's no "extra"
cost, except maybe by comparison to a system with neither tablespaces
nor multiple databases.
        regards, tom lane


Re: Big 7.1 open items

From
Chris Bitmead
Date:
Tom Lane wrote:

> Now that you mention it, this is another reason why system tables for
> each database have to live in a separate tablespace directory: there's
> no other good way to do that final stage of DROP DATABASE.  The
> DROP-each-table approach doesn't work for system tables (somewhere along
> about the point where you drop pg_attribute, DROP TABLE itself would
> stop working ;-)).

If drop table is extended to drop multiple tables at once, then you read
and cache everything you need before doing all the destruction in the
second stage.


Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Tom Lane writes:

> In practice, for someone who doesn't need to worry about tablespaces
> (because they put the installation on a disk with enough room for
> their purposes), the whole thing acts exactly the same as it does now.

But I'd venture the guess that for someone who wants to use tablespaces it
wouldn't work as expected. Table spaces should represent a physical
storage location. Creation of table spaces should be a restricted
operation, possibly more than, but at least differently from, databases.
Eventually, table spaces probably will have attributes, such as
optimization parameters (random_page_cost). This will not work as expected
if you intermix them with the databases.

I'd expect that if I have three disks and 50 databases, then I make three
tablespaces and assign the databases to them. I'll bet lunch that if we
don't do it that way that before long people will come along and ask for
something that does work this way.


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden



Re: Big 7.1 open items

From
Tom Lane
Date:
Peter Eisentraut <peter_e@gmx.net> writes:
> I'd expect that if I have three disks and 50 databases, then I make three
> tablespaces and assign the databases to them.

In our last installment, you were complaining that you didn't want to
be bothered with that ;-)

But I don't see any reason why CREATE DATABASE couldn't take optional
parameters indicating where to create the new DB's default tablespace.
We already have a LOCATION option for it that does something close to
that.

Come to think of it, it would probably make sense to adapt the existing
notion of "location" (cf initlocation script) into something meaning
"directory that users are allowed to create tablespaces (including
databases) in".  If there were an explicit table of allowed locations,
it could be used to address the protection issues you raise --- for
example, a location could be restricted so that only some users could
create tablespaces/databases in it.  $PGDATA/data would be just the
first location in every installation.
        regards, tom lane


Re: Big 7.1 open items

From
Peter Eisentraut
Date:
Tom Lane writes:

> Come to think of it, it would probably make sense to adapt the existing
> notion of "location" (cf initlocation script) into something meaning
> "directory that users are allowed to create tablespaces (including
> databases) in".

This is what I've been trying to push all along. But note that this
mechanism does allow multiple databases per location. :)


-- 
Peter Eisentraut                  Sernanders väg 10:115
peter_e@gmx.net                   75262 Uppsala
http://yi.org/peter-e/            Sweden