Thread: O_DIRECT and performance

O_DIRECT and performance

From
Doug McNaught
Date:
Well, O_DIRECT has finally made it into the Linux kernel.  It lets you 
open a file in such a way that reads and writes don't go to the buffer 
cache but straight to the disk.  Accesses must be aligned on
filesystem block boundaries.

Is there any case where PG would benefit from this?  I can see it
reducing memory pressure and duplication of data between PG shared
buffers and the block buffer cache.  OTOH, it does require that writes 
be batched up for decent performance, since each write has an implicit 
fsync() involved (just as with O_SYNC).

Anyone played with this on systems that already support it (Solaris?)

-Doug
-- 
In a world of steel-eyed death, and men who are fighting to be warm,
Come in, she said, I'll give you shelter from the storm.    -Dylan


Re: O_DIRECT and performance

From
Bruce Momjian
Date:
> Well, O_DIRECT has finally made it into the Linux kernel.  It lets you 
> open a file in such a way that reads and writes don't go to the buffer 
> cache but straight to the disk.  Accesses must be aligned on
> filesystem block boundaries.
> 
> Is there any case where PG would benefit from this?  I can see it
> reducing memory pressure and duplication of data between PG shared
> buffers and the block buffer cache.  OTOH, it does require that writes 
> be batched up for decent performance, since each write has an implicit 
> fsync() involved (just as with O_SYNC).
> 
> Anyone played with this on systems that already support it (Solaris?)

I have heard there are many cases there O_DIRECT on Solaris is slower
for databases than normal I/O.  I think bulk copy was faster but not
normal operation.  Probably not something we are going to get into soon.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


7.2b2 "make check" failure on Red Hat Linux 7.2

From
teg@redhat.com (Trond Eivind Glomsrød)
Date:
Red Hat Linux 7.2, postgresql 7.2b2 configured like this:

./configure --enable-syslog --prefix=/usr --sysconfdir=/etc --with-pam
--with-openssl --with-unixodbc --with-tcl --with-perl --with-python
--enable-locale --enable-recode --enable-multibyte --enable-nls

"make check" fails on initdb:

============== creating temporary installation        ==============
============== initializing database system           ==============

pg_regress: initdb failed
Examine ./log/initdb.log for the reason.


teg@halden postgresql-7.2b2]$ cat src/test/regress/log/initdb.log 
Running with noclean mode on. Mistakes will not be cleaned up.
/home/devel/teg/postgresql-7.2b2/src/test/regress/./tmp_check/install//usr/bin/pg_encoding: relocation error:
/home/devel/teg/postgresql-7.2b2/src/test/regress/./tmp_check/install//usr/bin/pg_encoding:undefined symbol:
pg_valid_server_encoding
initdb: pg_encoding failed

Perhaps you did not configure PostgreSQL for multibyte support or
the program was not successfully installed.
[teg@halden postgresql-7.2b2]$

-- 
Trond Eivind Glomsrød
Red Hat, Inc.


Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
teg@redhat.com (Trond Eivind Glomsrød)
Date:
teg@redhat.com (Trond Eivind Glomsrød) writes:

> Red Hat Linux 7.2, postgresql 7.2b2 configured like this:
> 
> ./configure --enable-syslog --prefix=/usr --sysconfdir=/etc --with-pam
> --with-openssl --with-unixodbc --with-tcl --with-perl --with-python
> --enable-locale --enable-recode --enable-multibyte --enable-nls

When just using "configure" it works - apart from a false failure in
the regression tests (geometry), it seems to work (a couple cases of
"-0.121320343559642" vs. "-0.121320343559643" - a better way than raw
diff for comparing would be useful).

-- 
Trond Eivind Glomsrød
Red Hat, Inc.


Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
Tom Lane
Date:
teg@redhat.com (Trond Eivind Glomsrød) writes:
> /home/devel/teg/postgresql-7.2b2/src/test/regress/./tmp_check/install//usr/bin/pg_encoding: relocation error:
/home/devel/teg/postgresql-7.2b2/src/test/regress/./tmp_check/install//usr/bin/pg_encoding:undefined symbol:
pg_valid_server_encoding
> initdb: pg_encoding failed

pg_encoding relies on libpq to supply the pg_valid_server_encoding()
subroutine, but that subroutine is only compiled into libpq in a
MULTIBYTE build.  I speculate that your executable was picking up
a non-MULTIBYTE libpq shared library from someplace.  Check ldconfig
and all that stuff...

FWIW, both multibyte and non-multibyte builds are working OK for me
in current sources.
        regards, tom lane


Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
teg@redhat.com (Trond Eivind Glomsrød)
Date:
Tom Lane <tgl@sss.pgh.pa.us> writes:

> teg@redhat.com (Trond Eivind Glomsrød) writes:
> > /home/devel/teg/postgresql-7.2b2/src/test/regress/./tmp_check/install//usr/bin/pg_encoding: relocation error:
/home/devel/teg/postgresql-7.2b2/src/test/regress/./tmp_check/install//usr/bin/pg_encoding:undefined symbol:
pg_valid_server_encoding
> > initdb: pg_encoding failed
> 
> pg_encoding relies on libpq to supply the pg_valid_server_encoding()
> subroutine, but that subroutine is only compiled into libpq in a
> MULTIBYTE build.  I speculate that your executable was picking up
> a non-MULTIBYTE libpq shared library from someplace.  Check ldconfig
> and all that stuff...

I have an existing installation of 7.1 on the system, that's why I did
"make check" in the build directory.

"--prefix=/usr" seems to be the "culprit" - without it, it regression
tests run just fine. 

-- 
Trond Eivind Glomsrød
Red Hat, Inc.


Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
teg@redhat.com (Trond Eivind Glomsrød)
Date:
Tom Lane <tgl@sss.pgh.pa.us> writes:

> teg@redhat.com (Trond Eivind Glomsrød) writes:
> > Tom Lane <tgl@sss.pgh.pa.us> writes:
> >> I speculate that your executable was picking up
> >> a non-MULTIBYTE libpq shared library from someplace.  Check ldconfig
> >> and all that stuff...
> 
> > I have an existing installation of 7.1 on the system, that's why I did
> > "make check" in the build directory.
> 
> > "--prefix=/usr" seems to be the "culprit" - without it, it regression
> > tests run just fine. 
> 
> The pg_regress script sets LD_LIBRARY_PATH to try to cause libpq and
> the other shlibs to be picked up from the temp installation tree.
> Perhaps this is wrong or insufficient on your platform.  It certainly
> sounds like the dynamic linker is choosing the installed libpq over
> the one that we want it to use.  Any thoughts on fixing that?

Since it works when prefix isn't /usr, I'd guess that the build
process sets rpath. This takes precedence over LD_LIBRARY_PATH.

Fix? Don't use rpath - it's evil, and should be avoided anyway.

ld(1) contains some info on how libraries are resolved.
-- 
Trond Eivind Glomsrød
Red Hat, Inc.


Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
Tom Lane
Date:
teg@redhat.com (Trond Eivind Glomsrød) writes:
> Tom Lane <tgl@sss.pgh.pa.us> writes:
>> I speculate that your executable was picking up
>> a non-MULTIBYTE libpq shared library from someplace.  Check ldconfig
>> and all that stuff...

> I have an existing installation of 7.1 on the system, that's why I did
> "make check" in the build directory.

> "--prefix=/usr" seems to be the "culprit" - without it, it regression
> tests run just fine. 

The pg_regress script sets LD_LIBRARY_PATH to try to cause libpq and
the other shlibs to be picked up from the temp installation tree.
Perhaps this is wrong or insufficient on your platform.  It certainly
sounds like the dynamic linker is choosing the installed libpq over
the one that we want it to use.  Any thoughts on fixing that?
        regards, tom lane


Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
Tom Lane
Date:
teg@redhat.com (Trond Eivind Glomsrød) writes:
> Since it works when prefix isn't /usr, I'd guess that the build
> process sets rpath. This takes precedence over LD_LIBRARY_PATH.

> Fix? Don't use rpath - it's evil, and should be avoided anyway.

Peter, any thoughts on that?  It's not clear to me that removing
rpath settings would be an improvement overall, even if it makes
"make check" more reliable.
        regards, tom lane


Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
Larry Rosenman
Date:
* Tom Lane <tgl@sss.pgh.pa.us> [011116 16:56]:
> teg@redhat.com (Trond Eivind Glomsrød) writes:
> > Since it works when prefix isn't /usr, I'd guess that the build
> > process sets rpath. This takes precedence over LD_LIBRARY_PATH.
> 
> > Fix? Don't use rpath - it's evil, and should be avoided anyway.
> 
> Peter, any thoughts on that?  It's not clear to me that removing
> rpath settings would be an improvement overall, even if it makes
> "make check" more reliable.
My general take is rpath is a *GOOD* thing, as it prevents USERS from
not being able to find REQUIRED shared libs. 

I vote for it staying.

LER

> 
>             regards, tom lane
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
> 
> http://archives.postgresql.org

-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749


Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
Peter Eisentraut
Date:
Trond Eivind Glomsrød writes:

> Fix? Don't use rpath - it's evil, and should be avoided anyway.

configure --disable-rpath is your friend.

Overall, rpath gives us much more benefit than the occasional trouble it
creates.

-- 
Peter Eisentraut   peter_e@gmx.net



Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
Trond Eivind Glomsrød
Date:
On Sun, 18 Nov 2001, Peter Eisentraut wrote:

> Trond Eivind Glomsrød writes:
> 
> > Fix? Don't use rpath - it's evil, and should be avoided anyway.
> 
> configure --disable-rpath is your friend.

Didn't know about that one - it will certainly be put to use :)

> Overall, rpath gives us much more benefit than the occasional trouble it
> creates.

When you install (not configure for, just install) into a separate tree 
(for easier packaging), it's a hole which can be exploited - some packages 
will rpath into /var/tmp/<foo>, for  instance. Hackers can then put their 
own library there. One big offender here is perl's automatic module 
creation script which will change the rpaths from what you told it when 
you built it to what you do when you install it.

-- 
Trond Eivind Glomsrød
Red Hat, Inc.



Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
Peter Eisentraut
Date:
Trond Eivind Glomsrød writes:

> When you install (not configure for, just install) into a separate tree
> (for easier packaging), it's a hole which can be exploited - some packages
> will rpath into /var/tmp/<foo>, for  instance. Hackers can then put their
> own library there.

"Some packages"... ;-)

> One big offender here is perl's automatic module
> creation script which will change the rpaths from what you told it when
> you built it to what you do when you install it.

This should be fixed now, although the perl module will actually not obey
the --disable-rpath switch.  Can't have everything, I guess...

-- 
Peter Eisentraut   peter_e@gmx.net



Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
teg@redhat.com (Trond Eivind Glomsrød)
Date:
Larry Rosenman <ler@lerctr.org> writes:

> * Tom Lane <tgl@sss.pgh.pa.us> [011116 16:56]:
> > teg@redhat.com (Trond Eivind Glomsrød) writes:
> > > Since it works when prefix isn't /usr, I'd guess that the build
> > > process sets rpath. This takes precedence over LD_LIBRARY_PATH.
> > 
> > > Fix? Don't use rpath - it's evil, and should be avoided anyway.
> > 
> > Peter, any thoughts on that?  It's not clear to me that removing
> > rpath settings would be an improvement overall, even if it makes
> > "make check" more reliable.
> My general take is rpath is a *GOOD* thing, as it prevents USERS from
> not being able to find REQUIRED shared libs. 

Lib-directories should be listed in /etc/ld.so.conf, and no rpath used
- at least on linux.

-- 
Trond Eivind Glomsrød
Red Hat, Inc.


Re: 7.2b2 "make check" failure on Red Hat Linux 7.2

From
Larry Rosenman
Date:
* Trond Eivind Glomsr?d <teg@redhat.com> [011119 10:47]:
> Larry Rosenman <ler@lerctr.org> writes:
> 
> > * Tom Lane <tgl@sss.pgh.pa.us> [011116 16:56]:
> > > teg@redhat.com (Trond Eivind Glomsrød) writes:
> > > > Since it works when prefix isn't /usr, I'd guess that the build
> > > > process sets rpath. This takes precedence over LD_LIBRARY_PATH.
> > > 
> > > > Fix? Don't use rpath - it's evil, and should be avoided anyway.
> > > 
> > > Peter, any thoughts on that?  It's not clear to me that removing
> > > rpath settings would be an improvement overall, even if it makes
> > > "make check" more reliable.
> > My general take is rpath is a *GOOD* thing, as it prevents USERS from
> > not being able to find REQUIRED shared libs. 
> 
> Lib-directories should be listed in /etc/ld.so.conf, and no rpath used
> - at least on linux.
no such animal on OpenUNIX 8 (In native "unix" mode).


> 
> -- 
> Trond Eivind Glomsrød
> Red Hat, Inc.

-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 972-414-9812                 E-Mail: ler@lerctr.org
US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749