Thread: Postgres performance on Veritas VxVM
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
I'm running Postgres 8.3.7 on Solaris 10 x64, using VxFS/VxVM 5.0 MP3.
I'm trying to determine the best storage configuration for my workload,
so I ran some tests of Postgres performance with various combinations of
VM and filesystem. What I noticed was that when all other factors were
the same, running on VxVM was nearly twice as slow as running on a plain
disk.
My configuration was a VxFS filesystem mounted at /sql, 'noatime,cio',
and another mounted at /sql/pg_xlog,
'noatime,cio,mincache=direct,convosync=direct'. This forced direct I/O
for the WAL. Without VxVM, these filesystems were on plain disk slices.
With VxVM, I added the single disk to a diskgroup and created two
volumes on it for the filesystems, layout=concat, with the default
options.
The backend storage is a 14-disk RAID10 using an internal RAID card (Sun
StorageTek STK-RAID-INT), with 256MB battery-backed write cache. The
system has 32GB RAM and nothing apart from Postgres was running on it
during the test. The same postgresql.conf was used for both tests; I
configured 256MB of shared_buffers and 20 checkpoint_segments.
The benchmark I'm using is a database import tool called "osm2pgsql",
with a subset of our actual workload. Without VxVM, the import
completed in 2 hours, 34 minutes. With VxVM, it took 5 hours 7 minutes.
Has anyone else tested Postgres on VxVM and noticed such a large
performance hit?
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (HP-UX)
iEYEARECAAYFAksWK6kACgkQIXd7fCuc5vIJowCfUVLjDQ7R34n4QIls4Uenahoq
lNMAn3CO7zr+HIwirKZMl3YRtiFdMaY+
=/kND
-----END PGP SIGNATURE-----
River Tarnell wrote: > My configuration was a VxFS filesystem mounted at /sql, 'noatime,cio', > and another mounted at /sql/pg_xlog, > 'noatime,cio,mincache=direct,convosync=direct'. This forced direct I/O > for the WAL. Without VxVM, these filesystems were on plain disk slices. > With VxVM, I added the single disk to a diskgroup and created two > volumes on it for the filesystems, layout=concat, with the default > options. > That convosync trick only does what you want here if you also change wal_sync_method=open_sync (or open_data_sync). As it is, you're still calling fsync all the time despite what you did with the mount options, and you're not getting direct writes because you're not writing synchronously to trigger the conversion. What you should do is the following: postgresql.conf: wal_sync_method = open_datasync /sql/pg_xlog: 'noatime,cio,mincache=direct,convosync=direct That should work quite well. See http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm for a lot more details about the various options for VxFS tuning of WAL writes. Note that I only recommend open_sync for VxFS there because that will work on Linux too. Since you're on Solaris, you should be able to get the smaller open_datasync writes and some improvements from using direct writes too stack on top of one another. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Greg Smith:
> What you should do is the following:
> postgresql.conf: wal_sync_method = open_datasync
> /sql/pg_xlog: 'noatime,cio,mincache=direct,convosync=direct
> That should work quite well.
Thanks. That reduced the import time to about 4hr 30 minutes. However,
this is still much slower than using VxFS with plain disks. Any other
ideas?
I'm now running a test using VxFS on SVM soft partitions to see if that
improves performance at all (but I'd much rather have the flexibility of
VxVM).
- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (HP-UX)
iEYEARECAAYFAksWlf0ACgkQIXd7fCuc5vLDQQCghI3Dm8/GRuDVOucrYlQoq5jS
sk0AoLR0SJyt+aK7k1F+iTEQ/t3Qto+U
=y3ed
-----END PGP SIGNATURE-----
River Tarnell wrote: > I'm now running a test using VxFS on SVM soft partitions to see if that > improves performance at all (but I'd much rather have the flexibility of > VxVM). > Flexibility is often expensive from a performance point of view. We regularly tell people here that they have to avoid using Linux's LVM for similar reasons--while it shouldn't be so slow, it is. Nothing you can do about it but use direct disk partitions instead if you need the performance to be good. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Greg Smith: > Flexibility is often expensive from a performance point of view. We > regularly tell people here that they have to avoid using Linux's LVM for > similar reasons--while it shouldn't be so slow, it is. Nothing you can > do about it but use direct disk partitions instead if you need the > performance to be good. Okay, that makes sense. What about using plain slices for the WAL, but using the VM for the data? For example, we have 14 disks, so I could allocate 2 for the log in RAID1 (146GB, which is more than enough), then use the remaining 12 under VxVM for the data. If I understand right, the critical factor is the WAL write speed; the VM is easily able to keep up with writes to the data files, since those are mostly asynchronous. Does this seems like a reasonable solution? (I'll benchmark this configuration anyway, but I'd be interested in any comments.) Thanks, River. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAksWwi0ACgkQIXd7fCuc5vKxzQCeMB0ECbxedXIcQ+YEhFcuUJzc 7egAn0zbzed5VL/E8UPFReZDhl50LTuK =NvX6 -----END PGP SIGNATURE-----