Thread: SYSV shared memory vs mmap performance
Hi, Given the recent decision to switch from SYSV shared memory to mmap and the concerns which were made with regard to performance on *BSD kernels, I've run a few Pgbench tests on a spare Xeon box. I tested PostgreSQL-9.3 from June 28th, as of commits: - c5b3451a8e72cb7825933d4f4827f467cb38b498 (mmap) - 5d594b73d988b1ac78c49d8a84deae6bae876d01 (sysv shared memory) I also used both Scientific Linux-6.2 and DragonFly BSD-3.1; the results are in the attached PDF document. To cut a long story short, Linux doesn't show any difference and DragonFly sees some heavy degradation under load. After a while, it starts swapping and performance goes to hell. The only *BSD system tested was DragonFly but I know from previous pgbench tests FreeBSD and NetBSD follow a similar performance curve The famous kern.ipc.shm_use_phys sysctl was set to 1, which is the default setting. -- Francois Tigeot
Attachment
On Thu, Sep 13, 2012 at 08:30:03AM +0200, Francois Tigeot wrote: > Hi, > > Given the recent decision to switch from SYSV shared memory to mmap and > the concerns which were made with regard to performance on *BSD kernels, > I've run a few Pgbench tests on a spare Xeon box. > > I tested PostgreSQL-9.3 from June 28th, as of commits: > - c5b3451a8e72cb7825933d4f4827f467cb38b498 (mmap) > - 5d594b73d988b1ac78c49d8a84deae6bae876d01 (sysv shared memory) > > I also used both Scientific Linux-6.2 and DragonFly BSD-3.1; the results > are in the attached PDF document. > > To cut a long story short, Linux doesn't show any difference and DragonFly > sees some heavy degradation under load. After a while, it starts swapping > and performance goes to hell. > > The only *BSD system tested was DragonFly but I know from previous pgbench > tests FreeBSD and NetBSD follow a similar performance curve > > The famous kern.ipc.shm_use_phys sysctl was set to 1, which is the default > setting. Just a reminder we might have *BSD performance issues with our use of Posix shared memory in Postgres 9.3. I am attaching the PDF the user posted. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Attachment
On Fri, Jan 25, 2013 at 8:40 AM, Bruce Momjian <bruce@momjian.us> wrote: > Just a reminder we might have *BSD performance issues with our use of > Posix shared memory in Postgres 9.3. I am attaching the PDF the user > posted. This is a good point. The question which I believe I asked before and haven't gotten an answer to is whether there's some way to get the benefit of shm_use_phys with an anonymous mapping. It seems to me to be slightly insane to impose draconian shared memory limits out of the box and then complain when people switch to some other type of shared memory to get around them. I realize that Dragonfly may not be doing that (because I think they may have raised the default shared-memory limits), but I believe some of the more mainstream BSDs are. I suppose we could add a GUC for this, but that's not very appealing, either. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Fri, Jan 25, 2013 at 08:47:51AM -0500, Robert Haas wrote: > On Fri, Jan 25, 2013 at 8:40 AM, Bruce Momjian <bruce@momjian.us> wrote: > > Just a reminder we might have *BSD performance issues with our use of > > Posix shared memory in Postgres 9.3. I am attaching the PDF the user > > posted. > > This is a good point. The question which I believe I asked before and > haven't gotten an answer to is whether there's some way to get the > benefit of shm_use_phys with an anonymous mapping. > > It seems to me to be slightly insane to impose draconian shared memory > limits out of the box and then complain when people switch to some > other type of shared memory to get around them. I realize that > Dragonfly may not be doing that (because I think they may have raised > the default shared-memory limits), but I believe some of the more > mainstream BSDs are. > > I suppose we could add a GUC for this, but that's not very appealing, either. I think we are going to need keep an eye on this during beta. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
Hi, Robert Haas wrote: > On Fri, Jan 25, 2013 at 8:40 AM, Bruce Momjian <bruce@momjian.us> > wrote: >> Just a reminder we might have *BSD performance issues with our use >> of Posix shared memory in Postgres 9.3. I am attaching the PDF the >> user posted. > > This is a good point. The question which I believe I asked before > and haven't gotten an answer to is whether there's some way to get > the benefit of shm_use_phys with an anonymous mapping. There is. Postgres 9.3+mmap performance on DragonFly is now much better than these old benchmark results show. After the initial disappointing result, I went on a benchmarking/tuning binge with our Dear Leader Matt Dillon. Taking advantage of some previous cpu topology work from Mihai Carabas, he heavily improved most performance shortcomings we found in the DragonFly kernel. There were a few mail about this changes on the DragonFly mailing-lists and Justin Sherill wrote some interesting articles on his blog. Some links with more details about improvements and final results: http://www.shiningsilence.com/dbsdlog/2012/09/19/10403.html http://www.shiningsilence.com/dbsdlog/2012/10/11/10544.html http://www.dragonflybsd.org/performance/ > It seems to me to be slightly insane to impose draconian shared > memory limits out of the box and then complain when people switch to > some other type of shared memory to get around them. I realize that > Dragonfly may not be doing that (because I think they may have > raised the default shared-memory limits), but I believe some of the > more mainstream BSDs are. The original SYSV limits looked like something straight from the 1980s; we're now autotuning them on DragonFly. FreeBSD and NetBSD still needed manual tuning last time I had a look. -- Francois Tigeot
On Fri, Jan 25, 2013 at 9:38 AM, Francois Tigeot <ftigeot@wolfpond.org> wrote: >> On Fri, Jan 25, 2013 at 8:40 AM, Bruce Momjian <bruce@momjian.us> >> wrote: >>> >>> Just a reminder we might have *BSD performance issues with our use >>> of Posix shared memory in Postgres 9.3. I am attaching the PDF the >>> user posted. >> >> This is a good point. The question which I believe I asked before >> and haven't gotten an answer to is whether there's some way to get >> the benefit of shm_use_phys with an anonymous mapping. > > There is. Postgres 9.3+mmap performance on DragonFly is now much better than > these old benchmark results show. > > After the initial disappointing result, I went on a benchmarking/tuning > binge with our Dear Leader Matt Dillon. Taking advantage of some previous > cpu topology work from Mihai Carabas, he heavily improved most performance > shortcomings we found in the DragonFly kernel. > > There were a few mail about this changes on the DragonFly mailing-lists and > Justin Sherill wrote some interesting articles on his blog. > > Some links with more details about improvements and final results: > http://www.shiningsilence.com/dbsdlog/2012/09/19/10403.html > http://www.shiningsilence.com/dbsdlog/2012/10/11/10544.html > http://www.dragonflybsd.org/performance/ Well, that looks pretty cool. Is there anything we can sensibly do to recover the lost performance on FreeBSD and NetBSD? >> It seems to me to be slightly insane to impose draconian shared >> memory limits out of the box and then complain when people switch to >> some other type of shared memory to get around them. I realize that >> Dragonfly may not be doing that (because I think they may have >> raised the default shared-memory limits), but I believe some of the >> more mainstream BSDs are. > > The original SYSV limits looked like something straight from the 1980s; > we're now autotuning them on DragonFly. Awesome! > FreeBSD and NetBSD still needed manual tuning last time I had a look. Bummer. :-( -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Fri, Jan 25, 2013 at 05:55:19PM -0500, Robert Haas wrote: > On Fri, Jan 25, 2013 at 9:38 AM, Francois Tigeot <ftigeot@wolfpond.org> wrote: > > > > Some links with more details about improvements and final results: > > http://www.shiningsilence.com/dbsdlog/2012/09/19/10403.html > > http://www.shiningsilence.com/dbsdlog/2012/10/11/10544.html > > http://www.dragonflybsd.org/performance/ > > Well, that looks pretty cool. Is there anything we can sensibly do to > recover the lost performance on FreeBSD and NetBSD? Apart from reintroducing SYSV shm usage, I don't think the Postgres team can do much about it. Mmap really needs kernel work to perform well. Some FreeBSD people are looking into the issue and have started to run benchmarks of their own to find out what needs to be fixed; I guess they will adapt in time. I'm less optimistic on the NetBSD front: even though I reported major show-stopper bugs (system died under load and was unable to complete a pgbench run), no one seemed to care. -- Francois Tigeot
On Sat, Jan 26, 2013 at 7:00 AM, Francois Tigeot <ftigeot@wolfpond.org> wrote: > On Fri, Jan 25, 2013 at 05:55:19PM -0500, Robert Haas wrote: >> On Fri, Jan 25, 2013 at 9:38 AM, Francois Tigeot <ftigeot@wolfpond.org> wrote: >> > >> > Some links with more details about improvements and final results: >> > http://www.shiningsilence.com/dbsdlog/2012/09/19/10403.html >> > http://www.shiningsilence.com/dbsdlog/2012/10/11/10544.html >> > http://www.dragonflybsd.org/performance/ >> >> Well, that looks pretty cool. Is there anything we can sensibly do to >> recover the lost performance on FreeBSD and NetBSD? > > Apart from reintroducing SYSV shm usage, I don't think the Postgres team > can do much about it. Mmap really needs kernel work to perform well. > > Some FreeBSD people are looking into the issue and have started to run > benchmarks of their own to find out what needs to be fixed; I guess they > will adapt in time. > > I'm less optimistic on the NetBSD front: even though I reported major > show-stopper bugs (system died under load and was unable to complete > a pgbench run), no one seemed to care. Hrm. I was hoping that you were going to say "yeah, just pass the MAP_DONT_BE_STUPID flag, and it'll work great". Frankly, if the NetBSD developers don't think it's important to make the test case not crash the system, then I'm not sure there's much point is us caring about whether the test case performs well. :-( -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
hi, > I'm less optimistic on the NetBSD front: even though I reported major > show-stopper bugs (system died under load and was unable to complete > a pgbench run), no one seemed to care. can you give me a pointer? YAMAMOTO Takashi
Hi, On Mon, Jan 28, 2013 at 03:27:28PM +0000, YAMAMOTO Takashi wrote: > > can you give me a pointer? This bug report for a start: http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=46833 This is the only one I've filled; I also remember having irc discussions with some netbsd developers about the system getting unresponsive under load. -- Francois Tigeot
hi, > Hi, > > On Mon, Jan 28, 2013 at 03:27:28PM +0000, YAMAMOTO Takashi wrote: >> >> can you give me a pointer? > > This bug report for a start: > http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=46833 thanks! YAMAMOTO Takashi > > This is the only one I've filled; I also remember having irc discussions > with some netbsd developers about the system getting unresponsive under > load. > > -- > Francois Tigeot > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers