Thread: pgbench to the MAXINT
At one point I was working on a patch to pgbench to have it adopt 64-bit math internally even when running on 32 bit platforms, which are currently limited to a dataabase scale of ~4000 before the whole process crashes and burns. But since the range was still plenty high on a 64-bit system, I stopped working on that. People who are only running 32 bit servers at this point in time aren't doing anything serious anyway, right? So what is the upper limit now? The way it degrades when you cross it amuses me: $ pgbench -i -s 21475 pgbench creating tables... set primary key... NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "pgbench_branches_pkey" for table "pgbench_branches" NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "pgbench_tellers_pkey" for table "pgbench_tellers" NOTICE: ALTER TABLE / ADD PRIMARY KEY will create implicit index "pgbench_accounts_pkey" for table "pgbench_accounts" vacuum...done. $ pgbench -S -t 10 pgbench starting vacuum...end. setrandom: invalid maximum number -2147467296 It doesn't throw any error during the initialization step, neither via client or database logs, even though it doesn't do anything whatsoever. It just turns into the quickest pgbench init ever. That's the exact threshold, because this works: $ pgbench -i -s 21474 pgbench creating tables... 10000 tuples done. 20000 tuples done. 30000 tuples done. ... So where we're at now is that the maximum database pgbench can create is a scale of 21474. That makes approximately a 313GB database. I can tell you the size for sure when that init finishes running, which is not going to be soon. That's not quite as big as I'd like to exercise a system with 128GB of RAM, the biggest size I run into regularly now, but it's close enough for now. This limit will need to finally got pushed upward soon though, because 256GB servers are getting cheaper every day--and the current pgbench can't make a database big enough to really escape cache on one of them. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
Em 07-01-2011 22:59, Greg Smith escreveu: > setrandom: invalid maximum number -2147467296 > It is failing at atoi() circa pgbench.c:1036. But it just the first one. There are some variables and constants that need to be converted to int64 and some functions that must speak 64-bit such as getrand(). Are you working on a patch? > It doesn't throw any error during the initialization step, neither via > client or database logs, even though it doesn't do anything whatsoever. > It just turns into the quickest pgbench init ever. That's the exact > threshold, because this works: > AFAICS that is because atoi() is so fragile. > So where we're at now is that the maximum database pgbench can create is > a scale of 21474. > That's because 21475 * 100,000 > INT_MAX. We must provide an alternative to atoi() that deals with 64-bit integers. -- Euler Taveira de Oliveira http://www.timbira.com/
Euler Taveira de Oliveira wrote: > Em 07-01-2011 22:59, Greg Smith escreveu: >> setrandom: invalid maximum number -2147467296 >> > It is failing at atoi() circa pgbench.c:1036. But it just the first > one. There are some variables and constants that need to be converted > to int64 and some functions that must speak 64-bit such as getrand(). > Are you working on a patch? http://archives.postgresql.org/pgsql-hackers/2010-01/msg02868.php http://archives.postgresql.org/message-id/4C326F46.4050801@2ndquadrant.com I thought we really needed to fix that before 9.0 shipped, but it turned out the limit wasn't so bad after all on 64-bit systems; I dropped worrying about it for a while. It's starting to look like it's back on the critical list for 9.1 again though. If anyone here wanted to pick that up and help with review, I could easily update to it to current git HEAD and re-post. There's enough people on this list who do tests on large machines that I was mainly alerting to where the breaking point is at, the fix required has already been worked on a bit. Someone with more patience than I to play around with multi-platform string conversion trivia is what I think is really needed next, followed by some performance tests on 32-bit systems. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us "PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
Em 10-01-2011 05:25, Greg Smith escreveu: > Euler Taveira de Oliveira wrote: >> Em 07-01-2011 22:59, Greg Smith escreveu: >>> setrandom: invalid maximum number -2147467296 >>> >> It is failing at atoi() circa pgbench.c:1036. But it just the first >> one. There are some variables and constants that need to be converted >> to int64 and some functions that must speak 64-bit such as getrand(). >> Are you working on a patch? > > http://archives.postgresql.org/pgsql-hackers/2010-01/msg02868.php > http://archives.postgresql.org/message-id/4C326F46.4050801@2ndquadrant.com > Greg, I just improved your patch. I tried to work around the problems pointed out in the above threads. Also, I want to raise some points: (i) If we want to support and scale factor greater than 21474 we have to convert some columns to bigint; it will change the test. From the portability point it is a pity but as we have never supported it I'm not too worried about it. Why? Because it will use bigint columns only if the scale factor is greater than 21474. Is it a problem? I don't think so because generally people compare tests with the same scale factor. (ii) From the performance perspective, we need to test if the modifications don't impact performance. I don't create another code path for 64-bit modifications (it is too ugly) and I'm afraid some modifications affect the 32-bit performance. I'm in a position to test it though because I don't have a big machine ATM. Greg, could you lead these tests? (iii) I decided to copy scanint8() (called strtoint64 there) from backend (Robert suggestion [1]) because Tom pointed out that strtoll() has portability issues. I replaced atoi() with strtoint64() but didn't do any performance tests. Comments? [1] http://archives.postgresql.org/pgsql-hackers/2010-07/msg00173.php -- Euler Taveira de Oliveira http://www.timbira.com/