Thread: Introducing the TPC-V benchmark, and its relationship to PostgreSQL
Hello PostgreSQL fans,
I would like to introduce myself and the TPC-V benchmark to the PostgreSQL community. I would then like to ask the community to help us make the TPC-V reference benchmarking kit a success, and establish PostgreSQL as a common DBMS used in measuring the performance of enterprise servers.
I am VMware’s rep to the TPC, and chair the TPC’s virtualization benchmark development subcommittee. For those of you who don’t know the TPC, it is an industry standards consortium, and its benchmarks are the main performance tests for enterprise-class database servers. For external (marketing) use, these benchmarks are the gold standard of comparing different servers, processors, databases, etc. For internal use, they are typically the biggest hammers an organization can use for performance stress testing of their products. TPC benchmarks are one of the workloads (if not the main workload) that processor vendors use to design their products. So the benchmarks are in much heavier use internal to companies than there are official disclosures.
TPC-V is a new benchmark under development for virtualized databases. A TPC-V configuration has:
- multiple virtual machines running a mix of DSS, OLTP, and business logic apps
- VMs running with throughputs ranging from 10% to 40% of the total system
- load elasticity emulating cloud characteristic: The benchmark maintains a constant overall tpsV load level, but the proportion directed to each VM changes every 10 minutes
A paper in the TPC Technical Conference track of VLDB 2010 described the initial motivation and architecture of TPC-V. A paper that has been accepted to the TPC TC track of VLDB 2012 describes in detail the current status of the benchmark.
All TPC results up to now have been on commercial databases. The majority of active results are on Oracle or Microsoft SQL Server, followed by DB2, Sybase, and other players. Again, keep in mind that these benchmarks aren’t meant to only compare DBMS products. In fact the majority of results are “sponsored” by server hardware companies. The server hardware, processor, storage, OS, etc. all contribute to the performance. But you can’t have a database server benchmark results without a good DBMS!
And that’s where PostgreSQL comes in. The TPC-V development subcommittee followed the usual path of TPC benchmarks by writing a functional specification, and looking to TPC members to develop benchmarking kits to implement the spec. TPC-V uses the schema and transactions of TPC-E, but the transaction mixes and the way the benchmark is run it totally new and virtualization-specific. We chose to start from TPC-E to accelerate the benchmark development phase: the specification would be easier to write, and DBMS vendors could create TPC-V kits starting from their existing TPC-E kits. Until now, benchmarking kits for various TPC benchmarks have been typically developed by DBMS vendors, and offered to their partners for internal testing or disclosures. So our expectation was that one or more DBMS companies that owned existing TPC-E benchmarking kits would allocate resources to modify their kits to execute the TPC-V transactions, and supply kits to subcommittee members for prototyping. This did not happen (let’s not get into the internal politics of the TPC!!), so the subcommittee moved forward with developing its own reference kit. The reference kit has been developed to run on PostgreSQL, and we are focusing our development efforts and testing on PostgreSQL.
The reference kit will be a first for the TPC, which until now has only published paper functional specifications. This kit will be publically available to anyone who wants to run TPC-V, whether for internal testing, academic studies, or official publications. Commercial DBMS vendors are allowed to develop their own kits and publish with them. Even if commercial DBMS vendors decide later on to develop TPC-V kits, we expect official TPC-V publications with this reference kit using PostgreSQL, and of course a lot of academic use of the kit. I think this will be a boost for the PostgreSQL community (correct me if I am wrong!!).
The most frequent question to the TPC is “do you offer a kit to run one of your benchmarks?”. There will finally be such a kit, and it will run on PGSQL.
But TPC benchmarks is where the big boys play. If we want the reference kit to be credible, it has to have good performance. We don’t expect it to beat the commercial databases, but it has to be in the ballpark. We have started our work running the kit in a simple, single-VM, TPC-E type configuration since TPC-E is a known animal with official publications available. We have compared our performance to Microsoft SQL results published on a similar platform. After waving our hands through a number of small differences between the platforms, we have calculated a CPU cost of around 3.2ms/transaction for the published MS SQL results, versus a measurement of 8.6ms/transaction for PostgreSQL. (TPC benchmarks are typically pushed to full CPU utilization. One removes all bottlenecks in storage, networking, etc., to achieve the 100% CPU usage. So CPU cost/tran is the final decider of performance.) So we need to cut the CPU cost of transactions in half to make publications with PostgreSQL comparable to commercial databases. It is OK to be slower than MS SQL or Oracle. The benchmark running PostgreSQL can still be used to compare the performance of servers, processors, and especially, hypervisors under a demanding database workload. But the slower we are, the less credible we are.
Sorry for the long post. I will follow up with specific questions next.
Thanks,
Reza Taheri
On 07/04/2012 07:08 AM, Reza Taheri wrote: > ... so the subcommittee moved forward with developing its own > reference kit. The reference kit has been developed to run on > PostgreSQL, and we are focusing our development efforts and testing on > PostgreSQL. That's a very positive step. The TPC seems to me to have a pretty poor reputation among open source database users and vendors. I think that's largely because the schema and tools are typically very closed and restrictively licensed, though the prohibition against publishing benchmarks by big commercial vendors doesn't help. This sounds like a promising change. The TPC benchmarks are really good for load-testing and regression testing, so having one that's directly PostgreSQL friendly will be a big plus, especially if it is appropriately licensed. The opportunity to audit the schema, queries, and test setup before the tool is finalized would certainly be appealing. What can you publish in draft form now? What license terms does the TPC plan to release the schema, queries, and data for TPC-V under? I've cc'd Greg Smith and Dave Page, both of whom I suspect will be interested in this development but could easily miss your message. If you haven't read Greg' book "PostgreSQL High Performance" it's probably a good idea to do so. -- Craig Ringer
Thanks for reply, Craig. As far as publishing a draft, we are planning to do something along those lines. For the schema and the queries, we are pretty much taking those wholesale from TPC-E, whose specification is public (http://www.tpc.org/tpce/spec/v1.12.0/TPCE-v1.12.0.pdf).The high-level differences with TPC-E are detailed in the 2010 and2012 TPC TC papers I mentioned. We will stick closely to the TPC-E schema and queries. Anything new means a long specificationwriting process, which we are trying to avoid. We want to get this benchmark out there quickly. I am not an expert in licensing. What I can tell you is that the kit will be available to anyone to download and use witha simple EULA based on existing TPC EULAs (although TPC hasn't had a complete end-to-end kit before, it has publishedpartial code modules for its benchmarks). We broached the idea of open sourcing the kit, but it didn't pan out.The people on the subcommittee represent their companies, and different companies have different rules when their employeescontribute to open source code. Satisfying the armies of lawyers would have been impossible. So the kit won't beopen source, but readily available for use. It will probably be similar to the licensing for SPEC benchmarks if you arefamiliar with them. I'll pick up Greg's book. We had been focusing on functionality, but our focus will shift to performance soon. To be blunt,the team is very experienced in benchmarks and in database performance, but most of us are new to PGSQL. Thanks, Reza > -----Original Message----- > From: Craig Ringer [mailto:ringerc@ringerc.id.au] > Sent: Tuesday, July 03, 2012 10:19 PM > To: pgsql-performance@postgresql.org > Cc: Reza Taheri; Andy Bond (abond@redhat.com); Greg Kopczynski; Jignesh > Shah; Greg Smith; Dave Page > Subject: Re: [PERFORM] Introducing the TPC-V benchmark, and its > relationship to PostgreSQL > > On 07/04/2012 07:08 AM, Reza Taheri wrote: > > > ... so the subcommittee moved forward with developing its own > > reference kit. The reference kit has been developed to run on > > PostgreSQL, and we are focusing our development efforts and testing on > > PostgreSQL. > That's a very positive step. The TPC seems to me to have a pretty poor > reputation among open source database users and vendors. I think that's > largely because the schema and tools are typically very closed and > restrictively licensed, though the prohibition against publishing benchmarks > by big commercial vendors doesn't help. > > This sounds like a promising change. The TPC benchmarks are really good for > load-testing and regression testing, so having one that's directly PostgreSQL > friendly will be a big plus, especially if it is appropriately licensed. > > The opportunity to audit the schema, queries, and test setup before the > tool is finalized would certainly be appealing. What can you publish in draft > form now? > > What license terms does the TPC plan to release the schema, queries, and > data for TPC-V under? > > I've cc'd Greg Smith and Dave Page, both of whom I suspect will be > interested in this development but could easily miss your message. If you > haven't read Greg' book "PostgreSQL High Performance" it's probably a good > idea to do so. > > -- > Craig Ringer
On 05/07/12 06:24, Reza Taheri wrote: > > I'll pick up Greg's book. We had been focusing on functionality, but our focus will shift to performance soon. To be blunt,the team is very experienced in benchmarks and in database performance, but most of us are new to PGSQL. > > The book is well worth a read - even if you are experienced (in fact you may get more out of it in that case). In addition I think it would be beneficial for you to share with us your non default tuning changes you made in postgresql.conf etc (as others have requested) - one of the major benefits of an open source community is the input of many minds...however we need the basic information concerning setup etc to even begin to help. regards Mark
On 07/03/2012 07:08 PM, Reza Taheri wrote: > TPC-V is a new benchmark under development for virtualized databases. A > TPC-V configuration has: > > - multiple virtual machines running a mix of DSS, OLTP, and business > logic apps > > - VMs running with throughputs ranging from 10% to 40% of the total system > .. I think this would be a lot more interesting to the traditional, dedicated hardware part of the PostgreSQL community if there was a clear way to run this with only a single active machine too. If it's possible for us to use this to compare instances of PostgreSQL on dedicated hardware, too, that is enormously more valuable to people with larger installations. It might be helpful to VMWare as well. Being able to say "this VM install gets X% of the performance of a bare-metal install" answers a question I get asked all the time--when people want to decide between dedicated and virtual setups. The PostgreSQL community could use a benchmark like this for its own performance regression testing too. A lot of that work is going to happen on a dedicated machines. > After waving our hands through a number > of small differences between the platforms, we have calculated a CPU > cost of around 3.2ms/transaction for the published MS SQL results, > versus a measurement of 8.6ms/transaction for PostgreSQL. (TPC > benchmarks are typically pushed to full CPU utilization. One removes all > bottlenecks in storage, networking, etc., to achieve the 100% CPU usage. > So CPU cost/tran is the final decider of performance.) So we need to cut > the CPU cost of transactions in half to make publications with > PostgreSQL comparable to commercial databases. I appreciate that getting close to parity here is valuable. This situation is so synthetic though--removing other bottlenecks and looking at CPU timing--that it's hard to get too excited about optimizing for it. There's a lot of things in PostgreSQL that we know are slower than commercial databases because they're optimized for flexibility (the way operators are implemented is the best example) or for long-term code maintenance. Microsoft doesn't care if they have terribly ugly code that runs faster, because no one sees that code. PostgreSQL does care. The measure that's more fair is a system cost based ones. What I've found is that a fair number of people note PostgreSQL's low-level code isn't quite as fast as some of the less flexible alternatives--hard coding operators is surely cheaper than looking them up each time--but the license cost savings more than pays for bigger hardware to offset that. I wish I had any customer whose database was CPU bound, that would be an awesome world to live in. Anyway, guessing at causes here is premature speculation. When there's some code for the test kit published, at that point discussing the particulars of why it's not running well will get interesting. -- Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
Hi Greg, Yes, a single-instance benchmark is a natural fall-out from the TPC-V kit. Our coding team (4 people working directly onthe benchmark with another 3-4 folks helping in various consulting capacities) is tasked with creating a multi-VM benchmark.The benchmark is still missing the critical Market Exchange Emulator function. Once that's done, it would be naturalfor someone else in the TPC to take our working prototype and simplify it for a single-system, TPC-E (not V) referencekit. The conversion is not technically difficult, but releasing kits is a new path for the TPC, and will take somework. Cheers, Reza > -----Original Message----- > From: Greg Smith [mailto:greg@2ndQuadrant.com] > Sent: Thursday, July 05, 2012 6:25 PM > To: Reza Taheri > Cc: pgsql-performance@postgresql.org; Andy Bond (abond@redhat.com); > Greg Kopczynski; Jignesh Shah > Subject: Re: [PERFORM] Introducing the TPC-V benchmark, and its > relationship to PostgreSQL > > On 07/03/2012 07:08 PM, Reza Taheri wrote: > > TPC-V is a new benchmark under development for virtualized databases. > > A TPC-V configuration has: > > > > - multiple virtual machines running a mix of DSS, OLTP, and business > > logic apps > > > > - VMs running with throughputs ranging from 10% to 40% of the total > > system .. > > I think this would be a lot more interesting to the traditional, dedicated > hardware part of the PostgreSQL community if there was a clear way to run > this with only a single active machine too. If it's possible for us to use this to > compare instances of PostgreSQL on dedicated hardware, too, that is > enormously more valuable to people with larger installations. It might be > helpful to VMWare as well. Being able to say "this VM install gets X% of the > performance of a bare-metal install" > answers a question I get asked all the time--when people want to decide > between dedicated and virtual setups. > > The PostgreSQL community could use a benchmark like this for its own > performance regression testing too. A lot of that work is going to happen on > a dedicated machines. > > > After waving our hands through a number > > of small differences between the platforms, we have calculated a CPU > > cost of around 3.2ms/transaction for the published MS SQL results, > > versus a measurement of 8.6ms/transaction for PostgreSQL. (TPC > > benchmarks are typically pushed to full CPU utilization. One removes > > all bottlenecks in storage, networking, etc., to achieve the 100% CPU > usage. > > So CPU cost/tran is the final decider of performance.) So we need to > > cut the CPU cost of transactions in half to make publications with > > PostgreSQL comparable to commercial databases. > > I appreciate that getting close to parity here is valuable. This situation is so > synthetic though--removing other bottlenecks and looking at CPU timing-- > that it's hard to get too excited about optimizing for it. There's a lot of > things in PostgreSQL that we know are slower than commercial databases > because they're optimized for flexibility (the way operators are > implemented is the best example) or for long-term code maintenance. > Microsoft doesn't care if they have terribly ugly code that runs faster, > because no one sees that code. PostgreSQL does care. > > The measure that's more fair is a system cost based ones. What I've found is > that a fair number of people note PostgreSQL's low-level code isn't quite as > fast as some of the less flexible alternatives--hard coding operators is surely > cheaper than looking them up each time--but the license cost savings more > than pays for bigger hardware to offset that. I wish I had any customer > whose database was CPU bound, that would be an awesome world to live > in. > > Anyway, guessing at causes here is premature speculation. When there's > some code for the test kit published, at that point discussing the particulars > of why it's not running well will get interesting. > > -- > Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD > PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
On 07/06/2012 11:22 AM, Reza Taheri wrote: > Hi Greg, > Yes, a single-instance benchmark is a natural fall-out from the TPC-V kit. Our coding team (4 people working directly onthe benchmark with another 3-4 folks helping in various consulting capacities) is tasked with creating a multi-VM benchmark.The benchmark is still missing the critical Market Exchange Emulator function. Once that's done, it would be naturalfor someone else in the TPC to take our working prototype and simplify it for a single-system, TPC-E (not V) referencekit. The conversion is not technically difficult, but releasing kits is a new path for the TPC, and will take somework. Please consider releasing sample versions early in the process, especially as such releases are new to the TPC. Giving others the opportunity to contribute different skill sets and experiences before everything is locked in to the final configuration is important, especially when trying new things. -- Craig Ringer
Yes, I hear you. TPC's usual mode of operation has been to release details after the benchmark is complete. But TPC doeshave a policy clause that allows publication of draft specifications to get public feedback before the benchmark is complete.Our 2012 TPC TC paper will have a lot of the high level details. We need to see if we can use the draft clause toalso release beta versions of code. Thanks, Reza > -----Original Message----- > From: Craig Ringer [mailto:ringerc@ringerc.id.au] > Sent: Thursday, July 05, 2012 8:41 PM > To: Reza Taheri > Cc: Greg Smith; pgsql-performance@postgresql.org > Subject: Re: [PERFORM] Introducing the TPC-V benchmark, and its > relationship to PostgreSQL > > On 07/06/2012 11:22 AM, Reza Taheri wrote: > > Hi Greg, > > Yes, a single-instance benchmark is a natural fall-out from the TPC-V kit. > Our coding team (4 people working directly on the benchmark with another > 3-4 folks helping in various consulting capacities) is tasked with creating a > multi-VM benchmark. The benchmark is still missing the critical Market > Exchange Emulator function. Once that's done, it would be natural for > someone else in the TPC to take our working prototype and simplify it for a > single-system, TPC-E (not V) reference kit. The conversion is not technically > difficult, but releasing kits is a new path for the TPC, and will take some > work. > Please consider releasing sample versions early in the process, especially as > such releases are new to the TPC. Giving others the opportunity to > contribute different skill sets and experiences before everything is locked > in to the final configuration is important, especially when trying new things. > > -- > Craig Ringer