Thread: An article about Etsy... and a migration from Postgres to MySQL
"When "clever" goes wrong: how Etsy overcame poor architectural choices" http://arstechnica.com/business/news/2011/10/when-clever-goes-wrong-how-etsy-overcame-poor-architectural-choices.ars -- Milen A. Radev
On Tue, Oct 4, 2011 at 10:18 AM, Milen A. Radev <milen@radev.net> wrote: > "When "clever" goes wrong: how Etsy overcame poor architectural choices" > > http://arstechnica.com/business/news/2011/10/when-clever-goes-wrong-how-etsy-overcame-poor-architectural-choices.ars > This is very well written. It almost sounds like the problem was caused by Postgres. I take it as first evidence of a marketing department at the new MySQL team. Hats off, guys and happy list monitoring. I think we should keep the quote on our website to remind people... "Etsy is the sum total of its data, so when it came to choosing where to store our critical information there was only one sound choice. PostgreSQL's strong emphasis on quality, stability, and data integrity contribute to making it the premier open source database." -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: An article about Etsy... and a migration from Postgres to MySQL
From
PostgreSQL - Hans-Jürgen Schönig
Date:
On Oct 4, 2011, at 3:10 PM, Simon Riggs wrote: > On Tue, Oct 4, 2011 at 10:18 AM, Milen A. Radev <milen@radev.net> wrote: > >> "When "clever" goes wrong: how Etsy overcame poor architectural choices" >> >> http://arstechnica.com/business/news/2011/10/when-clever-goes-wrong-how-etsy-overcame-poor-architectural-choices.ars >> > > This is very well written. It almost sounds like the problem was > caused by Postgres. > i got similar feeling when i have read this one ... why should a switch to a system which cannot even properly do stored procs magically solve the problem? it is like blaming the owner of a wine-yard for being drunk all day long ... > I take it as first evidence of a marketing department at the new MySQL > team. Hats off, guys and happy list monitoring. yes, same feelings here as well ... regards, hans -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de
Hi! On Tue, Oct 4, 2011 at 6:10 AM, Simon Riggs <simon@2ndquadrant.com> wrote: > > On Tue, Oct 4, 2011 at 10:18 AM, Milen A. Radev <milen@radev.net> wrote: > > > "When "clever" goes wrong: how Etsy overcame poor architectural choices" > > > > http://arstechnica.com/business/news/2011/10/when-clever-goes-wrong-how-etsy-overcame-poor-architectural-choices.ars > > > > This is very well written. It almost sounds like the problem was > caused by Postgres. I didn't read it that way. It seemed like a cautionary tale about overuse of stored procedures to express business logic in a company that had serious scaling and operational and communication problems between DBAs, sysadmins and developers. The new-hires turned to a technology stack that was well understood. And in the end, the article mentions that they still have Postgres at the core. I see this as a wake up call that our advocacy needs to focus on the case studies, like that of Urban Airship, to demonstrate how to scale infrastructure with Postgres. Keeping this information either secret or difficult to find results in throwing out or scaling back use of Postgres. -selena -- http://chesnok.com
Re: An article about Etsy... and a migration from Postgres to MySQL
From
"Jehan-Guillaume (ioguix) de Rorthais"
Date:
On 04/10/2011 17:12, Selena Deckelmann wrote: > Hi! > > On Tue, Oct 4, 2011 at 6:10 AM, Simon Riggs <simon@2ndquadrant.com> wrote: >> >> On Tue, Oct 4, 2011 at 10:18 AM, Milen A. Radev <milen@radev.net> wrote: >> >>> "When "clever" goes wrong: how Etsy overcame poor architectural choices" >>> >>> http://arstechnica.com/business/news/2011/10/when-clever-goes-wrong-how-etsy-overcame-poor-architectural-choices.ars >>> >> >> This is very well written. It almost sounds like the problem was >> caused by Postgres. > > I didn't read it that way. It seemed like a cautionary tale about > overuse of stored procedures to express business logic in a company > that had serious scaling and operational and communication problems > between DBAs, sysadmins and developers. The new-hires turned to a > technology stack that was well understood. And in the end, the article > mentions that they still have Postgres at the core. That was my understanding as well. The new tech chief came from Yahoo and built with his team an architecture they knew from their Flickr experience...that is based on mysql. In my opinion, it really means that they fail both in their choice technically and humanly. > I see this as a wake up call that our advocacy needs to focus on the > case studies, like that of Urban Airship, to demonstrate how to scale > infrastructure with Postgres. Keeping this information either secret > or difficult to find results in throwing out or scaling back use of > Postgres. +1 > -selena > -- > http://chesnok.com
On Tue, Oct 4, 2011 at 8:12 AM, Selena Deckelmann <selena@chesnok.com> wrote: > I didn't read it that way. It seemed like a cautionary tale about > overuse of stored procedures to express business logic in a company > that had serious scaling and operational and communication problems > between DBAs, sysadmins and developers. The new-hires turned to a > technology stack that was well understood. And in the end, the article > mentions that they still have Postgres at the core. That's close to how I read it. However there have to be some missing pieces because in general, my experience is that ORM's often write bad queries and that complex operations are often easier to tune in stored procs than in ORMs. Could the issue have to do with caching in the middleware? > > I see this as a wake up call that our advocacy needs to focus on the > case studies, like that of Urban Airship, to demonstrate how to scale > infrastructure with Postgres. Keeping this information either secret > or difficult to find results in throwing out or scaling back use of > Postgres. Agreed. Best Wishes, Chris Travers
> I see this as a wake up call that our advocacy needs to focus on the > case studies, like that of Urban Airship, to demonstrate how to scale > infrastructure with Postgres. Keeping this information either secret > or difficult to find results in throwing out or scaling back use of > Postgres. Hey, Ned Lilly - are you on this list? Do you have any examples of highly scaled xTuple installations? (For those who are unaware, xTuple is an open source ERP solution based on a Qt frontend. All of the business logic resides in Postgres stored procedures.) http://www.xtuple.org Cheers, -JK
On Tue, Oct 4, 2011 at 9:42 AM, Joshua Kramer <josh@globalherald.net> wrote: > >> I see this as a wake up call that our advocacy needs to focus on the >> case studies, like that of Urban Airship, to demonstrate how to scale >> infrastructure with Postgres. Keeping this information either secret >> or difficult to find results in throwing out or scaling back use of >> Postgres. > > Hey, Ned Lilly - are you on this list? Do you have any examples of highly > scaled xTuple installations? (For those who are unaware, xTuple is an open > source ERP solution based on a Qt frontend. All of the business logic > resides in Postgres stored procedures.) > I can't speak for xTuple, but for LedgerSMB, we have at least one user who processes payment runs of 5k invoices at a time via Perl/CGI and PostgreSQL (total of 20k invoices per week, and expecting this number to rise) with the main logic handled in stored procedures. If it was just a matter of selection, we could run that selection in a few seconds but a lot of info has to be written to the db so that states don't change between web requests. Consequently it takes only a bit longer than that. The real bottleneck is actually the CGI scripts which generate HTML forms representing 5000 invoices....... Keep in mind this is being run against a database with a million invoices in it, and probably 10 million invoice lines, and a portion of the lines must be summed up to ensure the invoice is fully paid off. We have found PostgreSQL performs very well. In general, I have found that tuning stored procedures is easier than tuning ORM-generated queries, which is why I am convinced there must be more to the story than we are being told. Best Wishes, Chris Travers
(Forgot to forward to the list) > I can't speak for xTuple, but for LedgerSMB, we have at least one user > who processes payment runs of 5k invoices at a time via Perl/CGI and > PostgreSQL (total of 20k invoices per week, and expecting this number > to rise) with the main logic handled in stored procedures. If it was > just a matter of selection, we could run that selection in a few > seconds but a lot of info has to be written to the db so that states > don't change between web requests. Consequently it takes only a bit > longer than that. The real bottleneck is actually the CGI scripts > which generate HTML forms representing 5000 invoices....... > Sorry, with the persistent, discretionary locking info written it takes only a bit over 10x the pure read-only performance. This could probably be optimized further for disk I/O, but the web interface is a much bigger bottleneck. Best Wishes, Chris Travers
Simon Riggs wrote: > On Tue, Oct 4, 2011 at 10:18 AM, Milen A. Radev <milen@radev.net> wrote: > >> "When "clever" goes wrong: how Etsy overcame poor architectural choices" >> >> http://arstechnica.com/business/news/2011/10/when-clever-goes-wrong-how-etsy-overcame-poor-architectural-choices.ars >> > > This is very well written. It almost sounds like the problem was > caused by Postgres. > > I take it as first evidence of a marketing department at the new MySQL > team. Hats off, guys and happy list monitoring. > > I think we should keep the quote on our website to remind people... > > "Etsy is the sum total of its data, so when it came to choosing where > to store our critical information there was only one sound choice. > PostgreSQL's strong emphasis on quality, stability, and data integrity > contribute to making it the premier open source database." > I don't see that quote in the article. I also saw comments talking about the article being rewritten. -- Darren Duncan
Hi Josh, We've got installations with hundreds of concurrent heads-down users doing very transaction-intensive things, with very high volumes of human-entered Sales Orders (thousands a day), machine-managed EDI (thousands of invoices, shipping notices, purchase orders, etc.), and tens of thousands of complex general ledger transactions a day, including serial or lot-controlled inventory movement across multiple locations. Some users have millions of SKUs, hundreds of thousands of contacts and accounts (customers, vendors, etc) in the CRM subsystem, etc. To the best of my knowledge, we have never had performance problems caused by PostgreSQL or our use of stored procedures to handle the business logic. When we do have issues, they are easily addressed by adding indexes or fixing inefficiences in what client-side code we do still have. Happily, living in the open source world, those issues are typically brought to our attention (and solved) rather quickly :) Cheers, Ned On 10/4/2011 12:42 PM, Joshua Kramer wrote: > >> I see this as a wake up call that our advocacy needs to focus on the >> case studies, like that of Urban Airship, to demonstrate how to scale >> infrastructure with Postgres. Keeping this information either secret >> or difficult to find results in throwing out or scaling back use of >> Postgres. > > Hey, Ned Lilly - are you on this list? Do you have any examples of > highly scaled xTuple installations? (For those who are unaware, xTuple > is an open source ERP solution based on a Qt frontend. All of the > business logic resides in Postgres stored procedures.) > > http://www.xtuple.org > > Cheers, > -JK > -- Ned Lilly President and CEO xTuple 119 West York Street Norfolk, VA 23510 tel. 757.461.3022 x101 email: ned@xtuple.com www.xtuple.com