Thread: Uber moving towards MySQL

From:
Santiago Zarate
Date:

Hello guys,

Has anyone read the following article?, i'm personlally inclined to think that some of these problems could have been addressed with enough expertise.

At our company we may move to PostgreSQL for one of our core products (we use oracle), and a colleague came with this today. While this started a very heated discussion on our side... I wonder, what do you guys think

https://eng.uber.com/mysql-migration/

--
--

Santiago Zarate
Phone Panama: (+507) 831.2500 / Ext. 8456
Phone Taiwan: (+886)  819.53000 / Ext. 8456
Cell Panama: +507 64271684
Cell Taipei: +886 979 665 114
  Skype: sz-smt
  http://foursixnine.io
  http://twitter.com/foursixnine
  http://ve.linkedin.com/in/santiagozarate
 

“Dreams permit each and every one of us to be quietly and safely insane every night of our lives.” - William Dement
From:
"Joshua D. Drake"
Date:

On 07/26/2016 02:57 PM, Santiago Zarate wrote:
> Hello guys,
>
> Has anyone read the following article?, i'm personlally inclined to
> think that some of these problems could have been addressed with enough
> expertise.

It is being discussed on -hackers and -general right now.

>
> At our company we may move to PostgreSQL for one of our core products
> (we use oracle), and a colleague came with this today. While this
> started a very heated discussion on our side... I wonder, what do you
> guys think

I think they have some valid points, especially around:

Inefficient data replication
Poor replica MVCC support
Difficulty upgrading to newer releases

Whilst Berkus brought up:

Inefficient architecture for writes


They are all good points but Oracle certainly has its own list too.

Sincerely,

JD

--
Command Prompt, Inc.                  http://the.postgres.company/
                         +1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.


From:
Josh Berkus
Date:

On 07/26/2016 02:57 PM, Santiago Zarate wrote:
> Hello guys,
>
> Has anyone read the following article?, i'm personlally inclined to
> think that some of these problems could have been addressed with enough
> expertise.

Uber moved to MySQL some time ago.  This was, not coincidentally, at the
same time they got a new CTO.

Note that Uber ran the whole company off a single PostgreSQL cluster for
years.  Given that they switched a couple years back, I'm unclear on the
reasons for making this post right now.

> At our company we may move to PostgreSQL for one of our core products
> (we use oracle), and a colleague came with this today. While this
> started a very heated discussion on our side... I wonder, what do you
> guys think
>
> https://eng.uber.com/mysql-migration/

They have some valid points; see the thread on -hackers.  I like their
analysis of the index update issue.

However, note that the article doesn't address any of the shortcomings
of MySQL as a platform, or why their "schemaless" tool couldn't be
implemented on top of Postgres, or that they're not just doing MySQL,
they're doing Cassandra as well, because MySQL isn't enough by itself.
Particularly, I'll note that they were very concerned about replication
data corruption in Postgres, but don't talk about that issue at all with
MySQL.

Databases are full of tradeoffs.

--
--
Josh Berkus
Red Hat OSAS
(any opinions are my own)


From:
Robert Bernier
Date:

Folks,

https://ayende.com/blog/175137/re-why-uber-engineering-switched-from-postgres-to-mysql

This is getting serious; two popularized articles on hacker news now slamming postgres.

I've read and caught up on our mailthread regarding the technical issues.

Although no way near as knowledgeable as many of you regarding the inner workings of MySQL, let alone postgres, I do
howeverrecognize navel gazing when I see it. The first article starting this thread was a convoluted justification of
one'sown religion. This second article is pure commercial leveraging of a thread that's garnered a lot of attention. 

It's time a response from the postgres community was forthcoming.


--
Robert Bernier
   
   206-407-9298
On Tuesday, July 26, 2016 04:57:28 PM Santiago Zarate wrote:
> Hello guys,
>
> Has anyone read the following article?, i'm personlally inclined to think
> that some of these problems could have been addressed with enough
> expertise.
>
> At our company we may move to PostgreSQL for one of our core products (we
> use oracle), and a colleague came with this today. While this started a
> very heated discussion on our side... I wonder, what do you guys think
>
> https://eng.uber.com/mysql-migration/
>
>



From:
James Keener
Date:

While I love PG to death and prosthelytize as much as possible; I
don't have much experience replicating MySQL or PostgreSQL and I find
I always run a little flat in this debate, as most complaints are
about replication problems.

On Thu, Jul 28, 2016 at 9:24 AM, Robert Bernier <> wrote:
> Folks,
>
> https://ayende.com/blog/175137/re-why-uber-engineering-switched-from-postgres-to-mysql
>
> This is getting serious; two popularized articles on hacker news now slamming postgres.
>
> I've read and caught up on our mailthread regarding the technical issues.
>
> Although no way near as knowledgeable as many of you regarding the inner workings of MySQL, let alone postgres, I do
howeverrecognize navel gazing when I see it. The first article starting this thread was a convoluted justification of
one'sown religion. This second article is pure commercial leveraging of a thread that's garnered a lot of attention. 
>
> It's time a response from the postgres community was forthcoming.
>
>
> --
> Robert Bernier
>    
>    206-407-9298
> On Tuesday, July 26, 2016 04:57:28 PM Santiago Zarate wrote:
>> Hello guys,
>>
>> Has anyone read the following article?, i'm personlally inclined to think
>> that some of these problems could have been addressed with enough
>> expertise.
>>
>> At our company we may move to PostgreSQL for one of our core products (we
>> use oracle), and a colleague came with this today. While this started a
>> very heated discussion on our side... I wonder, what do you guys think
>>
>> https://eng.uber.com/mysql-migration/
>>
>>
>
>
>
> --
> Sent via pgsql-advocacy mailing list ()
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-advocacy


From:
Bruce Momjian
Date:

On Thu, Jul 28, 2016 at 06:24:09AM -0700, Robert Bernier wrote:
> Although no way near as knowledgeable as many of you regarding the
> inner workings of MySQL, let alone postgres, I do however recognize
> navel gazing when I see it. The first article starting this thread was
> a convoluted justification of one's own religion. This second article
> is pure commercial leveraging of a thread that's garnered a lot of
> attention.
>
> It's time a response from the postgres community was forthcoming.

I think this new blog post does a good job of summarizing the issues
Uber had with Postgres.  The blog comments correct some things, get some
things wrong, but eventually right.  (The thread about disk performance
can be ignored.)

In some ways any PR is good PR, and I see the general issue being that
Uber didn't like some of the technical decisions we made, and tradeoffs
we decided, but few are saying those decisions were wrong, just focused
on different workloads.

I do think we are winning when we don't act defensively but are open to
criticism --- I think it shows confidence, and I think we have a right
to be confident.

--
  Bruce Momjian  <>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +


From:
Santiago Zarate
Date:



On Tue, Jul 26, 2016 at 5:37 PM, Josh Berkus <> wrote:
On 07/26/2016 02:57 PM, Santiago Zarate wrote:
> Hello guys,
>
> Has anyone read the following article?, i'm personlally inclined to
> think that some of these problems could have been addressed with enough
> expertise.

Uber moved to MySQL some time ago.  This was, not coincidentally, at the
same time they got a new CTO.

Note that Uber ran the whole company off a single PostgreSQL cluster for
years.  Given that they switched a couple years back, I'm unclear on the
reasons for making this post right now.

Intriguing indeed, to my understanding, after reading the blog post, it looks like they're more interested in just switching rather than trying to address the problem. Also,as someone mentioned on -general the post started highly technical and promising, but slowly lost its focus.

> At our company we may move to PostgreSQL for one of our core products
> (we use oracle), and a colleague came with this today. While this
> started a very heated discussion on our side... I wonder, what do you
> guys think
>
> https://eng.uber.com/mysql-migration/

They have some valid points; see the thread on -hackers.  I like their
analysis of the index update issue.

However, note that the article doesn't address any of the shortcomings
of MySQL as a platform, or why their "schemaless" tool couldn't be
implemented on top of Postgres, or that they're not just doing MySQL,
they're doing Cassandra as well, because MySQL isn't enough by itself.
Particularly, I'll note that they were very concerned about replication
data corruption in Postgres, but don't talk about that issue at all with
MySQL.

For me looks like dead ahead they wanted to move (Maybe someone asked a MySQL guy to do a PG thing, who knows...), but the issues are there... And while it is a problem that can be addressed, it seems to be creating enough FUD to make some people consider other options (MySQL was off the table a month ago here... but is now being considered again due to this... but has low chances of succeed hopefully lol)
 

Databases are full of tradeoffs.

--
--
Josh Berkus
Red Hat OSAS
(any opinions are my own)



--
--

Santiago Zarate
Phone Panama: (+507) 831.2500 / Ext. 8456
Phone Taiwan: (+886)  819.53000 / Ext. 8456
Cell Panama: +507 64271684
Cell Taipei: +886 979 665 114
  Skype: sz-smt
  http://foursixnine.io
  http://twitter.com/foursixnine
  http://ve.linkedin.com/in/santiagozarate
 

“Dreams permit each and every one of us to be quietly and safely insane every night of our lives.” - William Dement
From:
Gavin Flower
Date:

On 29/07/16 10:12, Bruce Momjian wrote:
> On Thu, Jul 28, 2016 at 06:24:09AM -0700, Robert Bernier wrote:
>> Although no way near as knowledgeable as many of you regarding the
>> inner workings of MySQL, let alone postgres, I do however recognize
>> navel gazing when I see it. The first article starting this thread was
>> a convoluted justification of one's own religion. This second article
>> is pure commercial leveraging of a thread that's garnered a lot of
>> attention.
>>
>> It's time a response from the postgres community was forthcoming.
> I think this new blog post does a good job of summarizing the issues
> Uber had with Postgres.  The blog comments correct some things, get some
> things wrong, but eventually right.  (The thread about disk performance
> can be ignored.)
>
> In some ways any PR is good PR, and I see the general issue being that
> Uber didn't like some of the technical decisions we made, and tradeoffs
> we decided, but few are saying those decisions were wrong, just focused
> on different workloads.
>
> I do think we are winning when we don't act defensively but are open to
> criticism --- I think it shows confidence, and I think we have a right
> to be confident.
>
While admittedly I'm biased towards pg, my overall impressions from the
threads is that:

 1. The Uber use case is very different from most contexts where pg is
    used & the areas pg has been optimized for

 2. There are real problems with pg that hurt Uber

 3. The above problems are of 2 types: ones that the pg developers seek
    to address, others relate to design decisions in pg that are
    unlikely to be changed

 4. pg developers are committed to learning useful lessons from all
    this, want to clearly identify what needs to be changed and pg
    people are adjusting their focus accordingly without compromising
    other projects underway (measured response rather than a panicky
    reaction)

 5. Some of the reported problems with pg are spurious

 6. there are problems with MySQL that affect Uber, but they seem to
    have been ignored

 7. some of the technical decisions by Uber seem suspect

So overall, I agree with Bruce.


Cheers,
Gavin



From:
Michael Banck
Date:

Hi,

Am Donnerstag, den 28.07.2016, 06:24 -0700 schrieb Robert Bernier:
> Folks,
>
> https://ayende.com/blog/175137/re-why-uber-engineering-switched-from-postgres-to-mysql
>
> This is getting serious; two popularized articles on hacker news now
> slamming postgres.
>
> I've read and caught up on our mailthread regarding the technical
> issues.
>
> Although no way near as knowledgeable as many of you regarding the
> inner workings of MySQL, let alone postgres, I do however recognize
> navel gazing when I see it. The first article starting this thread was
> a convoluted justification of one's own religion. This second article
> is pure commercial leveraging of a thread that's garnered a lot of
> attention.
>
> It's time a response from the postgres community was forthcoming.

Markus Winand wrote a pretty good (and leveled, in my opinion) analysis
of Uber's choices and their trade-offs here:

http://use-the-index-luke.com/blog/2016-07-29/on-ubers-choice-of-databases

That hasn't been picked up by Hacker News yet though.


Michael

--
Michael Banck
Projektleiter / Senior Berater
Tel.: +49 2166 9901-171
Fax:  +49 2166 9901-100
Email: 

credativ GmbH, HRB Mönchengladbach 12080
USt-ID-Nummer: DE204566209
Trompeterallee 108, 41189 Mönchengladbach
Geschäftsführung: Dr. Michael Meskes, Jörg Folz, Sascha Heuer




From:
"Jonathan S. Katz"
Date:


On Jul 28, 2016, at 6:12 PM, Bruce Momjian <> wrote:

On Thu, Jul 28, 2016 at 06:24:09AM -0700, Robert Bernier wrote:
Although no way near as knowledgeable as many of you regarding the
inner workings of MySQL, let alone postgres, I do however recognize
navel gazing when I see it. The first article starting this thread was
a convoluted justification of one's own religion. This second article
is pure commercial leveraging of a thread that's garnered a lot of
attention.

It's time a response from the postgres community was forthcoming.

I think this new blog post does a good job of summarizing the issues
Uber had with Postgres.  The blog comments correct some things, get some
things wrong, but eventually right.  (The thread about disk performance
can be ignored.)

In some ways any PR is good PR, and I see the general issue being that
Uber didn't like some of the technical decisions we made, and tradeoffs
we decided, but few are saying those decisions were wrong, just focused
on different workloads.

I do think we are winning when we don't act defensively but are open to
criticism --- I think it shows confidence, and I think we have a right
to be confident.

+1 but I think it is good if we do have a response as a community through a well-read channel that shows both that we can accept constructive criticism and how we learn from it - this is good PR.

For example, when Per Se was downgraded from 4-stars to 2-stars in the NYT restaurant review, Thomas Keller, the owner, issued a statement in a very similar fashion: taking pride in his team’s work but acknowledging there is room for improvement and have a willingness to do so:


Because the response was not defensive and included a commitment to improving, it was well-received.

I suggest we craft a response in a similar nature: acknowledging there is always room to improve and understanding the different use-cases for Postgres will help the team developing it to create a system that can be freely and easily adopted and maintained.   I suggest that the piece come from -core.  I recommend we  post it in the news section on .org as it comes straight from the source and so people can easily find and link to it .

I know what I recommend is unprecedented, but the growth and visibility the PostgreSQL community has now is also unprecedented in its history and it would demonstrate a maturity and willingness to learn and improve.

Jonathan
From:
Damien Clochard
Date:

Le 30.07.2016 19:01, Michael Banck a écrit :
> Hi,
>
> Am Donnerstag, den 28.07.2016, 06:24 -0700 schrieb Robert Bernier:
>> Folks,
>>
>> https://ayende.com/blog/175137/re-why-uber-engineering-switched-from-postgres-to-mysql
>>
>> This is getting serious; two popularized articles on hacker news now
>> slamming postgres.
>>
>> I've read and caught up on our mailthread regarding the technical
>> issues.
>>
>> Although no way near as knowledgeable as many of you regarding the
>> inner workings of MySQL, let alone postgres, I do however recognize
>> navel gazing when I see it. The first article starting this thread was
>> a convoluted justification of one's own religion. This second article
>> is pure commercial leveraging of a thread that's garnered a lot of
>> attention.
>>
>> It's time a response from the postgres community was forthcoming.
>
> Markus Winand wrote a pretty good (and leveled, in my opinion) analysis
> of Uber's choices and their trade-offs here:
>
> http://use-the-index-luke.com/blog/2016-07-29/on-ubers-choice-of-databases
>
> That hasn't been picked up by Hacker News yet though.
>

Markus' article didn't make it to the HN front page heaven :
https://news.ycombinator.com/item?id=12187797

However there's another post linking directly to pgsql-hackers:
https://news.ycombinator.com/item?id=12201353

So now let's all talk here about some Hacker News comments relative to a
PostgreSQL mailing-list and get stuck in an infinite loop :)

Joke aside, the HN comments really show that the key point is not about
the arguments we can object to Uber's article, but **how** we respond to
it.

some comments among others :

« they're admitting there's a problem (which exists for this very
specific user case) and not just deflecting the criticism. I'm
impressed, I'd say a lot of projects couldn't have handled this so
nicely »

« I have a lot more respect for this than say MongoDb which claims to be
great at everything. »

« The attitude with which the article is discussed in the mailing list
is admirable. »

« Couldn't help contrasting it to some popular programming language
mailing lists, especially a CSP inspired language by a major search
company. »



To me, this whole story is a nice example of how we can transform a
supposedly bad public announcement into a positive communication.


The medium is the message ! :)


--
--
Damien Clochard


From:
Bruce Momjian
Date:

On Tue, Aug  2, 2016 at 04:47:17PM +0200, Damien Clochard wrote:
> Markus' article didn't make it to the HN front page heaven :
> https://news.ycombinator.com/item?id=12187797
>
> However there's another post linking directly to pgsql-hackers:
> https://news.ycombinator.com/item?id=12201353
>
> So now let's all talk here about some Hacker News comments relative to a
> PostgreSQL mailing-list and get stuck in an infinite loop :)
>
> Joke aside, the HN comments really show that the key point is not about the
> arguments we can object to Uber's article, but **how** we respond to it.
>
> some comments among others :
>
> « they're admitting there's a problem (which exists for this very specific
> user case) and not just deflecting the criticism. I'm impressed, I'd say a
> lot of projects couldn't have handled this so nicely »
>
> « I have a lot more respect for this than say MongoDb which claims to be
> great at everything. »
>
> « The attitude with which the article is discussed in the mailing list is
> admirable. »
>
> « Couldn't help contrasting it to some popular programming language mailing
> lists, especially a CSP inspired language by a major search company. »
>
>
>
> To me, this whole story is a nice example of how we can transform a
> supposedly bad public announcement into a positive communication.

Agreed.  I know someone wanted core to post something, but that seems to
give too much credibility to the Uber complaints.  I also didn't see how
a core email would have the nuance of these email threads.

As the HN postings states, we were a "class act", and people picked up
on that.

--
  Bruce Momjian  <>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +


From:
"Jonathan S. Katz"
Date:

> On Aug 2, 2016, at 2:12 PM, Bruce Momjian <> wrote:
>
> On Tue, Aug  2, 2016 at 04:47:17PM +0200, Damien Clochard wrote:
>> Markus' article didn't make it to the HN front page heaven :
>> https://news.ycombinator.com/item?id=12187797
>>
>> However there's another post linking directly to pgsql-hackers:
>> https://news.ycombinator.com/item?id=12201353
>>
>> So now let's all talk here about some Hacker News comments relative to a
>> PostgreSQL mailing-list and get stuck in an infinite loop :)
>>
>> Joke aside, the HN comments really show that the key point is not about the
>> arguments we can object to Uber's article, but **how** we respond to it.
>>
>> some comments among others :
>>
>> « they're admitting there's a problem (which exists for this very specific
>> user case) and not just deflecting the criticism. I'm impressed, I'd say a
>> lot of projects couldn't have handled this so nicely »
>>
>> « I have a lot more respect for this than say MongoDb which claims to be
>> great at everything. »
>>
>> « The attitude with which the article is discussed in the mailing list is
>> admirable. »
>>
>> « Couldn't help contrasting it to some popular programming language mailing
>> lists, especially a CSP inspired language by a major search company. »
>>
>>
>>
>> To me, this whole story is a nice example of how we can transform a
>> supposedly bad public announcement into a positive communication.
>
> Agreed.  I know someone wanted core to post something, but that seems to
> give too much credibility to the Uber complaints.  I also didn't see how
> a core email would have the nuance of these email threads.

So I am that someone, and would be happy to explain that further off-list, but it seems like this is off the table at
thispoint..  Primarily, It was more about having a unified community message coming from us particularly around how we
asa community encourage feedback and are willing to learn from it, and to utilize the resources we have available (e.g.
mailinglists) to air questions and concerns so we can address them. 

If the consensus at this point is to let the discussion continue it’s organic course, I’m fine with that.

Thanks,

Jonathan

From:
Simon Riggs
Date:

> So I am that someone, and would be happy to explain that further off-list, but it seems like this is off the table at
thispoint..  Primarily, It was more about having a unified community message coming from us particularly around how we
asa community encourage feedback and are willing to learn from it, and to utilize the resources we have available (e.g.
mailinglists) to air questions and concerns so we can address them. 

Maybe not a fully unified message, but I have at least tried to address this.

Thoughts on Uber's List of Postgres Limitations
http://blog.2ndquadrant.com/thoughts-on-ubers-list-of-postgres-limitations/

I do also see this as a positive thing, allowing us to get publicity
and explain how and why we do things.
Let me know if y'all would like any words or phrases edited.

I've also been working through various design improvements based upon
this feedback.

--
Simon Riggs                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


From:
Bruce Momjian
Date:

On Tue, Aug  2, 2016 at 08:11:22PM +0100, Simon Riggs wrote:
> > So I am that someone, and would be happy to explain that further off-list, but it seems like this is off the table
atthis point..  Primarily, It was more about having a unified community message coming from us particularly around how
weas a community encourage feedback and are willing to learn from it, and to utilize the resources we have available
(e.g.mailing lists) to air questions and concerns so we can address them. 
>
> Maybe not a fully unified message, but I have at least tried to address this.

Right.  My point is that when leadership makes a statement, it can be
seen as trying to keep everyone in line with a single message.  When the
groups speak organically, it is seen as more genuine, and the Hacker
News posting picked up on that.

Maybe we need a new slogan, "Postgres, a Class Act".


--
  Bruce Momjian  <>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+                     Ancient Roman grave inscription +