Re: Largest PG database known to man! - Mailing list pgsql-general

From Stephen Frost
Subject Re: Largest PG database known to man!
Date
Msg-id 20131002015323.GI2706@tamriel.snowman.net
Whole thread Raw
In response to Re: Largest PG database known to man!  (John R Pierce <pierce@hogranch.com>)
Responses Re: Largest PG database known to man!
List pgsql-general
* John R Pierce (pierce@hogranch.com) wrote:
> if we assume the tables average 1KB/record (which is a fairly large
> record size even including indexing), you're looking at 400 billion
> records.   if you can populate these at 5000 records/second, it
> would take 2.5 years of 24/7 operation to populate that.

5000 1KB records per second is only 5MB/s or so, which is really quite
slow..  I can't imagine that they'd load all of this data by doing a
commit for each record and you could load a *huge* amount of data *very*
quickly, in parallel, by using either unlogged tables or wal_level =
minimal and creating the tables in the same transaction that's loading
them.

> this sort of big data system is probably more suitable for something
> like hadoop+mongo or whatever on a cloud of 1000 nodes, not a
> monolithic SQL relational database.

Or a federated PG database using FDWs..

Sadly, I've not personally worked with a data system on the 100+TB range
w/ PG (we do have a Hadoop environment along that scale) but I've built
systems as large as 25TB which, built correctly, work very well.  Still,
I don't think I'd recommend building a single-image PG database on that
scale but rather would shard it.

    Thanks,

        Stephen

Attachment

pgsql-general by date:

Previous
From: Jaime Casanova
Date:
Subject: Re: Postgres replication question :- One master 2 slaves 9.0.10
Next
From: John R Pierce
Date:
Subject: Re: Largest PG database known to man!