Thread: Simple database, multiple instances?

I have simple database schema, containing just three tables:

samples, drones, drones_history.

Now, those tables hold data for the drones for a simulation. Each
simulation dataset will grow to around 10 GB in around 6 months.

Since the data is not related in any way I was thinking in separating
each simulation into it's own database. That way it would be much easier
for me to, at later date, move some of the databases to other servers
(when dataset grows beyond the original server storage capacity limit).

But. At this time I have around 600 simulations, that would mean
creating 600 databases, and in future there could very well be around
5000 simulations. Is postgres going to have 'issues' with that large
number of databases?

Or do I model my system in a way that each database holds around 100
simulations?

    Mario

Mario Splivalo <mario.splivalo@megafon.hr> writes:
> I have simple database schema, containing just three tables:
>
> samples, drones, drones_history.
>
> Now, those tables hold data for the drones for a simulation. Each simulation
> dataset will grow to around 10 GB in around 6 months.
>
> Since the data is not related in any way I was thinking in separating each
> simulation into it's own database. That way it would be much easier for me
> to, at later date, move some of the databases to other servers (when dataset
> grows beyond the original server storage capacity limit).

Do you intend to run queries across multiple simulations at once? If
yes, you want to avoid multi databases. Other than that, I'd go with a
naming convention like samples_<simulation id> and maybe some
inheritance to ease querying multiple simulations.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

On 11/30/2010 12:45 PM, Dimitri Fontaine wrote:
> Mario Splivalo<mario.splivalo@megafon.hr>  writes:
>> I have simple database schema, containing just three tables:
>>
>> samples, drones, drones_history.
>>
>> Now, those tables hold data for the drones for a simulation. Each simulation
>> dataset will grow to around 10 GB in around 6 months.
>>
>> Since the data is not related in any way I was thinking in separating each
>> simulation into it's own database. That way it would be much easier for me
>> to, at later date, move some of the databases to other servers (when dataset
>> grows beyond the original server storage capacity limit).
>
> Do you intend to run queries across multiple simulations at once? If
> yes, you want to avoid multi databases. Other than that, I'd go with a
> naming convention like samples_<simulation id>  and maybe some
> inheritance to ease querying multiple simulations.

Nope, those 'realms' are completely separated, I'll just have hundreds
of them. But each of them is in it's separate 'universe', they're not
aware of each other in any way (i might be creating some statistics, but
that is going to be really rarely).

    Mario

I saw a presentation from Heroku where they discussed using a similar
paradigm, and they ran into trouble once they hit a couple thousand
databases. If memory serves, this was on an older version of
PostgreSQL and may not be relevant with 9.0 (or even 8.4?), but you
may want to try to track down one of their developers (maybe through
their mailing lists or forums?) and check.
---
Maciek Sakrejda | System Architect | Truviso

1065 E. Hillsdale Blvd., Suite 215
Foster City, CA 94404
(650) 242-3500 Main
www.truviso.com

> Having that many instances is not practical at all, so I'll have as many
> databases as I have 'realms'. I'll use pg_dump | nc  and nc | psql to
> move databases....
>
>     Mario

Then you can use schemas, too, it'll be easier.