Re: Migrate postgres to newer hardware - Mailing list pgsql-admin
| From | Iñigo Martinez Lasala |
|---|---|
| Subject | Re: Migrate postgres to newer hardware |
| Date | |
| Msg-id | 1270195138.7497.9.camel@deimos Whole thread Raw |
| In response to | Re: Migrate postgres to newer hardware (Renato Oliveira <renato.oliveira@grant.co.uk>) |
| Responses |
Re: Migrate postgres to newer hardware
Re: Migrate postgres to newer hardware |
| List | pgsql-admin |
I've never worked with Bucardo and Londiste, so I cannot give you any hint about it.
With slony it's not necessary to restore all database prior staring sync process. You only have to restore your schema but not data. Slony will keep them synced with source server.
Anyway, if you plan to migrate to 8.4 there are some issues with slony. For 8.3 and higher, you will go with slony 2.X. For 8.2 with slony 1.X
I'm not sure if Slony 1.x is compatible with Postgres >= 8.3, but I'm sure slony 2.x is not backwards compatible.
So, prior thinking about migrating to 8.4 you will have to check this. And more important, it's possible you have to slightly change your database schema to work in postgres 8.4, specially if you use tsearch2.
If using slony, I wouldn't upgrade to 8.4 before a deep analysis of changes in your existing database.
-----Original Message-----
From: Renato Oliveira <renato.oliveira@grant.co.uk>
To: Iñigo Martinez Lasala <imartinez@vectorsf.com>
Cc: pgsql-admin <pgsql-admin@postgresql.org>
Subject: RE: [ADMIN] Migrate postgres to newer hardware
Date: Wed, 31 Mar 2010 09:11:37 +0100
Hi Iñigo,
Thank you for your input, really appreciated.
I just had a thought; if I backup ‘pg_dump’ full database, then restore to my new machine new postgres 8.4, which one of these programs would work best to do the migration,
Slony, Bucardo or Londiste?
I would like to say that we did have slony and I was not impressed, it fell behind and could not catch up and caused a very high load on the system.
Also they way its philosophy works, it is very high maintenance, and the idea of creating tables, and triggers on both dbs...
It might be, it was not setup properly or well, we removed it as the db was screaming for some fresh air.
Ps I would like to point out that I am systems administrator and not a dba, so you can understand sometimes my questions...
I think Bucardo seems best for the task, for what I have read so far, but I do not know.
Thank you very much and I am sorry for this
Renato
Renato Oliveira
Systems Administrator
e-mail: renato.oliveira@grant.co.uk Tel: +44 (0)1763 260811
Fax: +44 (0)1763 262410
www.grant.co.uk Grant Instruments (Cambridge) Ltd
Company registered in England, registration number 658133
Registered office address:
29 Station Road,
Shepreth,
CAMBS SG8 6GB
UK
From: Iñigo Martinez Lasala [mailto:imartinez@vectorsf.com]
Sent: 30 March 2010 16:27
To: Renato Oliveira
Cc: pgsql-admin
Subject: RE: [ADMIN] Migrate postgres to newer hardware
Yes, you only have that two possibilities, I think.
PITR is not an option. I tested the same, from 7.4 32bit to 7.4 64bit and didn't work. Later, when I asked here, I was told why not.
The problem with slony is that you have to manually create tables in destination database and all database model (procedures, triggers, sequences, views, etc). If your application creates new tables, you will have to deal with this prior starting migration, or at least disable the creation of new tables.
Slony is asynchronous, so you will have to ensure that all changes have been committed to new database before changing your applications or exchanging IP addresses.
Slony also add many triggers and special tables to both databases (master and slave). So, after migration, you will have to delete them. It's not difficult but don't forget to do it.
By the way, are you sure your database is 160GB? Including indexes? There are strategies in order to perform a faster pg_restore...
For example, if you migrate your database schema but don't create indexes, then migrate data and finally create pending indexes restore will be faster. With pg 8.4 restore is very fast, so it will take less time that export.
Anyway, if you cannot leave database down for a day, I think slony will be your best bet, although it's not exempt of problems. :)
-----Original Message-----
From: Renato Oliveira <renato.oliveira@grant.co.uk>
To: Iñigo Martinez Lasala <imartinez@vectorsf.com>
Subject: RE: [ADMIN] Migrate postgres to newer hardware
Date: Tue, 30 Mar 2010 15:47:27 +0100
Hey Iñigo,
Thank you very much for your reply.
I would love to do just that, but unfortunately I can’t it is not as simple as that.
I would love if the application had been built in with this in mind…
To give you an idea; the pg_dump takes 15 hours and I attempted a restore yesterday and it took 14 hours and 21 min.
It would not be viable for us and specially I cannot have the system down more than maximum 30 min without the risk of losing data and customers not having alerts.
I don’t think I will be able to use PITR to migrate to new servers specially if it is 64 bit and to migrate to another 32 bit is no gain, as we need more memory.
As far as can gather there are only two ways:
a) Slony type
b) Pg_dump
Is that correct ? Do you guys have any other ways?
Renato
Renato Oliveira
Systems Administrator
e-mail: renato.oliveira@grant.co.uk Tel: +44 (0)1763 260811
Fax: +44 (0)1763 262410
www.grant.co.uk Grant Instruments (Cambridge) Ltd
Company registered in England, registration number 658133
Registered office address:
29 Station Road,
Shepreth,
CAMBS SG8 6GB
UK
From: Iñigo Martinez Lasala [mailto:imartinez@vectorsf.com]
Sent:30 March 2010 15:29
To:Renato Oliveira
Cc:pgsql-admin
Subject: Re: [ADMIN] Migrate postgres to newer hardware
Hi Renato.
I would follow the ancient method: perform a pg_dump / pg_restore
Yes, you will have to take offline database for a long period.
And yes, it would be a great moment to perform a 8.4 upgrade. Performance is far superior, restore is far faster...
... and yes, it could give you many problems if you don't perform many test in order to address all queries without explicit type conversions before real migration, but I think it's the best moment to deal with a very convenient upgrade.
We have performed this upgrade last week with a gforge (with only 25GB database) and having also to upgrade to new tsearch2 and everything is running smooth.
-----Original Message-----
From: Renato Oliveira <renato.oliveira@grant.co.uk>
To: pgsql-admin@postgresql.org <pgsql-admin@postgresql.org>
Subject: [ADMIN] Migrate postgres to newer hardware
Date: Tue, 30 Mar 2010 12:18:36 +0100
Dear All,
What would be the easiest and fastest way to migrate Postgres 8.2.24 32 BIT to a new server 64 Bit.
The existing server runs on 32 bit architecture and has a database as big as 160GB.
We initially thought of using PITR, but as one of the PITR requirements is both machines need to be similar.
This similarity needs to be even in architecture? I think I read something which says “Yes”.
If we cannot use PITR what would be the best approach, we can’t have down time I am afraid.
Any ideas or suggestions would be very welcome.
Thank you very much
Best regards
Renato
Renato Oliveira
Systems Administrator
e-mail: renato.oliveira@grant.co.uk Tel: +44 (0)1763 260811
Fax: +44 (0)1763 262410
www.grant.co.uk Grant Instruments (Cambridge) Ltd
Company registered in England, registration number 658133
Registered office address:
29 Station Road,
Shepreth,
CAMBS SG8 6GB
UK
P Please consider the environment before printing this email
CONFIDENTIALITY: The information in this e-mail and any attachments is confidential. It is intended only for the named recipients(s). If you are not the named recipient please notify the sender immediately and do not disclose the contents to another person or take copies.
VIRUSES: The contents of this e-mail or attachment(s) may contain viruses which could damage your own computer system. Whilst Grant Instruments (Cambridge) Ltd has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should therefore carry out your own virus checks before opening the attachment(s).
OpenXML: For information about the OpenXML file format in use within Grant Instruments please visit our website
P Please consider the environment before printing this email
CONFIDENTIALITY: The information in this e-mail and any attachments is confidential. It is intended only for the named recipients(s). If you are not the named recipient please notify the sender immediately and do not disclose the contents to another person or take copies.
VIRUSES: The contents of this e-mail or attachment(s) may contain viruses which could damage your own computer system. Whilst Grant Instruments (Cambridge) Ltd has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should therefore carry out your own virus checks before opening the attachment(s).
OpenXML: For information about the OpenXML file format in use within Grant Instruments please visit our website
P Please consider the environment before printing this email
CONFIDENTIALITY: The information in this e-mail and any attachments is confidential. It is intended only for the named recipients(s). If you are not the named recipient please notify the sender immediately and do not disclose the contents to another person or take copies.
VIRUSES: The contents of this e-mail or attachment(s) may contain viruses which could damage your own computer system. Whilst Grant Instruments (Cambridge) Ltd has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should therefore carry out your own virus checks before opening the attachment(s).
OpenXML: For information about the OpenXML file format in use within Grant Instruments please visit our website
With slony it's not necessary to restore all database prior staring sync process. You only have to restore your schema but not data. Slony will keep them synced with source server.
Anyway, if you plan to migrate to 8.4 there are some issues with slony. For 8.3 and higher, you will go with slony 2.X. For 8.2 with slony 1.X
I'm not sure if Slony 1.x is compatible with Postgres >= 8.3, but I'm sure slony 2.x is not backwards compatible.
So, prior thinking about migrating to 8.4 you will have to check this. And more important, it's possible you have to slightly change your database schema to work in postgres 8.4, specially if you use tsearch2.
If using slony, I wouldn't upgrade to 8.4 before a deep analysis of changes in your existing database.
-----Original Message-----
From: Renato Oliveira <renato.oliveira@grant.co.uk>
To: Iñigo Martinez Lasala <imartinez@vectorsf.com>
Cc: pgsql-admin <pgsql-admin@postgresql.org>
Subject: RE: [ADMIN] Migrate postgres to newer hardware
Date: Wed, 31 Mar 2010 09:11:37 +0100
Hi Iñigo,
Thank you for your input, really appreciated.
I just had a thought; if I backup ‘pg_dump’ full database, then restore to my new machine new postgres 8.4, which one of these programs would work best to do the migration,
Slony, Bucardo or Londiste?
I would like to say that we did have slony and I was not impressed, it fell behind and could not catch up and caused a very high load on the system.
Also they way its philosophy works, it is very high maintenance, and the idea of creating tables, and triggers on both dbs...
It might be, it was not setup properly or well, we removed it as the db was screaming for some fresh air.
Ps I would like to point out that I am systems administrator and not a dba, so you can understand sometimes my questions...
I think Bucardo seems best for the task, for what I have read so far, but I do not know.
Thank you very much and I am sorry for this
Renato
Renato Oliveira
Systems Administrator
e-mail: renato.oliveira@grant.co.uk Tel: +44 (0)1763 260811
Fax: +44 (0)1763 262410
www.grant.co.uk Grant Instruments (Cambridge) Ltd
Company registered in England, registration number 658133
Registered office address:
29 Station Road,
Shepreth,
CAMBS SG8 6GB
UK
From: Iñigo Martinez Lasala [mailto:imartinez@vectorsf.com]
Sent: 30 March 2010 16:27
To: Renato Oliveira
Cc: pgsql-admin
Subject: RE: [ADMIN] Migrate postgres to newer hardware
Yes, you only have that two possibilities, I think.
PITR is not an option. I tested the same, from 7.4 32bit to 7.4 64bit and didn't work. Later, when I asked here, I was told why not.
The problem with slony is that you have to manually create tables in destination database and all database model (procedures, triggers, sequences, views, etc). If your application creates new tables, you will have to deal with this prior starting migration, or at least disable the creation of new tables.
Slony is asynchronous, so you will have to ensure that all changes have been committed to new database before changing your applications or exchanging IP addresses.
Slony also add many triggers and special tables to both databases (master and slave). So, after migration, you will have to delete them. It's not difficult but don't forget to do it.
By the way, are you sure your database is 160GB? Including indexes? There are strategies in order to perform a faster pg_restore...
For example, if you migrate your database schema but don't create indexes, then migrate data and finally create pending indexes restore will be faster. With pg 8.4 restore is very fast, so it will take less time that export.
Anyway, if you cannot leave database down for a day, I think slony will be your best bet, although it's not exempt of problems. :)
-----Original Message-----
From: Renato Oliveira <renato.oliveira@grant.co.uk>
To: Iñigo Martinez Lasala <imartinez@vectorsf.com>
Subject: RE: [ADMIN] Migrate postgres to newer hardware
Date: Tue, 30 Mar 2010 15:47:27 +0100
Hey Iñigo,
Thank you very much for your reply.
I would love to do just that, but unfortunately I can’t it is not as simple as that.
I would love if the application had been built in with this in mind…
To give you an idea; the pg_dump takes 15 hours and I attempted a restore yesterday and it took 14 hours and 21 min.
It would not be viable for us and specially I cannot have the system down more than maximum 30 min without the risk of losing data and customers not having alerts.
I don’t think I will be able to use PITR to migrate to new servers specially if it is 64 bit and to migrate to another 32 bit is no gain, as we need more memory.
As far as can gather there are only two ways:
a) Slony type
b) Pg_dump
Is that correct ? Do you guys have any other ways?
Renato
Renato Oliveira
Systems Administrator
e-mail: renato.oliveira@grant.co.uk Tel: +44 (0)1763 260811
Fax: +44 (0)1763 262410
www.grant.co.uk Grant Instruments (Cambridge) Ltd
Company registered in England, registration number 658133
Registered office address:
29 Station Road,
Shepreth,
CAMBS SG8 6GB
UK
From: Iñigo Martinez Lasala [mailto:imartinez@vectorsf.com]
Sent:30 March 2010 15:29
To:Renato Oliveira
Cc:pgsql-admin
Subject: Re: [ADMIN] Migrate postgres to newer hardware
Hi Renato.
I would follow the ancient method: perform a pg_dump / pg_restore
Yes, you will have to take offline database for a long period.
And yes, it would be a great moment to perform a 8.4 upgrade. Performance is far superior, restore is far faster...
... and yes, it could give you many problems if you don't perform many test in order to address all queries without explicit type conversions before real migration, but I think it's the best moment to deal with a very convenient upgrade.
We have performed this upgrade last week with a gforge (with only 25GB database) and having also to upgrade to new tsearch2 and everything is running smooth.
-----Original Message-----
From: Renato Oliveira <renato.oliveira@grant.co.uk>
To: pgsql-admin@postgresql.org <pgsql-admin@postgresql.org>
Subject: [ADMIN] Migrate postgres to newer hardware
Date: Tue, 30 Mar 2010 12:18:36 +0100
Dear All,
What would be the easiest and fastest way to migrate Postgres 8.2.24 32 BIT to a new server 64 Bit.
The existing server runs on 32 bit architecture and has a database as big as 160GB.
We initially thought of using PITR, but as one of the PITR requirements is both machines need to be similar.
This similarity needs to be even in architecture? I think I read something which says “Yes”.
If we cannot use PITR what would be the best approach, we can’t have down time I am afraid.
Any ideas or suggestions would be very welcome.
Thank you very much
Best regards
Renato
Renato Oliveira
Systems Administrator
e-mail: renato.oliveira@grant.co.uk Tel: +44 (0)1763 260811
Fax: +44 (0)1763 262410
www.grant.co.uk Grant Instruments (Cambridge) Ltd
Company registered in England, registration number 658133
Registered office address:
29 Station Road,
Shepreth,
CAMBS SG8 6GB
UK
P Please consider the environment before printing this email
CONFIDENTIALITY: The information in this e-mail and any attachments is confidential. It is intended only for the named recipients(s). If you are not the named recipient please notify the sender immediately and do not disclose the contents to another person or take copies.
VIRUSES: The contents of this e-mail or attachment(s) may contain viruses which could damage your own computer system. Whilst Grant Instruments (Cambridge) Ltd has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should therefore carry out your own virus checks before opening the attachment(s).
OpenXML: For information about the OpenXML file format in use within Grant Instruments please visit our website
P Please consider the environment before printing this email
CONFIDENTIALITY: The information in this e-mail and any attachments is confidential. It is intended only for the named recipients(s). If you are not the named recipient please notify the sender immediately and do not disclose the contents to another person or take copies.
VIRUSES: The contents of this e-mail or attachment(s) may contain viruses which could damage your own computer system. Whilst Grant Instruments (Cambridge) Ltd has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should therefore carry out your own virus checks before opening the attachment(s).
OpenXML: For information about the OpenXML file format in use within Grant Instruments please visit our website
P Please consider the environment before printing this email
CONFIDENTIALITY: The information in this e-mail and any attachments is confidential. It is intended only for the named recipients(s). If you are not the named recipient please notify the sender immediately and do not disclose the contents to another person or take copies.
VIRUSES: The contents of this e-mail or attachment(s) may contain viruses which could damage your own computer system. Whilst Grant Instruments (Cambridge) Ltd has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should therefore carry out your own virus checks before opening the attachment(s).
OpenXML: For information about the OpenXML file format in use within Grant Instruments please visit our website
pgsql-admin by date: