Thread: Postgres Backup Utility

___________________________________________________

This email is intended for the named recipient. The information contained
in it is confidential. You should not copy it for any purposes, nor
disclose its contents to any other party. If you received this email
in error, please notify the sender immediately via email, and delete
it from your computer.

Any views or opinions presented are solely those of the author and do not
necessarily represent those of the company.

PCI Compliancy: Please note, we do not send or wish to receive banking,
credit or debit card information by email or any other form of
communication.

Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive
Wigston, Leicester LE18 1AT. Tel 0116 2888000
Registered in England and Wales, Reg No 00986161
VAT GB 115 5713 87 900
__________________________________________________

___________________________________________________

This email is intended for the named recipient. The information contained
in it is confidential. You should not copy it for any purposes, nor
disclose its contents to any other party. If you received this email
in error, please notify the sender immediately via email, and delete
it from your computer.

Any views or opinions presented are solely those of the author and do not
necessarily represent those of the company.

PCI Compliancy: Please note, we do not send or wish to receive banking,
credit or debit card information by email or any other form of
communication.

Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive
Wigston, Leicester LE18 1AT. Tel 0116 2888000
Registered in England and Wales, Reg No 00986161
VAT GB 115 5713 87 900
__________________________________________________

Brad, Google for "SQL Power Architect", download it, and try a schema comparison.  That might get you a ways down to road to what you want. Bob Lunney --- On Wed, 1/19/11, French, Martin <frenchm@cromwell.co.uk> wrote: From: French, Martin <frenchm@cromwell.co.uk> Subject: Re: [ADMIN] Postgres Backup Utility To: "Bradley Holbrook" <operations_bradley@servillian.ca> Cc: pgsql-admin@postgresql.org Date: Wednesday, January 19, 2011, 2:12 AM Ok, you say that you cannot drop and recreate, so you need to do this via alter statements only? That’s obviously going to complicate matters, as a straight dump, drop, recreate, restore would be the fastest and by far simplest method.   So, Ideally, you’ll need to do a table def comparison over the two databases, and generate the necessary sql to amend the tables in test accordingly?   Querying the pg_catalog/information_schema over the two db’s should give you the table ddl from which you can diff, and then generate the alter statements from the results.   Cheers   Martin   From: Bradley Holbrook [mailto:operations_bradley@servillian.ca] Sent: 18 January 2011 16:57 To: French, Martin Cc: pgsql-admin@postgresql.org Subject: RE: [ADMIN] Postgres Backup Utility   Well, I can’t just go dropping and recreating tables… it needs to create the correct alter statements if existing tables and or functions already exist.   Secondly, when I’m finished changing the structure, I need to be able to select the list of tables that will have content updates.   Using a script might be more work maintaining then it’s worth. I have a backup utility that can do the job, but 3 tedious steps per schema, that only work about 10% of the time (and no batching options so that I can create a list of actions and run the list).     From: French, Martin [mailto:frenchm@cromwell.co.uk] Sent: January-18-11 5:47 AM To: Bradley Holbrook; pgsql-admin@postgresql.org Subject: RE: [ADMIN] Postgres Backup Utility   I’m assuming that this needs to be tightly controlled and as such a replication tool is out of the question?   In that case; The first thing to pop into my head here would be to use either use shell scripting, or to use the pg API and write a c program to handle it.   I remember doing something very similar with Oracle a few years back.   Cheers   Martin     From: pgsql-admin-owner@postgresql.org [mailto:pgsql-admin-owner@postgresql.org] On Behalf Of Bradley Holbrook Sent: 18 January 2011 00:08 To: pgsql-admin@postgresql.org Subject: [ADMIN] Postgres Backup Utility   Hello!   First day on the new mailing list as I have need of some expert’s advice.   I need to be able to quickly apply the structure updates from a development database to a testing database, and do selective data updates (like on lookup tables, but not content tables).   Any help would be appreciated!   Brad ___________________________________________________ This email is intended for the named recipient. The information contained in it is confidential. You should not copy it for any purposes, nor disclose its contents to any other party. If you received this email in error, please notify the sender immediately via email, and delete it from your computer. Any views or opinions presented are solely those of the author and do not necessarily represent those of the company. PCI Compliancy: Please note, we do not send or wish to receive banking, credit or debit card information by email or any other form of communication. Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive Wigston, Leicester LE18 1AT. Tel 0116 2888000 Registered in England and Wales, Reg No 00986161 VAT GB 115 5713 87 900 __________________________________________________ ___________________________________________________ This email is intended for the named recipient. The information contained in it is confidential. You should not copy it for any purposes, nor disclose its contents to any other party. If you received this email in error, please notify the sender immediately via email, and delete it from your computer. Any views or opinions presented are solely those of the author and do not necessarily represent those of the company. PCI Compliancy: Please note, we do not send or wish to receive banking, credit or debit card information by email or any other form of communication. Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive Wigston, Leicester LE18 1AT. Tel 0116 2888000 Registered in England and Wales, Reg No 00986161 VAT GB 115 5713 87 900 __________________________________________________

On Wed, Jan 19, 2011 at 12:12 AM, French, Martin <frenchm@cromwell.co.uk> wrote:
> Ok, you say that you cannot drop and recreate, so you need to do this via
> alter statements only? That’s obviously going to complicate matters, as a
> straight dump, drop, recreate, restore would be the fastest and by far
> simplest method.
>
>
>
> So, Ideally, you’ll need to do a table def comparison over the two
> databases, and generate the necessary sql to amend the tables in test
> accordingly?

Sorry but this is exactly backwards from good procedure.  What you do
it have your developers checkin the SQL code they used to alter the
tables / add rows in the test database, and you apply that to testing
/ QA / staging / production.  Any other way is a recipe for both
disaster and DBA burnout.

To make it easier, you can always turn on ddl logging on the
developer's databases and then just trawl the logs for what they did.
Still easier than trying to compare schemas and figure out what's
different and how to write the SQL to make it happen.

Having been a C/C++ developer many years before being a DBA, and having
written ITIL software; How is migrating structure from a Development
database to a test database whilst maintaining test data backwards?

Besides, the OP was asking how to diff to databases and create ddl, not
asking for us to comment on why he's doing it. Personally, I'd rather
not go trawling through what can only be described as hundreds of
thousands of lines of PostgreSQL log to find THE RIGHT DDL statements.
To me that's surely a recipe for disaster, and certainly not a role I
would expect to complete as a DBA. Yes the Developers should've kept the
DDL for the changes, however; I know from experience that this isn't
always the case, and have found it necessary to "Diff" databases on a
regular basis for one reason or another.

Each time I've done something like this I have resulted to coding shell
scripts, or C programs that are bespoke to accomplish this, as I've yet
to find a tool that does this to my satisfaction levels.

-----Original Message-----
From: Scott Marlowe [mailto:scott.marlowe@gmail.com]
Sent: 20 January 2011 04:41
To: French, Martin
Cc: Bradley Holbrook; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Postgres Backup Utility

On Wed, Jan 19, 2011 at 12:12 AM, French, Martin
<frenchm@cromwell.co.uk> wrote:
> Ok, you say that you cannot drop and recreate, so you need to do this
via
> alter statements only? That's obviously going to complicate matters,
as a
> straight dump, drop, recreate, restore would be the fastest and by far
> simplest method.
>
>
>
> So, Ideally, you'll need to do a table def comparison over the two
> databases, and generate the necessary sql to amend the tables in test
> accordingly?

Sorry but this is exactly backwards from good procedure.  What you do
it have your developers checkin the SQL code they used to alter the
tables / add rows in the test database, and you apply that to testing
/ QA / staging / production.  Any other way is a recipe for both
disaster and DBA burnout.

To make it easier, you can always turn on ddl logging on the
developer's databases and then just trawl the logs for what they did.
Still easier than trying to compare schemas and figure out what's
different and how to write the SQL to make it happen.

___________________________________________________

This email is intended for the named recipient. The information contained
in it is confidential.  You should not copy it for any purposes, nor
disclose its contents to any other party.  If you received this email
in error, please notify the sender immediately via email, and delete it from
your computer.

Any views or opinions presented are solely those of the author and do not
necessarily represent those of the company.

PCI Compliancy: Please note, we do not send or wish to receive banking, credit
or debit card information by email or any other form of communication.

Cromwell Tools Limited, PO Box 14, 65 Chartwell Drive
Wigston, Leicester LE18 1AT. Tel 0116 2888000
Registered in England and Wales, Reg No 00986161
VAT GB 115 5713 87 900
__________________________________________________

On Thu, Jan 20, 2011 at 5:00 AM, French, Martin <frenchm@cromwell.co.uk> wrote:
> Having been a C/C++ developer many years before being a DBA, and having
> written ITIL software; How is migrating structure from a Development
> database to a test database whilst maintaining test data backwards?

It's not.  doing by running diffs on ddl / schema is.

This is what the OP asked for:

"I need to be able to quickly apply the structure updates from a
development database to a testing database, and do selective data
updates (like on lookup tables, but not content tables)."

>
> Besides, the OP was asking how to diff to databases and create ddl, not
> asking for us to comment on why he's doing it. Personally, I'd rather

Re-read what he asked for.  He did not ask how to run diffs of schema,
he asked how to update it with changes from a test database.  The best
way to do this is to check in changes to the database and lookup
tables so the DBA can then apply them when needed to the various other
databases.

But hey, feel free to answer a question the OP didn't ask with an
answer he doesn't need.

On Thu, Jan 20, 2011 at 5:00 AM, French, Martin <frenchm@cromwell.co.uk> wrote:

>  Personally, I'd rather
> not go trawling through what can only be described as hundreds of
> thousands of lines of PostgreSQL log to find THE RIGHT DDL statements.

Oh that's easy.  Grep out the statements that start with alter etc,
ptu them all in a big email, send them to all the developers and tell
them to let you know which ones need applying to QA / Staging / etc.
Can't get your DDL to the DBA?  Your code won't work.  You break the
build because of that, you get to fix it.

On Thu, Jan 20, 2011 at 8:46 AM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
> On Thu, Jan 20, 2011 at 5:00 AM, French, Martin <frenchm@cromwell.co.uk> wrote:
>
>>  Personally, I'd rather
>> not go trawling through what can only be described as hundreds of
>> thousands of lines of PostgreSQL log to find THE RIGHT DDL statements.
>
> Oh that's easy.  Grep out the statements that start with alter etc,
> ptu them all in a big email, send them to all the developers and tell
> them to let you know which ones need applying to QA / Staging / etc.
> Can't get your DDL to the DBA?  Your code won't work.  You break the
> build because of that, you get to fix it.

Note that I also mentioned turning on logging JUST ddl statements.
Assumiing you're not logging EVERYTHING in development (and why would
you) you should only have a few dozen lines of log to grep, and little
or none of it will be things other than DDL.
--
To understand recursion, one must first understand recursion.

Thanks Scott... a couple comments.

Our developers never decide what goes to where... they just happily plumb
away on the development db until we're ready to take our product to testing
(at regular intervals), once QA is passed, we wish to apply these to live.
We have several diff tools and sync tools, but they take forever (especially
the ones that only go one schema at a time).

The DDL Logging sounds like a sufficient solution, can it be configured to
only record create and alter commands (or create or replace commands on
functions or updates on sequences, etc)? I'd likely write a script to have
this emailed to me at the end of every day. I'm going to google DDL logging
(never heard of it), but any good resources off the top of your head?

Martin French is right though, ask your developers to write down all their
SQL struct changes and they look at you funny... and being a developer
myself I'd look at me funny. If you forget just once you're screwed into a
day sifting through tables and code.



-----Original Message-----
From: Scott Marlowe [mailto:scott.marlowe@gmail.com]
Sent: January-20-11 7:40 AM
To: French, Martin
Cc: Bradley Holbrook; pgsql-admin@postgresql.org
Subject: Re: [ADMIN] Postgres Backup Utility

On Thu, Jan 20, 2011 at 5:00 AM, French, Martin <frenchm@cromwell.co.uk>
wrote:
> Having been a C/C++ developer many years before being a DBA, and
> having written ITIL software; How is migrating structure from a
> Development database to a test database whilst maintaining test data
backwards?

It's not.  doing by running diffs on ddl / schema is.

This is what the OP asked for:

"I need to be able to quickly apply the structure updates from a development
database to a testing database, and do selective data updates (like on
lookup tables, but not content tables)."

>
> Besides, the OP was asking how to diff to databases and create ddl,
> not asking for us to comment on why he's doing it. Personally, I'd
> rather

Re-read what he asked for.  He did not ask how to run diffs of schema, he
asked how to update it with changes from a test database.  The best way to
do this is to check in changes to the database and lookup tables so the DBA
can then apply them when needed to the various other databases.

But hey, feel free to answer a question the OP didn't ask with an answer he
doesn't need.

On Thu, Jan 20, 2011 at 10:42 AM, Bradley Holbrook
<operations_bradley@servillian.ca> wrote:
> Thanks Scott... a couple comments.
>
> Our developers never decide what goes to where... they just happily plumb
> away on the development db until we're ready to take our product to testing
> (at regular intervals), once QA is passed, we wish to apply these to live.
> We have several diff tools and sync tools, but they take forever (especially
> the ones that only go one schema at a time).
>
> The DDL Logging sounds like a sufficient solution, can it be configured to
> only record create and alter commands (or create or replace commands on
> functions or updates on sequences, etc)? I'd likely write a script to have
> this emailed to me at the end of every day. I'm going to google DDL logging
> (never heard of it), but any good resources off the top of your head?

It's basically logging anything that changes the structure of the
database.  It would be easy enough to grep out what you do and don't
want later.

> Martin French is right though, ask your developers to write down all their
> SQL struct changes and they look at you funny... and being a developer
> myself I'd look at me funny. If you forget just once you're screwed into a
> day sifting through tables and code.

I've worked in three different shops now as a dev-dba and sysadmin,
and in all three, all DDL changes had to be committed and / or handed
over to the DBAs.  period.  Look funny all they want, they either give
up the DDL or their code doesn't get pushed off dev servers onto
anything else.  At the very least they should be able to tell you
which tables changed to go with which code changes, or you're not sure
what code you can and can't push.  I get both of your point on this,
but it's a discipline issue that needs sorting out with the developers
if you want to have reproduceable ddl changes in all your systems that
match the code changes.

Followup, note that you can set the log_statement='ddl' for an entire
pg cluster, for a single database, or for a single user, if that
helps.  logging ddl does not log dml, or data changes, just structural
changes.

On Thu, Jan 20, 2011 at 10:52 AM, Scott Marlowe <scott.marlowe@gmail.com> wrote:
> Followup, note that you can set the log_statement='ddl' for an entire
> pg cluster, for a single database, or for a single user, if that
> helps.  logging ddl does not log dml, or data changes, just structural
> changes.

also also wik, you can use the log_line_prefix to make it so that you
can grep out what you need easier too.

#log_line_prefix = ''                   # special values:
                                        #   %u = user name
                                        #   %d = database name
                                        #   %r = remote host and port
                                        #   %h = remote host
                                        #   %p = process ID
                                        #   %t = timestamp without milliseconds
                                        #   %m = timestamp with milliseconds
                                        #   %i = command tag
                                        #   %c = session ID
                                        #   %l = session line number
                                        #   %s = session start timestamp
                                        #   %v = virtual transaction ID
                                        #   %x = transaction ID (0 if none)
                                        #   %q = stop here in non-session
                                        #        processes
                                        #   %% = '%'
                                        # e.g. '<%u%%%d> '

> -----Original Message-----
> From: Scott Marlowe [mailto:scott.marlowe@gmail.com]
> Sent: Thursday, January 20, 2011 12:51 PM
> To: Bradley Holbrook
> Cc: French, Martin; pgsql-admin@postgresql.org
> Subject: Re: Postgres Backup Utility
>
> On Thu, Jan 20, 2011 at 10:42 AM, Bradley Holbrook
> <operations_bradley@servillian.ca> wrote:
> > Thanks Scott... a couple comments.
> >
> > Our developers never decide what goes to where... they just happily
> > plumb away on the development db until we're ready to take
> our product
> > to testing (at regular intervals), once QA is passed, we
> wish to apply these to live.
> > We have several diff tools and sync tools, but they take forever
> > (especially the ones that only go one schema at a time).
> >
> > The DDL Logging sounds like a sufficient solution, can it be
> > configured to only record create and alter commands (or create or
> > replace commands on functions or updates on sequences, etc)? I'd
> > likely write a script to have this emailed to me at the end
> of every
> > day. I'm going to google DDL logging (never heard of it),
> but any good resources off the top of your head?
>
> It's basically logging anything that changes the structure of
> the database.  It would be easy enough to grep out what you
> do and don't want later.
>
> > Martin French is right though, ask your developers to write
> down all
> > their SQL struct changes and they look at you funny... and being a
> > developer myself I'd look at me funny. If you forget just
> once you're
> > screwed into a day sifting through tables and code.
>
> I've worked in three different shops now as a dev-dba and
> sysadmin, and in all three, all DDL changes had to be
> committed and / or handed over to the DBAs.  period.  Look
> funny all they want, they either give up the DDL or their
> code doesn't get pushed off dev servers onto anything else.
> At the very least they should be able to tell you which
> tables changed to go with which code changes, or you're not
> sure what code you can and can't push.  I get both of your
> point on this, but it's a discipline issue that needs sorting
> out with the developers if you want to have reproduceable ddl
> changes in all your systems that match the code changes.
>

Completely agree with Scott.

Only want to add that in this kind of development environment:
development/test/production -
Source code versioning software is absolute necessity (there are many:
CSV, SourceSafe, Perforce, ... - pick your choice).

Regards,
Igor Neyman

>> On Thu, Jan 20, 2011 at 10:42 AM, Bradley Holbrook
>> <operations_bradley@servillian.ca>  wrote:
>>> Thanks Scott... a couple comments.
>>>
>>> Our developers never decide what goes to where... they just happily
>>> plumb away on the development db until we're ready to take
>> our product
>>> to testing (at regular intervals), once QA is passed, we
>> wish to apply these to live.
>>> We have several diff tools and sync tools, but they take forever
>>> (especially the ones that only go one schema at a time).
>>>
>>> The DDL Logging sounds like a sufficient solution, can it be
>>> configured to only record create and alter commands (or create or
>>> replace commands on functions or updates on sequences, etc)? I'd
>>> likely write a script to have this emailed to me at the end
>> of every
>>> day. I'm going to google DDL logging (never heard of it),
>> but any good resources off the top of your head?
>>
>> It's basically logging anything that changes the structure of
>> the database.  It would be easy enough to grep out what you
>> do and don't want later.
>>
>>> Martin French is right though, ask your developers to write
>> down all
>>> their SQL struct changes and they look at you funny... and being a
>>> developer myself I'd look at me funny. If you forget just
>> once you're
>>> screwed into a day sifting through tables and code.
>>
>> I've worked in three different shops now as a dev-dba and
>> sysadmin, and in all three, all DDL changes had to be
>> committed and / or handed over to the DBAs.  period.  Look
>> funny all they want, they either give up the DDL or their
>> code doesn't get pushed off dev servers onto anything else.
>> At the very least they should be able to tell you which
>> tables changed to go with which code changes, or you're not
>> sure what code you can and can't push.  I get both of your
>> point on this, but it's a discipline issue that needs sorting
>> out with the developers if you want to have reproduceable ddl
>> changes in all your systems that match the code changes.
>>
>
> Completely agree with Scott.

Me too. The idea that a developer can change the schema without telling anyone is laughable. The idea that someone else
hasto reverse engineer schema changes is ludicrous. 

We have a strict procedure that every schema change has to be accompanied by a script that applies the changes. It has
tobe checked into Subversion along with all the other code. 

Make a new rule: Developers have to document schema changes. Institute a zero-tolerance policy for omissions.  If a
developercan create the DDL to make the change, surely it's not too much trouble to spend another minute adding it to a
fileand checking that file in. 

Craig

On Jan 20, 2011, at 12:42 PM, Bradley Holbrook wrote:

> Our developers never decide what goes to where... they just happily plumb
> away on the development db until we're ready to take our product to testing
> (at regular intervals), once QA is passed, we wish to apply these to live.
> We have several diff tools and sync tools, but they take forever (especially
> the ones that only go one schema at a time).

Using a diff tool, how would you know if a column was dropped and another one added versus simply renaming a column?
Won'tthat kind of difference matter when you want to apply the changes to your production database? 



John DeSoi, Ph.D.

On Thu, Jan 20, 2011 at 6:08 PM, John DeSoi <desoi@pgedit.com> wrote:
>
> On Jan 20, 2011, at 12:42 PM, Bradley Holbrook wrote:
>
>> Our developers never decide what goes to where... they just happily plumb
>> away on the development db until we're ready to take our product to testing
>> (at regular intervals), once QA is passed, we wish to apply these to live.
>> We have several diff tools and sync tools, but they take forever (especially
>> the ones that only go one schema at a time).
>
> Using a diff tool, how would you know if a column was dropped and another one added versus simply renaming a column?
Won'tthat kind of difference matter when you want to apply the changes to your production database? 

Yikes!  That could be bad.  Drop column xyz.  Hope that data wasn't
important.  To see if the column was renamed or something you'd have
to examine its rows to see if they matched a previous version.

I think that at the least I'd trawl the logs with ddl logging turned
on to get changes.  But seriously, I'd just lay down the law.  You
want changes in the dbs outside development, you give me, the DBA,
your DDL statements.  No statements, no code push.

On Jan 20, 2011, at 10:42 AM, Bradley Holbrook wrote:

> Martin French is right though, ask your developers to write down all their
> SQL struct changes and they look at you funny... and being a developer
> myself I'd look at me funny.

Well, it's what I do and it's trivial. Just don't type DDL directly into a psql session, edit it in your favorite text
editor,in a document called updates-pending.sql, and copy & paste into psql. Sometimes a tiny bit of *process* can
actuallyhelp productivity ;-) 

--
Scott Ribe
scott_ribe@elevated-dev.com
http://www.elevated-dev.com/
(303) 722-0567 voice