Thread: Shell Script help for backup

Shell Script help for backup

From
ratlhead@ratlhead.com (ratlhead)
Date:
Hey all,

I'm thinkin' maybe I should post this in a shell scripting group but
thought someone here may be able to help me out...

I'm lookin' to run a shell script through the crontab on a nightly
basis for backup, and not keep more than a week of backups.  In those
terms, I have a script working.

But I wanna take the script to another level.  Basically, I want it to
email me if pg_dump fails, but I'm having a hard time doing so.
here's what I got


#!/bin/bash
DATE=`date +%Y-%m-%d`
PGUSER=<my username>
PGPASSWORD=<my password>
export PGUSER PGPASSWORD
pg_dump <dbname> | gzip > /folder/pg_backup.$DATE.gzip
if [ !$? ]
then
  find /folder/pg_backup/* -mtime 8 -exec rm -f {} \;
  echo "The PostGreSQL backup for today completed successfully" | mail
ratlhead@ratlhead.com
else
  echo "The PostGreSQL backup for today was unsuccessful" | mail
ratlhead@ratlhead.com
fi


The problem is that is always gets detected as being successful, even
if I say, comment out the export line of my user/pass.  Any
suggestions on how I can detect whether or not it worked?

Something else I'll be lookin' to do is FTP'ing the backup to another
server.  If anyone has suggestions on where I can look for help on
that, it'd be great.

Thanks!

Re: Shell Script help for backup

From
strange@nsk.yi.org
Date:
On Sat, Jul 20, 2002 at 02:25:48AM -0700, ratlhead wrote:
>
> #!/bin/bash
> DATE=`date +%Y-%m-%d`
> PGUSER=<my username>
> PGPASSWORD=<my password>
> export PGUSER PGPASSWORD
> pg_dump <dbname> | gzip > /folder/pg_backup.$DATE.gzip
> if [ !$? ]
> then
>   find /folder/pg_backup/* -mtime 8 -exec rm -f {} \;
>   echo "The PostGreSQL backup for today completed successfully" | mail
> ratlhead@ratlhead.com
> else
>   echo "The PostGreSQL backup for today was unsuccessful" | mail
> ratlhead@ratlhead.com
> fi
>
>
> The problem is that is always gets detected as being successful, even
> if I say, comment out the export line of my user/pass.  Any
> suggestions on how I can detect whether or not it worked?

The script always detects a successful dump because the last command in
the pipe is gzip and it always succeeds.

You could change the [ !$? ] to something like:
[ "$( zcat /folder/pg_backup.$DATE.gzip | head -c1 | wc -c )" -eq 1 ]

> Something else I'll be lookin' to do is FTP'ing the backup to another
> server.  If anyone has suggestions on where I can look for help on
> that, it'd be great.

Use ncftpput or scp with no passphrase.

Regards,
Luciano Rocha

--
Consciousness: that annoying time between naps.

Re: Shell Script help for backup

From
david.morgan@s4c.co.uk
Date:
You don't have to compromise and use a clear passphrase, you can use
keychain, this uses ssh-agent and caches the user's passphrases:

http://www-106.ibm.com/developerworks/library/l-keyc2/





-----Original Message-----
From: strange@nsk.yi.org [mailto:strange@nsk.yi.org]
Sent: 22 July 2002 15:09
To: pgsql-general@postgresql.org
Subject: Re: [GENERAL] Shell Script help for backup


On Sat, Jul 20, 2002 at 02:25:48AM -0700, ratlhead wrote:
>
> #!/bin/bash
> DATE=`date +%Y-%m-%d`
> PGUSER=<my username>
> PGPASSWORD=<my password>
> export PGUSER PGPASSWORD
> pg_dump <dbname> | gzip > /folder/pg_backup.$DATE.gzip
> if [ !$? ]
> then
>   find /folder/pg_backup/* -mtime 8 -exec rm -f {} \;
>   echo "The PostGreSQL backup for today completed successfully" | mail
> ratlhead@ratlhead.com
> else
>   echo "The PostGreSQL backup for today was unsuccessful" | mail
> ratlhead@ratlhead.com
> fi
>
>
> The problem is that is always gets detected as being successful, even
> if I say, comment out the export line of my user/pass.  Any
> suggestions on how I can detect whether or not it worked?

The script always detects a successful dump because the last command in
the pipe is gzip and it always succeeds.

You could change the [ !$? ] to something like:
[ "$( zcat /folder/pg_backup.$DATE.gzip | head -c1 | wc -c )" -eq 1 ]

> Something else I'll be lookin' to do is FTP'ing the backup to another
> server.  If anyone has suggestions on where I can look for help on
> that, it'd be great.

Use ncftpput or scp with no passphrase.

Regards,
Luciano Rocha

--
Consciousness: that annoying time between naps.

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

Mae�r neges e-bost hon ac unrhyw ffeiliau sydd ynghlwm wrthi yn gwbl gyfrinachol ac wedi�u bwriadu at sylw yr unigolyn
neu�rendid y maent wedi eu cyfeirio ato yn unig. Gallant gynnwys gwybodaeth sy�n freintiedig yn gyfreithiol a/neu�n
gyfrinacholac os nad chi yw�r sawl y cyfeiriwyd y neges ato, yna peidiwch � chopio, dosbarthu na chymryd unrhyw gamau o
ganlyniada gan ddibynnu arni. Os ydych wedi derbyn y neges e-bost hon ar gam, a wnewch chi ein hysbysu o hyn cyn gynted
�phosib a�i dileu os gwelwch yn dda. Barn neu safbwyntiau�r awdur yw�r rhai a fynegir yn y neges e-bost hon ac/neu�n
unrhywatodiadau ac nid ydynt yn adlewyrchu o anghenraid barn neu safbwyntiau S4C. 

This e-mail message and any files attached are strictly confidential and intended solely for the attention of the
personor entity to whom they are addressed. They may contain legally privileged and/or confidential information and if
youare not the addressee, you should not copy, distribute nor take any action in reliance on them. If you have received
thise-mail message in error, please notify us as soon as possible and delete it. The views and opinions expressed in
thise-mail message and any attachments are the author�s own and they do not reflect necessarily the views and opinions
ofS4C. 


Re: Shell Script help for backup

From
ratlhead@ratlhead.com (ratlhead)
Date:
Hi all,

This is how I ended up solving my problem.  Not sure if any of this
will help you, but here's the scripts I use.  This is what it does...

-Uses PG environmental variables to connect to the PG server
-Dumps the specified database
-Checks the backup folder for any backups older than a week and
deletes those
-Checks to be sure the backup was successful by checking for a 0k file
for that day
-Sends email saying if backup was successful or failed
-If successful, checks the day and if it's Saturday, uploads most
current backup to specified ftp site
-.netrc file required for FTP connection


##### pg_backup.sh #####
#!/bin/bash
DATE=`date +%Y-%m-%d`
PGUSER=<my user>
PGPASSWORD=<my pass>
export PGUSER PGPASSWORD
/path/pg_dump <db name> | gzip > /path/pg_backup.$DATE.gzip
find /path/pg_backup/* -mtime 7 -exec rm -f {} \;
if test -s /path/pg_backup/pg_backup.$DATE.gzip
  then
    echo "The PostGreSQL backup for today was successful!" | mailx -s
"PostGreSQ
L backup successful" <email address>
    /path/day_pg.sh
  else
    echo "The PostGreSQL backup for today failed!" | mail -s
"PostGreSQL backup
failed" <email address>
fi


##### day_pg.sh #####
DAY=`date +%a`
if [ $DAY = "Sat" ]
  then
  /path/upload_pg.sh
fi


##### upload_pg #####
DATE=`date +%Y-%m-%d`
ftp <host> << EOF
cd path
put /path/pg_backup.$DATE.gzip pg_backup.$DATE.gzip
quit
EOF


##### .netrc #####
machine <host> login <username> password <password>



-Mike Graben
ratlhead@ratlhead.com

Re: Shell Script help for backup

From
strange@nsk.yi.org
Date:
On Mon, Jul 22, 2002 at 03:35:01PM -0700, ratlhead wrote:
> Hi all,
>
> This is how I ended up solving my problem.  Not sure if any of this
> will help you, but here's the scripts I use.  This is what it does...
>
> -Uses PG environmental variables to connect to the PG server
> -Dumps the specified database
> -Checks the backup folder for any backups older than a week and
> deletes those
> -Checks to be sure the backup was successful by checking for a 0k file
> for that day

What's wrong with the solution I sent?

And I doubt the checking for a 0k file will work for you as gzip will
create a 20 byte file even for a 0 byte input (some headers, etc.).

Regards,
Luciano Rocha

--
Consciousness: that annoying time between naps.

Re: Shell Script help for backup

From
ratlhead@ratlhead.com (ratlhead)
Date:
strange@nsk.yi.org wrote in message news:<20020723150902.A5173@nsk.yi.org>...
> What's wrong with the solution I sent?
>
> And I doubt the checking for a 0k file will work for you as gzip will
> create a 20 byte file even for a 0 byte input (some headers, etc.).
>
> Regards,
> Luciano Rocha

Luciano,

Was wondering if you could provide some help with the solution you
sent...

[ "$( zcat /folder/pg_backup.$DATE.gzip | head -c1 | wc -c )" -eq 1 ]

First, I can't seem to use zcat, because it likes to add on a ".Z" to
the filename for some reason or another.  Also, I find with the
version of 'head' installed on my machine, that -c isn't available,
but instead, -n.  I tried changing that and zcat to cat, but was
getting a fail on something that should be valid.

This basically checks to see if the file contains data right?

Thanks man!

Re: Shell Script help for backup

From
Bruce Momjian
Date:
You can do 'zcat < file' to prevent the *.Z requirement.  Yes, head -n
should work fine.

---------------------------------------------------------------------------

ratlhead wrote:
> strange@nsk.yi.org wrote in message news:<20020723150902.A5173@nsk.yi.org>...
> > What's wrong with the solution I sent?
> >
> > And I doubt the checking for a 0k file will work for you as gzip will
> > create a 20 byte file even for a 0 byte input (some headers, etc.).
> >
> > Regards,
> > Luciano Rocha
>
> Luciano,
>
> Was wondering if you could provide some help with the solution you
> sent...
>
> [ "$( zcat /folder/pg_backup.$DATE.gzip | head -c1 | wc -c )" -eq 1 ]
>
> First, I can't seem to use zcat, because it likes to add on a ".Z" to
> the filename for some reason or another.  Also, I find with the
> version of 'head' installed on my machine, that -c isn't available,
> but instead, -n.  I tried changing that and zcat to cat, but was
> getting a fail on something that should be valid.
>
> This basically checks to see if the file contains data right?
>
> Thanks man!
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org
>

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 853-3000
  +  If your life is a hard drive,     |  830 Blythe Avenue
  +  Christ can be your backup.        |  Drexel Hill, Pennsylvania 19026

Re: Shell Script help for backup

From
strange@nsk.yi.org
Date:
On Sun, Jul 28, 2002 at 11:22:22PM -0400, Bruce Momjian wrote:
>
> You can do 'zcat < file' to prevent the *.Z requirement.  Yes, head -n
> should work fine.

Hm... I missed the other email, so I'm replying to this:

The "$( zcat /folder/pg_backup.$DATE.gzip | head -c1 | wc -c )" is to
check if there are any bytes stored in the gzip file.

If the zcat expects an .Z file, try with gzcat file, gzcat < file, or
gzip -d < file.

head -c1 is meant to stop the decompression at the first byte, I didn't
want it to wait until full decompression of the gzipped file, and I didn't
use -n because then it would depend on the internal format of the
decompressed file (if it was a binary file, a newline could appear only
after quite some time).

If you're going to use head -n1, then change the test to:
[ "$( gzip -d < /folder/pg_backup.$DATE.gzip | head -n1 | wc -c )" -ge 1 ]

(ie, instead of checking if bytes == 1, check if bytes >= 1).

Regards,
Luciano Rocha

--
Consciousness: that annoying time between naps.