Thread: Website build script

Hi all,

The following script is currently used to build our main website. Static
HTML versions of each of the pages are generated from php scripts on an
hourly basis by this script, but the other day for some reason it
generated an empty index page, but gave no errors that we have been able
to find.

We've more or less resigned ourselves to this being a glitch, but
figured it wouldn't hurt to get a few extra sets of eyes over the script
anyway. Any comments or suggestions for improvements would be
appreciated.

Regards, Dave.


#! /bin/sh

########################################################################
#######
# chksvr - Check the status of the specified server
#
# $1 = Server
# $2 = Port
# $3 = Database
# $4 = User Name
########################################################################
#######
chksvr()
{ # Dummy query to check the server status status=`/usr/local/bin/psql -h $1 -p $2 -U $4 -d $3 -A -t -F '' -c
"SELECT 'SERVER_UP'"`
 if [ "$status" != "SERVER_UP" ]; then   mail -s "WARNING: Server $1:$2 is down!!" webmaster@postgresql.org <
/tmp/svr-status   rm /tmp/svr-status   exit 1 fi
}

########################################################################
#######
# buildpage - build an HTML page from a PHP page
#
# $1 = page name with path, but no extension
########################################################################
#######
buildpage()
{ /usr/local/bin/lynx -source http://www.postgresql.com$1.php \   | /usr/bin/perl -pe '$_ = " $_ "; tr/ \t/ /s; $_ =
substr($_,1,-1)'
\   > /usr/local/www/www.postgresql.org$1.html
}

########################################################################
#######
# Check the database servers
########################################################################
#######
echo Checking databases...

chksvr dbsvr1.postgresql.org 5432 186_portal portal > /tmp/svr-status
2>&1
chksvr dbsvr2.postgresql.org 5432 gborg www > /tmp/svr-status 2>&1
chksvr dbsvr3.postgresql.org 5432 mirrors vev > /tmp/svr-status 2>&1

########################################################################
#######
# Build the site
########################################################################
#######

echo Building site...

# The main portal
buildpage /index
buildpage /licence
buildpage /lists
buildpage /unavailable

# News/Events
buildpage /eventform
buildpage /newsform
buildpage /event-archive
buildpage /news-archive

# The Mirrors
buildpage /mirrors-ftp

# Docs
buildpage /docs/index
buildpage /docs/awbook
buildpage /docs/bookoneyearon
buildpage /docs/booktips

# User's Lounge
buildpage /users-lounge/index
buildpage /users-lounge/books
buildpage /users-lounge/features
buildpage /users-lounge/interfaces
buildpage /users-lounge/limitations
buildpage /users-lounge/related

# Cleanup
rm /tmp/svr-status

Seems possible that Lynx timed out and generated and sh left only an
empty file.

Guard against this by writing the output of lynx to a temporary file and
either check it for stuff you expect to be there (<!-- comments -->?) or
for some reasonable size. There are better was to do this, of course...

Gavin

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



As far as the problem of generating an empty index file, you could have 
the buildpage() function redirect the lynx output to a temporary file 
($1.temp) and check that file for validity somehow (i.e. non-zero size 
or grep for a known string ("Postgres"?)) If it fails, send mail to 
webmaster and exit. If it succeeds, run mv $1.temp $1.html. Or use cp 
instead of mv and you have a copy of the previous page always preserved.


- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200303111731
-----BEGIN PGP SIGNATURE-----
Comment: http://www.turnstep.com/pgp.html

iD8DBQE+bmSnvJuQZxSWSsgRAv3RAJ9vJ1Jaupd1XivctHzDunVv5sbWgwCdHmCC
9ehCuTJI5QOn79aV6AlYAxo=
=2BZ3
-----END PGP SIGNATURE-----

> -----Original Message-----
> From: Gavin Sherry [mailto:swm@linuxworld.com.au] 
> Sent: 11 March 2003 22:38
> To: Dave Page
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Website build script
> 
> 
> Seems possible that Lynx timed out and generated and sh left 
> only an empty file.
> 
> Guard against this by writing the output of lynx to a 
> temporary file and either check it for stuff you expect to be 
> there (<!-- comments -->?) or for some reasonable size. There 
> are better was to do this, of course...

Thanks, we did consider checking the size and for </html>. A known
comment might be better of course, as lynx might add </html> even if it
times out.

Regards, Dave.

On Tue, Mar 11, 2003 at 10:40:08PM -0000, greg@turnstep.com wrote:

> As far as the problem of generating an empty index file, you could have 
> the buildpage() function redirect the lynx output to a temporary file 
> ($1.temp) and check that file for validity somehow (i.e. non-zero size 
> or grep for a known string ("Postgres"?)) If it fails, send mail to 
> webmaster and exit. If it succeeds, run mv $1.temp $1.html. Or use cp 
> instead of mv and you have a copy of the previous page always preserved.

It seems much better to check $? on exit, and also to check stderr which
should be empty.

Same for the perl output.

-- 
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"El conflicto es el camino real hacia la union"

I have a customer porting an application from informix to postgres. They
require 2 things:

1) Cursors outside of transactions.
2) For update cursors as well as where current of 

If anyone is interested in this work, please reply off list.

-- 
Dave Cramer <dave@fastcrypt.com>
Cramer Consulting

Thought I would change the title to see if anyone would respond?

It wasn't clear but the intent was to support the following in the
backend.


On Tue, 2003-03-11 at 18:20, Dave Cramer wrote:
> I have a customer porting an application from informix to postgres. They
> require 2 things:
> 
> 1) Cursors outside of transactions.
> 2) For update cursors as well as where current of 
> 
> If anyone is interested in this work, please reply off list.
-- 
Dave Cramer <dave@fastcrypt.com>
Cramer Consulting