Thread: Server unreliability

Server unreliability

From
Bruce Momjian
Date:
It is my opinion that we have to make major changes in the way we
provide hosting for our servers.  There are several problems:

o  Location of servers

The location of our servers in Panama is a problem.  They are too far
for any PostgreSQL maintainers to access.  Changing hardware or
diagnosing problems has been too hard.  I have had like 2 days of
downtime on my home machine in the past 12 years.  We have had more than
2 days of downtime in the past 6 months.  My wife would not accept such
a reliability level.

o  FreeBSD

The use of FreeBSD jails can cause servers to take +8 hours to fsck on a
server crash or power failure.  Again, I would never accept such
problems on my home server so it is hard to fathom how a project with
thousands of users can accept that.  Either we need to find a fix, stop
using jails, or get another operating system, but continuing to use a
setup with a known problem is just asking for trouble.

o  Web site

We have been talking about a new web page layout for years at this
point.  I almost don't care if they just put a dancing bear up on the
web site.  Let's do something!

o  Archives

The archives situation is a continual problem.  Again, maybe a dancing
bear can help.  :-)

Basically, with no money and no one offering servers, I don't see a good
solution to any of these problems, but I think we need to recognize
these are problems and that we will continue to suffer until they are
addressed.

Are there any proposals, no matter how radical, to correct these?

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: [pgsql-www] Server unreliability

From
Devrim GUNDUZ
Date:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi,

On Wed, 29 Sep 2004, Bruce Momjian wrote:

> o  Location of servers
>
> The location of our servers in Panama is a problem.  They are too far
> for any PostgreSQL maintainers to access.  Changing hardware or
> diagnosing problems has been too hard.  I have had like 2 days of
> downtime on my home machine in the past 12 years.  We have had more than
> 2 days of downtime in the past 6 months.  My wife would not accept such
> a reliability level.

I think mirroring of the web & FTP sites helped hub.org too much, but I
still agree with you. Do you have any suggestions for that? Who
would/could support/donate such a bandwidth?

> o  FreeBSD
>
> The use of FreeBSD jails can cause servers to take +8 hours to fsck on a
> server crash or power failure.  Again, I would never accept such
> problems on my home server so it is hard to fathom how a project with
> thousands of users can accept that.  Either we need to find a fix, stop
> using jails, or get another operating system, but continuing to use a
> setup with a known problem is just asking for trouble.

I hope we'll not begin a OS/distro war! (O.K, it'd be good if Red Hat
contributes RHEL to PostgreSQL project ;) Anyone from Red Hat in here, or
in -core? ;) )

> o  Web site
>
> We have been talking about a new web page layout for years at this
> point.  I almost don't care if they just put a dancing bear up on the
> web site.  Let's do something!

Like this one? : http://www.jensummerfield.supanet.com/bearaniy.gif

Heh heh.

Well, web team (mostly Alexey) is working on it. We have a fabulous new
design (you know...). I think it'll be ready before 8.0.0. See the -www
archives for that (Archives? Umm maybe Google...)

> o  Archives
>
> The archives situation is a continual problem.  Again, maybe a dancing
> bear can help.  :-)

:-) A Google search box could be better sometimes...

Regards,

- --
Devrim GUNDUZ
devrim~gunduz.org                devrim.gunduz~linux.org.tr
             http://www.tdmsoft.com
             http://www.gunduz.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFBWvA6tl86P3SPfQ4RAkzLAKCVF+7lDhNbyWW00x9yXpXVdxew/gCeOp1C
Mf0/wlCZiX5ERu4e7Ef2xPc=
=NGbG
-----END PGP SIGNATURE-----

Re: Server unreliability

From
Robert Bernier
Date:

Bruce Momjian wrote:

>It is my opinion that we have to make major changes in the way we
>provide hosting for our servers.  There are several problems:
>
>o  Location of servers
>
>The location of our servers in Panama is a problem.  They are too far
>for any PostgreSQL maintainers to access.  Changing hardware or
>diagnosing problems has been too hard.  I have had like 2 days of
>downtime on my home machine in the past 12 years.  We have had more than
>2 days of downtime in the past 6 months.  My wife would not accept such
>a reliability level.
>
>

Could somebody explain to me how panama came to be chosen. Those reasons
may have changed since then.

>o  FreeBSD
>
>The use of FreeBSD jails can cause servers to take +8 hours to fsck on a
>server crash or power failure.  Again, I would never accept such
>problems on my home server so it is hard to fathom how a project with
>thousands of users can accept that.  Either we need to find a fix, stop
>using jails, or get another operating system, but continuing to use a
>setup with a known problem is just asking for trouble.
>
>

I'm very comfortable using Debian and would be willing to remote admin
using that. As for sticking with FreeBSD its the better choice if
there's somebody up to speed with security. The best resource person I
know is OReilly's FreeBSD columnist Dru, dlavigne6.sympatico.ca. She
might be convinced to "advise" what to do.

>o  Archives
>
>The archives situation is a continual problem.  Again, maybe a dancing
>bear can help.  :-)
>
>

How does the archives work i.e. what's running it? I hate to state the
obvious but can we have an itemized list as to what's wrong with them.
I've got my own ideas but I'd like to be on the same page as everybody else.

>Are there any proposals, no matter how radical, to correct these?
>
>

I've got thoughts as to server hosting but I don't even want to discuss
it until the guts of the site has been updated.


Robert

Re: Server unreliability

From
Darcy Buskermolen
Date:
On September 29, 2004 10:13 am, you wrote:
> It is my opinion that we have to make major changes in the way we
> provide hosting for our servers.  There are several problems:
>
> o  Location of servers
>
> The location of our servers in Panama is a problem.  They are too far
> for any PostgreSQL maintainers to access.  Changing hardware or
> diagnosing problems has been too hard.  I have had like 2 days of
> downtime on my home machine in the past 12 years.  We have had more than
> 2 days of downtime in the past 6 months.  My wife would not accept such
> a reliability level.

Yes location is a bit of a concern, but I think it's muchless of a concern
than the rest of the issues. I think with hardware/jail issues sorted out
this one will for the most part take careof it's self, or at least be far
less apparent.

>
> o  FreeBSD
>
> The use of FreeBSD jails can cause servers to take +8 hours to fsck on a
> server crash or power failure.  Again, I would never accept such
> problems on my home server so it is hard to fathom how a project with
> thousands of users can accept that.  Either we need to find a fix, stop
> using jails, or get another operating system, but continuing to use a
> setup with a known problem is just asking for trouble.

The over use of jails I think is complicating things, what are the odds that
someone could find it in their heart to raid the old hardware pile in their
serverback room, and come up with a PIII 800MHz + system with a gig of ram
that they could donate to the cause, and we move some of the more critical
services (www/ftp/anoncvs) to that box without jails.

Yahoo, Pair, Fugtisu, Affilias , Somebody, Anybody???

>
> o  Web site
>
> We have been talking about a new web page layout for years at this
> point.  I almost don't care if they just put a dancing bear up on the
> web site.  Let's do something!

One thing I know we can do here to help at least the availability of the
website itself, is to use a round-robin DNS based setup.  I know Marc
adjusted DNS records to point the main website at www3.ca a while back during
a time that www was stuck in fsck hell.  As for the website redesign, I agree
fully, it's been talked about for at least 3 full releases now and from my
understanding it's no closer now than it was before.
Can anybody comment on the hold up here?




>
> o  Archives
>
> The archives situation is a continual problem.  Again, maybe a dancing
> bear can help.  :-)

What about combining the efforts od pgsql.ru and command prompt to do this, or
maybe even monarc or google groups ?


>
> Basically, with no money and no one offering servers, I don't see a good
> solution to any of these problems, but I think we need to recognize
> these are problems and that we will continue to suffer until they are
> addressed.
Ok i know how much we all hate banner adds and the like, but what about using
Google ads especially on the archives to help generate income to fund these
requirements?


>
> Are there any proposals, no matter how radical, to correct these?

--
Darcy Buskermolen
Wavefire Technologies Corp.
ph: 250.717.0200
fx:  250.763.1759
http://www.wavefire.com

Re: [pgsql-www] Server unreliability

From
"Gavin M. Roy"
Date:
I think mirroring of the web & FTP sites helped hub.org too much, but I still agree with you. Do you have any suggestions for that? Who would/could support/donate such a bandwidth?

I'm always willing to donate such things.  I've got about 80mbit unused right now.

I hope we'll not begin a OS/distro war! (O.K, it'd be good if Red Hat contributes RHEL to PostgreSQL project ;) Anyone from Red Hat in here, or in -core? ;) )

Heh, I use Gentoo if it makes any difference.

Well, web team (mostly Alexey) is working on it. We have a fabulous new design (you know...). I think it'll be ready before 8.0.0. See the -www archives for that (Archives? Umm maybe Google...)

Hopefully the logo banner gets updated with my change or the author can clean it up a bit... the current one is too fuzzy, the glassy logo is nice but didnt resize well.  I can make one sized correctly if wanted.

Gavin

Re: [pgsql-www] Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Bruce Momjian wrote:

> It is my opinion that we have to make major changes in the way we
> provide hosting for our servers.  There are several problems:
>
> o  Location of servers
>
> The location of our servers in Panama is a problem.  They are too far
> for any PostgreSQL maintainers to access.  Changing hardware or
> diagnosing problems has been too hard.  I have had like 2 days of
> downtime on my home machine in the past 12 years.  We have had more than
> 2 days of downtime in the past 6 months.  My wife would not accept such
> a reliability level.

This is currently being worked on ... we are looking at various remote
management solutions so that we don't have to deal with waiting for a
technician to get 'on the scene' ...

> o  FreeBSD
>
> The use of FreeBSD jails can cause servers to take +8 hours to fsck on a
> server crash or power failure.  Again, I would never accept such
> problems on my home server so it is hard to fathom how a project with
> thousands of users can accept that.  Either we need to find a fix, stop
> using jails, or get another operating system, but continuing to use a
> setup with a known problem is just asking for trouble.

Actually, again, this one is being addressed ... there is a solution in
the pipeline to fix the cause of the 8+ hour fsck, but, since it is a fix
to fsck itself, it hasn't been put into the mainstream code yet, due to
*obvious* testing reasons ...

We've also added in hot failover as an option ... I've posted to -www
asking about putting www.postgresql.org onto it, but so far, the only
responses back have been along the lines of 'how are you doing it?' ...

The "risk" is that its not real time replication between the live and
failover server ... on our high performance servers, the 'delay' is about
5 minutes ...

... now, knowing that, if you feel comfortable with me putting this onto
the mailing lists/cvsroot as well, knowing that there is a possibility of
something being written 'in the gap' before failover, I'll do that VM also
...

note that altho the replication has a gap, the heartbeat process runs
every minute ... as soon as it can't ping anymore, it fails over ...

> o  Web site
>
> We have been talking about a new web page layout for years at this
> point.  I almost don't care if they just put a dancing bear up on the
> web site.  Let's do something!

What's wrong with the existing one?  Have you designed the dancing bear
you'd like us to put up in place of what we have now?

> o  Archives
>
> The archives situation is a continual problem.  Again, maybe a dancing
> bear can help.  :-)

What is wrong with it now?  I'm cleaning up the code itself, but that is
due to it being a mess right now, not due to any problems reported to me,
removed one of the banner ads so that loading is a bit faster, and John
has done, I think, a fantastic job on the search engine itself, including
sending me changes for the archives themselves so that the 'time searches'
should now work properly ...

So, do you have something specific you'd like to point out to us that
we've overlooked and haven't fixed yet?

> Basically, with no money and no one offering servers

So far, I've had one person donate $10 ... in order to put a dedicated
server onto the network, I'd need alot more of those ... that would pretty
much eliminate your second point about the fsck's, since its only our
*loaded* servers, that we have that problem with ... but, as I said, the
fsck issue is being addressed as well ...



----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: [pgsql-www] Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Gavin M. Roy wrote:

>   /I think mirroring of the web & FTP sites helped hub.org too much,
>   but I still agree with you. Do you have any suggestions for that?
>   Who would/could support/donate such a bandwidth?
>
>   /
>
> I'm always willing to donate such things.  I've got about 80mbit unused
> right now./

a few things to note ... the only 'critical' thing right now that can't be
mirrored is the cvsroot/mailing lists ... web and ftp are both well
mirrored, and, in fact, ftp.postgresql.org is on a seperate network/server
that I have ... archives was also moved there, and over the next few days,
will be pointed to one of commandprompt's web sites ...

in fact, bittorrent, ftp and archives are all on a US based server ...
what I'd like to do is setup www.postgresql.org to point, via dns round
robin, to the www.us.* mirrors, so that hits to it are more load balanced,
but that requires some major changes to the mirrors themselves, and few of
them are configured to recognize a virtual host of www.postgresql.org
right now, as I found out not too long ago :(

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Robert Bernier wrote:

> Could somebody explain to me how panama came to be chosen. Those reasons
> may have changed since then.

Client requirements ... and, quite frankly, the issues that Bruce is
bringing up are in the works of being fixed by getting some remote
management hardware so that I can deal with 99% of the problems from here
...

> I'm very comfortable using Debian and would be willing to remote admin
> using that. As for sticking with FreeBSD its the better choice if
> there's somebody up to speed with security. The best resource person I
> know is OReilly's FreeBSD columnist Dru, dlavigne6.sympatico.ca. She
> might be convinced to "advise" what to do.

the problem isn't a security issue ... the problem is that the unionfs
creates these 'zero length directories' all over the place in the normal
course of operations.  fsck was never coded to handle them properly, so
when it does do the fsck, it goes through a removes them all (not harmful,
just time consuming) ... one of the developers was recently able to come
up with a patch to correct the behaviour, but is rightfully nervous of
putting it into the source tree, so is doing some more testing on it first
...

As I also mentioned in the other thread, we've setup a hot failover option
that I'd like to put on the VMs, but haven't had any good feedback on the
-www lists about doing this ... for www.*, I don't believe there is any
risk with doign it, but for stuff like pgfoundry/gborg and mailing lists,
those VMs all have a cvsroot, so am a bit afraid to have it hotfailover
and potentially lose someone's commit ... if ppl feel that is an
acceptable risk to avoid the lengthy downtimes, I can enable it on all of
the *.postgresql.org VMs right now ... but I need some feedback on that
first ...

> How does the archives work i.e. what's running it? I hate to state the
> obvious but can we have an itemized list as to what's wrong with them.
> I've got my own ideas but I'd like to be on the same page as everybody
> else.

I'd love to see a list as well ... I thought that John/Dave and I had
addressed all of the outstandings, but obviously Bruce doesn't feel the
same way ... or, of course, he could be doing like the last time and
rambling off past problems that have since been fixed :(

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Darcy Buskermolen wrote:

> The over use of jails I think is complicating things, what are the odds
> that someone could find it in their heart to raid the old hardware pile
> in their serverback room, and come up with a PIII 800MHz + system with a
> gig of ram that they could donate to the cause, and we move some of the
> more critical services (www/ftp/anoncvs) to that box without jails.

jails are not the problem ... the problem is the file system we are using
to share applications ... but, yes, if we could put a dedicated box in
place, we could very easily eliminate the use of the file system ... we
have never had a jail related crash on those servers, they have always
come down to the file system itself ...

> One thing I know we can do here to help at least the availability of the
> website itself, is to use a round-robin DNS based setup.  I know Marc
> adjusted DNS records to point the main website at www3.ca a while back
> during a time that www was stuck in fsck hell.

To do this, and I'd love to, we'd need to have *at least* all of the
www*.us.postgresql.org mirrors setup a ServerAlias for www.postgresql.org
... not sure what would be involved in getting that done though, but the
DNS side wouldn't take much ...

> What about combining the efforts od pgsql.ru and command prompt to do
> this, or maybe even monarc or google groups ?

Until someone can *list* the problems, how can a solution be suggested?
John and I know of no issues at this time ... the last one was the 'time
period' searches, which we believe we've just fixed ...

> Ok i know how much we all hate banner adds and the like, but what about
> using Google ads especially on the archives to help generate income to
> fund these requirements?

Huh?  Google ads?  I know what you mean, just how do we make use of them?

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: [pgsql-www] Server unreliability

From
Peter Eisentraut
Date:
Bruce Momjian wrote:
> Basically, with no money and no one offering servers, I don't see a
> good solution to any of these problems, but I think we need to
> recognize these are problems and that we will continue to suffer
> until they are addressed.

I'm sure people would offer money, servers, and pipes if there were some
public accounting and governance.  Right now, we don't really know

- What services are being run?
- How many resources does that take?
- Who provides those resources?
- Who administers these services?

Take a look at this: <http://db.debian.org/machines.cgi>.  This is the
sort of thing we need before we can continue planning our system
resources.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/


Re: Server unreliability

From
Mitch Pirtle
Date:
Hi Bruce, we met in Manhattan for your NYPHP presentation - I was the
lone PostgreSQL cheerleader in the back ;-)

You ask for proposals, no matter how radical, so here goes:

o  Location of servers

Pending bandwidth requirements, I am willing to sponsor hosting of
these sites, possibly on dedicated hardware.  But I need some ideas of
bandwidth usage so I can figure out which strategies make the most
sense financially.  BTW be glad your servers are not in Florida, I
have one client that is in a significant amount of pain right now.

o  FreeBSD

I'm agnostic in this regard, but have experienced that Linux distros
seem to be the easiest to maintain (Debian, Fedora).

o  Web site

I am a core developer on the Mambo project (www.mamboserver.com), and
joined the core team primarily to implement database abstraction
(currently tied to MySQL gasp!).  Mambo has grown quite famous for
being easy to use, has lightweight requirements, and has a huge 3rd
party community that provides all kinds of extra goodies (forums, i18n
support, commerce, you name it).  I would happily volunteer the
efforts to migrate the existing site to Mambo to make it easier for
the core gang to keep the site current.  I would also consider having
PostgreSQL.org running Mambo being excellent incentive to prioritize
the implementation of ADOdb ;-)

o  Archives

Depending on the format, Mambo natively supports archiving of content.
 This may be a big win for PostgreSQL; and I would just need to get
more information on the scope (and current format) of the archived
content.

o  About me

I have been very active as a PostgreSQL booster (not as a developer),
so none of you have ever heard of me - but I have been very vocal in
both Switzerland (where I lived until last year) and in Manhattan.
Writing articles for International PHP Magazine (www.php-mag.net) also
gives me a chance for some subliminal advertising.

There you have it, consider it as a preliminary offer - it is the best
I can do, being that all of you have given me such a powerful database
for free.  Perhaps being a web developer means that the best way for
me to contribute would be on the website?

Regards,

Mitch Pirtle
Mambo core eveloper (www.mamboserver.com)
International PHP Magazine author (www.php-mag.net)

Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Mitch Pirtle wrote:

> Hi Bruce, we met in Manhattan for your NYPHP presentation - I was the
> lone PostgreSQL cheerleader in the back ;-)
>
> You ask for proposals, no matter how radical, so here goes:
>
> o  Location of servers
>
> Pending bandwidth requirements, I am willing to sponsor hosting of
> these sites, possibly on dedicated hardware.  But I need some ideas of
> bandwidth usage so I can figure out which strategies make the most
> sense financially.  BTW be glad your servers are not in Florida, I
> have one client that is in a significant amount of pain right now.

The server location is not changing, but thank you for your offer ...

> o  FreeBSD
>
> I'm agnostic in this regard, but have experienced that Linux distros
> seem to be the easiest to maintain (Debian, Fedora).

Again, operating system isn't changing, but thank you for your offer ...

Both of the above issues that Bruce pointed out are being worked
on/addressed, just not as fast as Bruce would like to see happen ...

> o  Web site
>
> I am a core developer on the Mambo project (www.mamboserver.com), and
> joined the core team primarily to implement database abstraction
> (currently tied to MySQL gasp!).  Mambo has grown quite famous for
> being easy to use, has lightweight requirements, and has a huge 3rd
> party community that provides all kinds of extra goodies (forums, i18n
> support, commerce, you name it).  I would happily volunteer the
> efforts to migrate the existing site to Mambo to make it easier for
> the core gang to keep the site current.  I would also consider having
> PostgreSQL.org running Mambo being excellent incentive to prioritize
> the implementation of ADOdb ;-)

There is currently a re-write of the web site being worked on ... if you
wish to aid in that, please feel free to subscribe to pgsql-www, where the
discussions on this are taking place.  Alexey(sp?) has spent a good deal
of time working on/cleaning up the backend code itself, and there has been
one site redesign proposal that I *believe* has been accepted already for
how the new site will look ... the fun part is trying to find man hours to
implement it :(

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: [pgsql-www] Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Peter Eisentraut wrote:

> Bruce Momjian wrote:
>> Basically, with no money and no one offering servers, I don't see a
>> good solution to any of these problems, but I think we need to
>> recognize these are problems and that we will continue to suffer
>> until they are addressed.
>
> I'm sure people would offer money, servers, and pipes if there were some
> public accounting and governance.  Right now, we don't really know
>
> - What services are being run?
> - How many resources does that take?
> - Who provides those resources?
> - Who administers these services?
>
> Take a look at this: <http://db.debian.org/machines.cgi>.  This is the
> sort of thing we need before we can continue planning our system
> resources.

If someone wishes to create the form/interface, I'll fill in the blanks
... right off the top of my head, we have the following VMs running:

svr1.postgresql.org     - mailing lists, developer.postgresql.org, cvsroot
                         - host server: neptune, Dual Xeon

www.postgresql.org      - techdocs, advocacy, www
                         - host server: venus, Dual Athlon

pgfoundry.org           - host server: jupiter, Dual-PIII

gborg.postgresql.org    - host server: venus, Dual Athlon

archives.postgresql.org - mailing list archives
                         - moving to one of commandprompt.com's servers

ftp.postgresql.org      - also bittorrent and anoncvs
                         - host server: saturn, leased computer

pugs.postgresql.org     - host server: venus, Dual Athlon

search.postgresql.org    - hosted on John's server

Did I miss any?

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Server unreliability

From
Darcy Buskermolen
Date:
On September 29, 2004 11:09 am, Marc G. Fournier wrote:
> On Wed, 29 Sep 2004, Darcy Buskermolen wrote:
----8<----[snip]
 > jails are not the problem ... the problem is the file system we are using
> to share applications ... but, yes, if we could put a dedicated box in
> place, we could very easily eliminate the use of the file system ... we
> have never had a jail related crash on those servers, they have always
> come down to the file system itself ...

Poor use of terminlogy on my part, but the net effect is the same, the FS is
being used to facilitate the jail, take away the jail, take away the need of
the stacked FS.


> To do this, and I'd love to, we'd need to have *at least* all of the
> www*.us.postgresql.org mirrors setup a ServerAlias for www.postgresql.org
> ... not sure what would be involved in getting that done though, but the
> DNS side wouldn't take much ...

*nod* In the interim there is nothing stoping getting the process rolling with
those sites that are known to work, as well as contacting the mirror admins.
of the unknown sites..

>
> > What about combining the efforts od pgsql.ru and command prompt to do
> > this, or maybe even monarc or google groups ?
>
> Until someone can *list* the problems, how can a solution be suggested?
> John and I know of no issues at this time ... the last one was the 'time
> period' searches, which we believe we've just fixed ...

I'm not sure what the issues have been other than availability.

>
> > Ok i know how much we all hate banner adds and the like, but what about
> > using Google ads especially on the archives to help generate income to
> > fund these requirements?
>
> Huh?  Google ads?  I know what you mean, just how do we make use of them?

Specificly the program out lines at: https://www.google.com/adsense/  I know
of a few reasonably high traffic sites that are generating about 5k/mth
income from this program.

>
> ----
> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)

--
Darcy Buskermolen
Wavefire Technologies Corp.
ph: 250.717.0200
fx:  250.763.1759
http://www.wavefire.com

Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Darcy Buskermolen wrote:

> On September 29, 2004 11:09 am, Marc G. Fournier wrote:
>> On Wed, 29 Sep 2004, Darcy Buskermolen wrote:
> ----8<----[snip]
> > jails are not the problem ... the problem is the file system we are using
>> to share applications ... but, yes, if we could put a dedicated box in
>> place, we could very easily eliminate the use of the file system ... we
>> have never had a jail related crash on those servers, they have always
>> come down to the file system itself ...
>
> Poor use of terminlogy on my part, but the net effect is the same, the FS is
> being used to facilitate the jail, take away the jail, take away the need of
> the stacked FS.

Actuallym, not true ... the FS is used to reduce the overall amount of
disk space required, since ~90% of the jail is redundant ... when I can
get a dedicated server in place, and only have the 10 or so VMs running on
it, then disk space is no longer an issue, so I can easily get rid of the
FS component ... that would get rid of the crashes ...

> *nod* In the interim there is nothing stoping getting the process
> rolling with those sites that are known to work, as well as contacting
> the mirror admins. of the unknown sites..

I think you are the only 'known to work site' ... so, if I setup a round
robin between the Panama and Canadian location for www.postgresql.org,
would you be okay with the extra load? :)  If so, I'll get to work on the
coding changing required for the DNS ...

> Specificly the program out lines at: https://www.google.com/adsense/ I
> know of a few reasonably high traffic sites that are generating about
> 5k/mth income from this program.

'k, will have to read through, but first scan, would the replace our
current search system, or would the 'Google search box' be on the right of
the screen, or ... ?  Without me reading it all the way through, do you
already know?

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Server unreliability

From
Josh Berkus
Date:
Bruce,

> It is my opinion that we have to make major changes in the way we
> provide hosting for our servers.  There are several problems:

You're not the only one to have noticed this, believe it or not.  ;-)

> The location of our servers in Panama is a problem.  They are too far
> for any PostgreSQL maintainers to access.  Changing hardware or
> diagnosing problems has been too hard.  I have had like 2 days of
> downtime on my home machine in the past 12 years.  We have had more than
> 2 days of downtime in the past 6 months.  My wife would not accept such
> a reliability level.

To be fair, PostgreSQL uses an enormous bandwidth and traffic that gives our
systems very little tolerance for anominalies.   Not that having the stuff in
Panama doesn't slow down fixes, but hosting postgresql.org is a bit more
demanding than hosting, say, your or my personal web site.

> The use of FreeBSD jails can cause servers to take +8 hours to fsck on a
> server crash or power failure.  Again, I would never accept such
> problems on my home server so it is hard to fathom how a project with
> thousands of users can accept that.  Either we need to find a fix, stop
> using jails, or get another operating system, but continuing to use a
> setup with a known problem is just asking for trouble.

I agree that when we have more servers dedicated to PostgreSQL, we should use
a different approach than the jails.  Marc, what are the advantages of the
jails on a dedicated server, since we are so familiar with their drawbacks?

> We have been talking about a new web page layout for years at this
> point.  I almost don't care if they just put a dancing bear up on the
> web site.  Let's do something!

I'd suggest finding some money to sponsor somebody full-time for a month to
accomplish the migration to the new web structure, which is more-or-less
finished.   That's what's holding us up right now AFAIK.

> The archives situation is a continual problem.  Again, maybe a dancing
> bear can help.  :-)

But being solved.   The archives have been copied to CommandPrompt, and the
various archive search tools have been distributed.   Once we can change the
web interface, there should always be one tool and/or version available
regardless of individual server/hosting failures.

Further, SF.net has offered to host our mailing lists, which is tempting
because of their direct mailing-list-to-database tool, as well as their more
distributed hosting structure which prevents DNS-based failures.   I expect
to find out the terms for this next week, and then discuss it on WWW.

> Basically, with no money and no one offering servers, I don't see a good
> solution to any of these problems, but I think we need to recognize
> these are problems and that we will continue to suffer until they are
> addressed.
>
> Are there any proposals, no matter how radical, to correct these?

The big obstacle in moving anything is bandwidth.   While any number of
companies will offer to host stuff for us, we can't match the amount of
bandwidth we're currently using in Panama -- and hosting donors won't support
anywhere near that level (which runs to scores of GB per month).  So while we
can move individual, less-crucial components, the mailing lists and the main
WWW require either Hub.org's generous bandwitdh or other hosting that we
"own".

I have actually been working on raising money for additional servers.   The
obstacles which have made this a year-long process are more complicated than
I want to go over right now, but there are people thinking about it.

--
Josh Berkus
Aglio Database Solutions
San Francisco

Re: [pgsql-www] Server unreliability

From
"Gavin M. Roy"
Date:
Josh Berkus wrote:

>The big obstacle in moving anything is bandwidth.   While any number of
>companies will offer to host stuff for us, we can't match the amount of
>bandwidth we're currently using in Panama -- and hosting donors won't support
>anywhere near that level (which runs to scores of GB per month).  So while we
>can move individual, less-crucial components, the mailing lists and the main
>WWW require either Hub.org's generous bandwitdh or other hosting that we
>"own".
>
>I have actually been working on raising money for additional servers.   The
>obstacles which have made this a year-long process are more complicated than
>I want to go over right now, but there are people thinking about it.
>
>
>
I'd be more than happy to donate 50mbit per sec or more on an ongoing basis.

Gavin

Re: Server unreliability

From
Josh Berkus
Date:
Marc, Darcy, Bruce, Peter, et. al.

> Poor use of terminlogy on my part, but the net effect is the same, the FS
> is being used to facilitate the jail, take away the jail, take away the
> need of the stacked FS.

The other BSD experts I've talked to all agree that the Jail setup on the
current servers is a severe performance drain.   I think we really need to
look at not using jails for the the dedicated PostgreSQL server(s), when we
have them.

> One thing I know we can do here to help at least the availability of the
> website itself, is to use a round-robin DNS based setup.

FWIW, some folks at SourceForge have offered their help in configuring this.
I'm currently waiting for Adi to get back from vacation to proceed.

> I'm sure people would offer money, servers, and pipes if there were some
> public accounting and governance.  Right now, we don't really know
>
> - What services are being run?
> - How many resources does that take?
> - Who provides those resources?
> - Who administers these services?

This is why I'd like to do some of this stuff -- particularly fundraising --
through the Foundation.   However, as you know Peter, there have been some
delays in that regard.

--
Josh Berkus
Aglio Database Solutions
San Francisco

Re: [pgsql-www] Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Josh Berkus wrote:

> I agree that when we have more servers dedicated to PostgreSQL, we
> should use a different approach than the jails.  Marc, what are the
> advantages of the jails on a dedicated server, since we are so familiar
> with their drawbacks?

'k, just to re-emphasize ... the jails are not, and have never been, the
problem ... when I can put a dedicated server in, though, the thing that
will fix the problem is getting rid of the unionfs that we're using to
converse disk space ...

as to what advantages the jail's provide ... mainly, security ... we can
easily provide root access to pugs.postgresql.org, as an example, so that
Fred (please tell me I remembered right?) can do whatever work he wants,
without giving him full root on the developer jail ... same with all of
the other jails ... Chris has full root on gborg, so that he doesn't have
to ask someone else to do something when he notices a problem (ie. restart
mailman) ...

The other advantage ... seperate process space ... so, as an example, if
someone wanted to run a different server then apache on port 80, they
could ...

> But being solved.  The archives have been copied to CommandPrompt, and
> the various archive search tools have been distributed.  Once we can
> change the web interface, there should always be one tool and/or version
> available regardless of individual server/hosting failures.

Joshua and I should have this switched over by end of day today ... I just
did a bunch of "clean ups" to fix some of the issues with the search form,
but to also balance things out a bit more ... also removed the right
banner ad, so that page loading is a bit faster ... and I think we have
the 'Last-modified' stuff finally working properly ...

> The big obstacle in moving anything is bandwidth.   While any number of
> companies will offer to host stuff for us, we can't match the amount of
> bandwidth we're currently using in Panama -- and hosting donors won't support
> anywhere near that level (which runs to scores of GB per month).  So while we
> can move individual, less-crucial components

And this is somethign that we have been working on over the past several
months ... search has moved to John's location/servers, archives has done
one move so far, and is going to be moving to CPs location, etc ...

And, now that we have, at least, an option for the hot failover, those VMs
that it is safe to do it with, we can get that going so that downtime of
one server isn't near as critical as it once was ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Server unreliability

From
Darcy Buskermolen
Date:
On September 29, 2004 11:49 am, Marc G. Fournier wrote:
> On Wed, 29 Sep 2004, Darcy Buskermolen wrote:
> > On September 29, 2004 11:09 am, Marc G. Fournier wrote:
> >> On Wed, 29 Sep 2004, Darcy Buskermolen wrote:
----8<----[snip]

> > *nod* In the interim there is nothing stoping getting the process
> > rolling with those sites that are known to work, as well as contacting
> > the mirror admins. of the unknown sites..
>
> I think you are the only 'known to work site' ... so, if I setup a round
> robin between the Panama and Canadian location for www.postgresql.org,
> would you be okay with the extra load? :)  If so, I'll get to work on the
> coding changing required for the DNS ...

I have no issues with that being setup now, I'll keep na eye on the bandwith
and let you know if it becomes problematic.

>
> > Specificly the program out lines at: https://www.google.com/adsense/ I
> > know of a few reasonably high traffic sites that are generating about
> > 5k/mth income from this program.
>
> 'k, will have to read through, but first scan, would the replace our
> current search system, or would the 'Google search box' be on the right of
> the screen, or ... ?  Without me reading it all the way through, do you
> already know?
Not the search system, but the addsence.  it's just a banner serving
technology where you embed piece of javascript to imbed in the site and away
we go..
For example here is a sample I just fired up based on what I use in another
site....  http://www.dbitech.ca/
and the JS to pull it of is...

<script type="text/javascript"><!--
google_ad_client = "pub-6839445463094639";
google_ad_width = 120;
google_ad_height = 240;
google_ad_format = "120x240_as";
google_ad_channel ="";
google_color_border = "DFF2FD";
google_color_bg = "DFF2FD";
google_color_link = "000000";
google_color_url = "336699";
google_color_text = "333333";
//--></script>
<script type="text/javascript"
  src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
</script>


>
> ----
> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

--
Darcy Buskermolen
Wavefire Technologies Corp.
ph: 250.717.0200
fx:  250.763.1759
http://www.wavefire.com

Re: Server unreliability

From
"Magnus Hagander"
Date:
> > We have been talking about a new web page layout for years at this
> > point.  I almost don't care if they just put a dancing bear
> up on the
> > web site.  Let's do something!
>
> I'd suggest finding some money to sponsor somebody full-time
> for a month to accomplish the migration to the new web
> structure, which is more-or-less
> finished.   That's what's holding us up right now AFAIK.

What kind of work is required for this, precisely? Is it coding, or just
moving plain content over? If it's just plain content, how much work
would it be to make an interface so several different people can do just
a couple of pages each in a fully distributed way?

Is there a list somewhere on what needs to be done (like the TODO list)?
I see a few on the gborg site, but surely that's not all that remains?
(they're all techdocs only..)


//Magnus


Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Darcy Buskermolen wrote:

>> I think you are the only 'known to work site' ... so, if I setup a round
>> robin between the Panama and Canadian location for www.postgresql.org,
>> would you be okay with the extra load? :)  If so, I'll get to work on the
>> coding changing required for the DNS ...
>
> I have no issues with that being setup now, I'll keep na eye on the bandwith
> and let you know if it becomes problematic.

'k, will play with the DNS code tonight then ...

> <script type="text/javascript"><!--
> google_ad_client = "pub-6839445463094639";
> google_ad_width = 120;
> google_ad_height = 240;
> google_ad_format = "120x240_as";
> google_ad_channel ="";
> google_color_border = "DFF2FD";
> google_color_bg = "DFF2FD";
> google_color_link = "000000";
> google_color_url = "336699";
> google_color_text = "333333";
> //--></script>
> <script type="text/javascript"
>  src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
> </script>

Sweet ... okay, will look at getting that added to archives then ...
thanks for the pointer ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Magnus Hagander wrote:

>>> We have been talking about a new web page layout for years at this
>>> point.  I almost don't care if they just put a dancing bear
>> up on the
>>> web site.  Let's do something!
>>
>> I'd suggest finding some money to sponsor somebody full-time
>> for a month to accomplish the migration to the new web
>> structure, which is more-or-less
>> finished.   That's what's holding us up right now AFAIK.
>
> What kind of work is required for this, precisely? Is it coding, or just
> moving plain content over? If it's just plain content, how much work
> would it be to make an interface so several different people can do just
> a couple of pages each in a fully distributed way?

I *believe* that this was one of hte reasons we looked at Bricolage awhile
back, no?

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Server unreliability

From
Josh Berkus
Date:
Marc,

> I *believe* that this was one of hte reasons we looked at Bricolage awhile
> back, no?

Well, no.   Bricolage was (and is) considered just for Techdocs.   The holdup
there was:
1) getting a Mason expert volunteer to do the template design (we've had one
offer), and
2) waiting for the new www.postgresql.org design to be completed to integrate
with.

However, Bric was not considered for the main site.   For more details, read
the archives of the WWW list.

--
Josh Berkus
Aglio Database Solutions
San Francisco

Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Josh Berkus wrote:

> The other BSD experts I've talked to all agree that the Jail setup on
> the current servers is a severe performance drain.  I think we really
> need to look at not using jails for the the dedicated PostgreSQL
> server(s), when we have them.

k, have your experts talk to my experts :)

more seriously, I have two Dual-PIIIs running with ~25 jails each, and
there are no performance issues that are readily apparent, either with
processes or network ... even neptune, where there are currently 59
running, isn't breaking much of a sweat, to the point that our
conversation here is almost real time, and that is where the mailing lists
are running off of ...

All the 'jail' does is a chroot to a specific directory structure, and
tags the processes running on the server with the 'jail-id' so that ps
inside of the jail only shows those processed appropriately tags ...
unless I've been mis-informed, there is no other overhead ...

once we can put dedicated servers in place, the # of jails running won't
be more then 10, unless there is some major growth in requirements ...

we could probably reduce the # of jails though ... pugs could be merged in
with www quite easily, but I wouldn't merge cvs/pgfoundry and gborg into
the same jail for what I would *hope* are obvious reasons ...

If your experts can give me something more substantial to work with then
opinions, though, I'd happily read it ...

> FWIW, some folks at SourceForge have offered their help in configuring
> this. I'm currently waiting for Adi to get back from vacation to
> proceed.

As there isn't much required in order to config this (as its stuff that
we're already doing to load balance things like the banner ads), not quite
sure what they have in mind *shrug*

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Server unreliability

From
Bruce Momjian
Date:
Marc G. Fournier wrote:
> On Wed, 29 Sep 2004, Mitch Pirtle wrote:
>
> > Hi Bruce, we met in Manhattan for your NYPHP presentation - I was the
> > lone PostgreSQL cheerleader in the back ;-)
> >
> > You ask for proposals, no matter how radical, so here goes:
> >
> > o  Location of servers
> >
> > Pending bandwidth requirements, I am willing to sponsor hosting of
> > these sites, possibly on dedicated hardware.  But I need some ideas of
> > bandwidth usage so I can figure out which strategies make the most
> > sense financially.  BTW be glad your servers are not in Florida, I
> > have one client that is in a significant amount of pain right now.
>
> The server location is not changing, but thank you for your offer ...
>
> > o  FreeBSD
> >
> > I'm agnostic in this regard, but have experienced that Linux distros
> > seem to be the easiest to maintain (Debian, Fedora).
>
> Again, operating system isn't changing, but thank you for your offer ...

I assume you mean "not changing" while you are administering the
servers.  I don't remember any community agreement that the current
setup was permanent, and my thinking now is that all avenues are open
for exploration.

I am not ready to dismiss this offer/idea.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: [pgsql-www] Server unreliability

From
Alvaro Herrera
Date:
On Wed, Sep 29, 2004 at 04:07:20PM -0300, Marc G. Fournier wrote:

> as to what advantages the jail's provide ... mainly, security ... we can
> easily provide root access to pugs.postgresql.org, as an example, so that
> Fred (please tell me I remembered right?) can do whatever work he wants,
> without giving him full root on the developer jail ... same with all of
> the other jails ... Chris has full root on gborg, so that he doesn't have
> to ask someone else to do something when he notices a problem (ie. restart
> mailman) ...

This business about anyone having to restart anything is strange.  I
don't run mailman lists personally anymore, but back when I used to,
there was no need to restart anything.  This sounds like there are other
reliability problems that should be attacked.

Also I understand that it's a problem that fsck takes 8 hours to run.
However, why on earth is fsck running in the first place?  Do the
servers go down unexpectedly on regular basis?  This also screams of a
more fundamental problem that should be solved.

Another thing: I've seen several times already PHP warnings show up
about pg_query not getting a good connection or similar problems.  Isn't
normal advice to check return codes before even trying to use the
connections?  Why isn't this done in the main postgresql.org code, where
anyone can see it, is beyond me.  Of course the solution to the
underlying problem is to restart the Postgres server, but why should we
inform the user that Postgres' own database server is down, in the worst
possible way?

Just random thoughts ...

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Hay dos momentos en la vida de un hombre en los que no debería
especular: cuando puede permitírselo y cuando no puede" (Mark Twain)


Re: [pgsql-www] Server unreliability

From
Bruce Momjian
Date:
Devrim GUNDUZ wrote:
> I hope we'll not begin a OS/distro war! (O.K, it'd be good if Red Hat
> contributes RHEL to PostgreSQL project ;) Anyone from Red Hat in here, or
> in -core? ;) )

I have to say I am about as pro-*BSD as they come, but if Linux is what
it will take to make things better, I am ready to try it.  If it takes
SCO Unixware, I am ready to try it.  However, I think Marc has pointed
out some less drastic changes that will fix the fsck problem.  My point
is that the fsck problem has gone on too long and it is time to address
it head-on.  Let's take the approach with the best possibility of
success and implement it.

> > o  Web site
> >
> > We have been talking about a new web page layout for years at this
> > point.  I almost don't care if they just put a dancing bear up on the
> > web site.  Let's do something!
>
> Like this one? : http://www.jensummerfield.supanet.com/bearaniy.gif

Yes, and its animated too.  Everyone knows animation on a web site
spices it up, as does blinking text and text that changes color.  :-)

> Well, web team (mostly Alexey) is working on it. We have a fabulous new
> design (you know...). I think it'll be ready before 8.0.0. See the -www
> archives for that (Archives? Umm maybe Google...)

Yes, but how far is it from the image I saw to the completed site?  I
have no idea, and why has there been so little progress in the past few
years?

> > o  Archives
> >
> > The archives situation is a continual problem.  Again, maybe a dancing
> > bear can help.  :-)
>
> :-) A Google search box could be better sometimes...

Yes, except Google has decided to stop archiving our groups since
September 16.  We reported this on the www list, I think.  I think they
had the best archive interface, and now it is gone.  Take a look:

    http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&c2coff=1&group=comp.databases.postgresql.hackers

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Server unreliability

From
David Fetter
Date:
On Wed, Sep 29, 2004 at 04:08:03PM -0400, Bruce Momjian wrote:
> Marc G. Fournier wrote:
> > On Wed, 29 Sep 2004, Mitch Pirtle wrote:
> >
> > > Hi Bruce, we met in Manhattan for your NYPHP presentation - I
> > > was the lone PostgreSQL cheerleader in the back ;-)
> > >
> > > You ask for proposals, no matter how radical, so here goes:
> > >
> > > o  Location of servers
> > >
> > > Pending bandwidth requirements, I am willing to sponsor hosting
> > > of these sites, possibly on dedicated hardware.  But I need some
> > > ideas of bandwidth usage so I can figure out which strategies
> > > make the most sense financially.  BTW be glad your servers are
> > > not in Florida, I have one client that is in a significant
> > > amount of pain right now.
> >
> > The server location is not changing, but thank you for your offer
> > ...
> >
> > > o  FreeBSD
> > >
> > > I'm agnostic in this regard, but have experienced that Linux
> > > distros seem to be the easiest to maintain (Debian, Fedora).
> >
> > Again, operating system isn't changing, but thank you for your
> > offer ...
>
> I assume you mean "not changing" while you are administering the
> servers.  I don't remember any community agreement that the current
> setup was permanent, and my thinking now is that all avenues are
> open for exploration.
>
> I am not ready to dismiss this offer/idea.

Bruce, it isn't just you who finds this airy dismissal ridiculous.  In
fact, the only person who's acting like this is dismissed out of hand
is Marc.

Marc, you've been a great contributor to the community, but lately,
you've gotten quite severely off track.  You can continue to help by
straightening up and flying right, but you are, as you have been
reminded publicly in the past, *not* the dictator on this project.
You are not indispensable, and if you continue to pull this nonsense,
you *will* be dispensed with.

Sincerely,
David.
--
David Fetter david@fetter.org http://fetter.org/
phone: +1 510 893 6100   mobile: +1 415 235 3778

Remember to vote!

Re: Server unreliability

From
Bruce Momjian
Date:
Darcy Buskermolen wrote:
> On September 29, 2004 10:13 am, you wrote:
> > It is my opinion that we have to make major changes in the way we
> > provide hosting for our servers.  There are several problems:
> >
> > o  Location of servers
> >
> > The location of our servers in Panama is a problem.  They are too far
> > for any PostgreSQL maintainers to access.  Changing hardware or
> > diagnosing problems has been too hard.  I have had like 2 days of
> > downtime on my home machine in the past 12 years.  We have had more than
> > 2 days of downtime in the past 6 months.  My wife would not accept such
> > a reliability level.
>
> Yes location is a bit of a concern, but I think it's muchless of a concern
> than the rest of the issues. I think with hardware/jail issues sorted out
> this one will for the most part take careof it's self, or at least be far
> less apparent.

I agree, and Marc has an idea for remote administration too.

> >
> > o  FreeBSD
> >
> > The use of FreeBSD jails can cause servers to take +8 hours to fsck on a
> > server crash or power failure.  Again, I would never accept such
> > problems on my home server so it is hard to fathom how a project with
> > thousands of users can accept that.  Either we need to find a fix, stop
> > using jails, or get another operating system, but continuing to use a
> > setup with a known problem is just asking for trouble.
>
> The over use of jails I think is complicating things, what are the odds that
> someone could find it in their heart to raid the old hardware pile in their
> serverback room, and come up with a PIII 800MHz + system with a gig of ram
> that they could donate to the cause, and we move some of the more critical
> services (www/ftp/anoncvs) to that box without jails.

I have a dual PIII 550MHz here that I can give away.  It was my previous
server.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Server unreliability

From
Josh Berkus
Date:
David,

> Marc, you've been a great contributor to the community, but lately,
> you've gotten quite severely off track.  You can continue to help by
> straightening up and flying right, but you are, as you have been
> reminded publicly in the past, *not* the dictator on this project.
> You are not indispensable, and if you continue to pull this nonsense,
> you *will* be dispensed with.

Hey!   Take it easy!    Everyone here has been working hard to promote and
support PostgreSQL, particularly Marc, and taking a superior tone does
nothing to improve the real problems.

Even if we were donated a 5-server data center at Globix with unlimited
bandwidth tommorrow, Marc would still he helping us with list/server admin,
and it would still be the case that Hub.org has supported this project for
years, including periods in which there was no/little benefit in doing so.

--
--Josh

Josh Berkus
Aglio Database Solutions
San Francisco

Re: [pgsql-www] Server unreliability

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Yes, but how far is it from the image I saw to the completed site?  I
> have no idea, and why has there been so little progress in the past few
> years?

Lack of people doing the work is why.  Complaining that nothing is
getting done is just sucking time away from more productive activities
... and I think that extends to all the points made in this thread.
We all know perfectly well what the problems are --- it's a matter of
finding time, and sometimes money, to solve them.  We don't need "new
ideas" unless the ideas are about how to produce those resources.

            regards, tom lane

Re: [pgsql-www] Server unreliability

From
Bruce Momjian
Date:
Marc G. Fournier wrote:
> On Wed, 29 Sep 2004, Bruce Momjian wrote:
>
> > It is my opinion that we have to make major changes in the way we
> > provide hosting for our servers.  There are several problems:
> >
> > o  Location of servers
> >
> > The location of our servers in Panama is a problem.  They are too far
> > for any PostgreSQL maintainers to access.  Changing hardware or
> > diagnosing problems has been too hard.  I have had like 2 days of
> > downtime on my home machine in the past 12 years.  We have had more than
> > 2 days of downtime in the past 6 months.  My wife would not accept such
> > a reliability level.
>
> This is currently being worked on ... we are looking at various remote
> management solutions so that we don't have to deal with waiting for a
> technician to get 'on the scene' ...

OK.

> > o  FreeBSD
> >
> > The use of FreeBSD jails can cause servers to take +8 hours to fsck on a
> > server crash or power failure.  Again, I would never accept such
> > problems on my home server so it is hard to fathom how a project with
> > thousands of users can accept that.  Either we need to find a fix, stop
> > using jails, or get another operating system, but continuing to use a
> > setup with a known problem is just asking for trouble.
>
> Actually, again, this one is being addressed ... there is a solution in
> the pipeline to fix the cause of the 8+ hour fsck, but, since it is a fix
> to fsck itself, it hasn't been put into the mainstream code yet, due to
> *obvious* testing reasons ...

Even if it is experimental, I think we should try it.  But the larger
issue is why we are having OS crashes in the first place.  Not all the
fsck downtime was because of power failure.   What are the causes of the
other downtime, and if we don't know or can't fix it, I think we need to
look at more dramatic changes to increase reliability.

> We've also added in hot failover as an option ... I've posted to -www
> asking about putting www.postgresql.org onto it, but so far, the only
> responses back have been along the lines of 'how are you doing it?' ...
>
> The "risk" is that its not real time replication between the live and
> failover server ... on our high performance servers, the 'delay' is about
> 5 minutes ...
>
> ... now, knowing that, if you feel comfortable with me putting this onto
> the mailing lists/cvsroot as well, knowing that there is a possibility of
> something being written 'in the gap' before failover, I'll do that VM also
> ...
>
> note that altho the replication has a gap, the heartbeat process runs
> every minute ... as soon as it can't ping anymore, it fails over ...

Right, failover is nice, but we need to find the cause of why we need
the failover so often.  From my perspective, if your servers can't be as
reliable as my home machine, there is something fundementally wrong,
either in hardware purchase, hosting provider, operating system,
software infrastructure, something.  I don't know what the problem is,
but I know a problem when I see it.

> > o  Web site
> >
> > We have been talking about a new web page layout for years at this
> > point.  I almost don't care if they just put a dancing bear up on the
> > web site.  Let's do something!
>
> What's wrong with the existing one?  Have you designed the dancing bear
> you'd like us to put up in place of what we have now?

Looking around now.  Perhaps a dancing elephant.  WARNING:  This will
make you ill:

    http://janetskiles.com/ART/greeting/greet-ani/dancing-elephant.jpg

:-)

> > The archives situation is a continual problem.  Again, maybe a dancing
> > bear can help.  :-)
>
> What is wrong with it now?  I'm cleaning up the code itself, but that is
> due to it being a mess right now, not due to any problems reported to me,
> removed one of the banner ads so that loading is a bit faster, and John
> has done, I think, a fantastic job on the search engine itself, including
> sending me changes for the archives themselves so that the 'time searches'
> should now work properly ...
>
> So, do you have something specific you'd like to point out to us that
> we've overlooked and haven't fixed yet?

"It not working" seems to be a continual problem.  I don't know the
details and maybe it is already being fixed.

> > Basically, with no money and no one offering servers
>
> So far, I've had one person donate $10 ... in order to put a dedicated
> server onto the network, I'd need alot more of those ... that would pretty
> much eliminate your second point about the fsck's, since its only our
> *loaded* servers, that we have that problem with ... but, as I said, the
> fsck issue is being addressed as well ...

Yep, that is pretty pathetic.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: Server unreliability

From
Bruce Momjian
Date:
Josh Berkus wrote:
> David,
>
> > Marc, you've been a great contributor to the community, but lately,
> > you've gotten quite severely off track.  You can continue to help by
> > straightening up and flying right, but you are, as you have been
> > reminded publicly in the past, *not* the dictator on this project.
> > You are not indispensable, and if you continue to pull this nonsense,
> > you *will* be dispensed with.
>
> Hey!   Take it easy!    Everyone here has been working hard to promote and
> support PostgreSQL, particularly Marc, and taking a superior tone does
> nothing to improve the real problems.
>
> Even if we were donated a 5-server data center at Globix with unlimited
> bandwidth tommorrow, Marc would still he helping us with list/server admin,
> and it would still be the case that Hub.org has supported this project for
> years, including periods in which there was no/little benefit in doing so.

"Years" where there was no/little benefit.  We had first-class
infrastructure when other open source projects were run on servers that
moved every few months.

My problem is that our reliability has gone _down_ in the past few years
when it should have been getting better, and I am trying to find out the
cause and push us to adopt a solution.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: [pgsql-www] Server unreliability

From
Peter Eisentraut
Date:
Josh Berkus wrote:
> The big obstacle in moving anything is bandwidth.   While any number
> of companies will offer to host stuff for us, we can't match the
> amount of bandwidth we're currently using in Panama

... which amounts to exactly what?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/


Re: Server unreliability

From
elein
Date:
I took a look at the google ad site.  There is a lot more to it
than the part where you put the java script onto the page.
You have to create an account and be approved, etc. etc. etc.

--elein


On Wed, Sep 29, 2004 at 04:13:05PM -0300, Marc G. Fournier wrote:
> On Wed, 29 Sep 2004, Darcy Buskermolen wrote:
>
> >>I think you are the only 'known to work site' ... so, if I setup a round
> >>robin between the Panama and Canadian location for www.postgresql.org,
> >>would you be okay with the extra load? :)  If so, I'll get to work on the
> >>coding changing required for the DNS ...
> >
> >I have no issues with that being setup now, I'll keep na eye on the
> >bandwith
> >and let you know if it becomes problematic.
>
> 'k, will play with the DNS code tonight then ...
>
> ><script type="text/javascript"><!--
> >google_ad_client = "pub-6839445463094639";
> >google_ad_width = 120;
> >google_ad_height = 240;
> >google_ad_format = "120x240_as";
> >google_ad_channel ="";
> >google_color_border = "DFF2FD";
> >google_color_bg = "DFF2FD";
> >google_color_link = "000000";
> >google_color_url = "336699";
> >google_color_text = "333333";
> >//--></script>
> ><script type="text/javascript"
> > src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
> ></script>
>
> Sweet ... okay, will look at getting that added to archives then ...
> thanks for the pointer ...
>
> ----
> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
>      subscribe-nomail command to majordomo@postgresql.org so that your
>      message can get through to the mailing list cleanly

Re: [pgsql-www] Server unreliability

From
Peter Eisentraut
Date:
Tom Lane wrote:
> We all know perfectly well what the problems are ---
> it's a matter of finding time, and sometimes money, to solve them.
> We don't need "new ideas" unless the ideas are about how to produce
> those resources.

Throwing more resources at an entity that does not scale won't achieve
anything.  In order to improve, we need to listen to competing concepts
and pick the best one, not keep giving the current one more time over
and over.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/


Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, elein wrote:

> I took a look at the google ad site.  There is a lot more to it
> than the part where you put the java script onto the page.
> You have to create an account and be approved, etc. etc. etc.

yup, already put through the application, Darcy says it takes about 48
hours or so to be approved ...
>
> --elein
>
>
> On Wed, Sep 29, 2004 at 04:13:05PM -0300, Marc G. Fournier wrote:
>> On Wed, 29 Sep 2004, Darcy Buskermolen wrote:
>>
>>>> I think you are the only 'known to work site' ... so, if I setup a round
>>>> robin between the Panama and Canadian location for www.postgresql.org,
>>>> would you be okay with the extra load? :)  If so, I'll get to work on the
>>>> coding changing required for the DNS ...
>>>
>>> I have no issues with that being setup now, I'll keep na eye on the
>>> bandwith
>>> and let you know if it becomes problematic.
>>
>> 'k, will play with the DNS code tonight then ...
>>
>>> <script type="text/javascript"><!--
>>> google_ad_client = "pub-6839445463094639";
>>> google_ad_width = 120;
>>> google_ad_height = 240;
>>> google_ad_format = "120x240_as";
>>> google_ad_channel ="";
>>> google_color_border = "DFF2FD";
>>> google_color_bg = "DFF2FD";
>>> google_color_link = "000000";
>>> google_color_url = "336699";
>>> google_color_text = "333333";
>>> //--></script>
>>> <script type="text/javascript"
>>> src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
>>> </script>
>>
>> Sweet ... okay, will look at getting that added to archives then ...
>> thanks for the pointer ...
>>
>> ----
>> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
>> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 3: if posting/reading through Usenet, please send an appropriate
>>      subscribe-nomail command to majordomo@postgresql.org so that your
>>      message can get through to the mailing list cleanly
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: [pgsql-www] Server unreliability

From
Josh Berkus
Date:
Marc,

> yup, already put through the application, Darcy says it takes about 48
> hours or so to be approved ...

Um, can we vote on this?  It's my long-term goal to get rid of all ads on
postgresql.org, so I'd like to hunt for other funding sources before doing
this.

--
--Josh

Josh Berkus
Aglio Database Solutions
San Francisco

Re: [pgsql-www] Server unreliability

From
Devrim GUNDUZ
Date:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi,

On Wed, 29 Sep 2004, Bruce Momjian wrote:

>> :-) A Google search box could be better sometimes...
>
> Yes, except Google has decided to stop archiving our groups since
> September 16.  We reported this on the www list, I think.  I think they
> had the best archive interface, and now it is gone.  Take a look:
>
>     http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&c2coff=1&group=comp.databases.postgresql.hackers

Dave has contacted with Google, and I'm sure that they'll solve this
problem. See:

http://archives.postgresql.org/pgsql-www/2004-09/msg00191.php

Regards,
- --
Devrim GUNDUZ
devrim~gunduz.org                devrim.gunduz~linux.org.tr
             http://www.tdmsoft.com
             http://www.gunduz.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFBWzD8tl86P3SPfQ4RAiiYAJ9JOO2n0E55slW8e8STLCGfej4P8ACg19zk
6smnsBQgIUxTWRDlj9TimHE=
=JCZW
-----END PGP SIGNATURE-----

Re: [pgsql-www] Server unreliability

From
Alvaro Herrera
Date:
On Thu, Sep 30, 2004 at 01:02:33AM +0300, Devrim GUNDUZ wrote:

> Dave has contacted with Google, and I'm sure that they'll solve this
> problem. See:
>
> http://archives.postgresql.org/pgsql-www/2004-09/msg00191.php

Now that people are talking about www and archives, please take a look
at a problem I've pointed a couple of times in the past and had no
answer: the pgsql-es-ayuda list is not on the left column.  It's also
not listed on the " Search in: " combo box at the top either.

http://archives.postgresql.org/pgsql-www/2004-09/msg00191.php

Why is this?  Any chance to correct it?

Note that, of the regional lists, this is the one with the most traffic,
and the oldest (don't be fooled by the length of the month list ... the
last item has articles going back to 1997, it's just badly built)

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Cuando no hay humildad las personas se degradan" (A. Christie)


Re: Server unreliability

From
Josh Berkus
Date:
Marc,

> I got fast approval on the AdSense stuff from Google, so am working this
> evening on implementing that ... should help some ...

Let's please discuss this.   It will piss people off, and will disrupt the
attempt to move to the new webserver code.

--Josh

--
__Aglio Database Solutions_______________
Josh Berkus               Consultant
josh@agliodbs.com     www.agliodbs.com
Ph: 415-752-2500    Fax: 415-752-2387
2166 Hayes Suite 200    San Francisco, CA

Re: [pgsql-www] Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Alvaro Herrera wrote:

> about pg_query not getting a good connection or similar problems.
> Isn't normal advice to check return codes before even trying to use the
> connections?  Why isn't this done in the main postgresql.org code, where
> anyone can see it, is beyond me.  Of course the solution to the
> underlying problem is to restart the Postgres server, but why should we
> inform the user that Postgres' own database server is down, in the worst
> possible way?

Just curious here, but when/where?  We haven't had a database issue in
quite awhile that *I'm* aware of ... in fact, the whole web site is
static, generated periodically from .php files, so that the mirrors can
pick things up properly, so there should never be a 'cannot connect to
database' issue, since there are no connections to the database being made
...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: [pgsql-www] Server unreliability

From
"Marc G. Fournier"
Date:
Fixed ...

On Wed, 29 Sep 2004, Alvaro Herrera wrote:

> On Thu, Sep 30, 2004 at 01:02:33AM +0300, Devrim GUNDUZ wrote:
>
>> Dave has contacted with Google, and I'm sure that they'll solve this
>> problem. See:
>>
>> http://archives.postgresql.org/pgsql-www/2004-09/msg00191.php
>
> Now that people are talking about www and archives, please take a look
> at a problem I've pointed a couple of times in the past and had no
> answer: the pgsql-es-ayuda list is not on the left column.  It's also
> not listed on the " Search in: " combo box at the top either.
>
> http://archives.postgresql.org/pgsql-www/2004-09/msg00191.php
>
> Why is this?  Any chance to correct it?
>
> Note that, of the regional lists, this is the one with the most traffic,
> and the oldest (don't be fooled by the length of the month list ... the
> last item has articles going back to 1997, it's just badly built)
>
> --
> Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
> "Cuando no hay humildad las personas se degradan" (A. Christie)
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
>      subscribe-nomail command to majordomo@postgresql.org so that your
>      message can get through to the mailing list cleanly
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Server unreliability

From
"Greg Sabino Mullane"
Date:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> Just curious here, but when/where?  We haven't had a database issue in
> quite awhile that *I'm* aware of

Earlier this very day I got a page full of php erros when I tried
to view one of the mailing list "by thread". I would have wrote down
the errors and reported it. but it seems so par for the course I
did not bother.

My other concern is that often myself or someone else points out these
errors on the list, and *then* they are fixed. They should be caught
before this, either by testing by the people who make the changes
(which I've seen fail to happen), or by some sort of automated script
that does some quick basic checks.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200409291947

-----BEGIN PGP SIGNATURE-----




Re: Server unreliability

From
Mark Harrison
Date:
Bruce Momjian wrote:
> It is my opinion that we have to make major changes in the way we
> provide hosting for our servers.  There are several problems:
>
> o  Location of servers
>
> The location of our servers in Panama is a problem.  They are too far
> for any PostgreSQL maintainers to access.  Changing hardware or
> diagnosing problems has been too hard.  I have had like 2 days of
> downtime on my home machine in the past 12 years.  We have had more than
> 2 days of downtime in the past 6 months.  My wife would not accept such
> a reliability level.
>
> o  FreeBSD
>
> The use of FreeBSD jails can cause servers to take +8 hours to fsck on a
> server crash or power failure.  Again, I would never accept such
> problems on my home server so it is hard to fathom how a project with
> thousands of users can accept that.  Either we need to find a fix, stop
> using jails, or get another operating system, but continuing to use a
> setup with a known problem is just asking for trouble.
>
> o  Web site
>
> We have been talking about a new web page layout for years at this
> point.  I almost don't care if they just put a dancing bear up on the
> web site.  Let's do something!
>
> o  Archives
>
> The archives situation is a continual problem.  Again, maybe a dancing
> bear can help.  :-)
>
> Basically, with no money and no one offering servers, I don't see a good
> solution to any of these problems, but I think we need to recognize
> these are problems and that we will continue to suffer until they are
> addressed.
>


> Are there any proposals, no matter how radical, to correct these?

Here are a few radical ideas... The basic idea is to leverage as many
outside resources as possible.

1.  Move primary file downloads to sourceforge.  That should reduce the
     traffic load a lot.

2.  Move the mailing lists to a mailing list server (e.g. yahoo).

3.  Base the new website on a blog-style package that makes it
     easy to update for multiple people (disclaimer, I don't know
     what the current site is running, so maybe that's already
     been done).

4.  For the website, concentrate on simple static pages that can
     be served efficiently.

5.  I like the google ads idea, it will possibly generate a bit
     of money and connect people with organizations interested
     in postgresql.

All just IMHO, and with immense gratitude for all the work
that's already been done!

Mark

--
Mark Harrison
Pixar Animation Studios

Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Josh Berkus wrote:

> Marc,
>
>> I got fast approval on the AdSense stuff from Google, so am working this
>> evening on implementing that ... should help some ...
>
> Let's please discuss this.   It will piss people off, and will disrupt the
> attempt to move to the new webserver code.

As I mentioned in IRC, just so that everyone knows ... this is *only* for
the archives, not the main site ... and there is no "contract that I have
to put it up", so it can come down at a later date if we don't want it
anymore ... but, there was no thought/intention of it for any of the other
sites, only to tap into the archives site itself ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: [pgsql-www] Server unreliability

From
Alvaro Herrera
Date:
On Wed, Sep 29, 2004 at 08:35:51PM -0300, Marc G. Fournier wrote:
>
> Fixed ...

Thanks ... there is a typo though: on the upper menu, the combo box says
"German" where it should say "Spanish".

> On Wed, 29 Sep 2004, Alvaro Herrera wrote:
>
> >On Thu, Sep 30, 2004 at 01:02:33AM +0300, Devrim GUNDUZ wrote:
> >
> >>Dave has contacted with Google, and I'm sure that they'll solve this
> >>problem. See:
> >>
> >>http://archives.postgresql.org/pgsql-www/2004-09/msg00191.php
> >
> >Now that people are talking about www and archives, please take a look
> >at a problem I've pointed a couple of times in the past and had no
> >answer: the pgsql-es-ayuda list is not on the left column.  It's also
> >not listed on the " Search in: " combo box at the top either.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"God is real, unless declared as int"


Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Greg Sabino Mullane wrote:

>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>> Just curious here, but when/where?  We haven't had a database issue in
>> quite awhile that *I'm* aware of
>
> Earlier this very day I got a page full of php erros when I tried
> to view one of the mailing list "by thread". I would have wrote down
> the errors and reported it. but it seems so par for the course I
> did not bother.

My fault, sorry ... I've been going through and cleaning up the code,
removed the right banner ad, and missed an include directory :(

> My other concern is that often myself or someone else points out these
> errors on the list, and *then* they are fixed. They should be caught
> before this, either by testing by the people who make the changes
> (which I've seen fail to happen), or by some sort of automated script
> that does some quick basic checks.

In your case, you caught it between edits ... again, sorry about that ...
the main web site itself is tested on a beta site first, so if you ever
see an error there, its definitely not from lack of testing by the -www
crew ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: [pgsql-www] Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Alvaro Herrera wrote:

> On Wed, Sep 29, 2004 at 08:35:51PM -0300, Marc G. Fournier wrote:
>>
>> Fixed ...
>
> Thanks ... there is a typo though: on the upper menu, the combo box says
> "German" where it should say "Spanish".

Oops, sorry, was even looking for that and missed it :(

Fixed now ...


>
>> On Wed, 29 Sep 2004, Alvaro Herrera wrote:
>>
>>> On Thu, Sep 30, 2004 at 01:02:33AM +0300, Devrim GUNDUZ wrote:
>>>
>>>> Dave has contacted with Google, and I'm sure that they'll solve this
>>>> problem. See:
>>>>
>>>> http://archives.postgresql.org/pgsql-www/2004-09/msg00191.php
>>>
>>> Now that people are talking about www and archives, please take a look
>>> at a problem I've pointed a couple of times in the past and had no
>>> answer: the pgsql-es-ayuda list is not on the left column.  It's also
>>> not listed on the " Search in: " combo box at the top either.
>
> --
> Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
> "God is real, unless declared as int"
>
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Server unreliability

From
"Greg Sabino Mullane"
Date:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>> Basically, with no money and no one offering servers

> So far, I've had one person donate $10 ... in order to put a dedicated
> server onto the network, I'd need alot more of those

FWIW, I've been more than willing to put my money where my mouth is,
but as long as we have banner ads all over out site, I'm not willing
to provide money. I seem to recall that people offered to offset the
costs of the banner ads earlier, but the discusion never got anywhere,
mostly because nobody could produce the amount that the banner ads
brought in.

As far as suggestions, why not all chip in and get a nice dedicated server
in a centralized place (e.g. Washington DC), get it up and running,
and migrate things over one by one.


- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200409292037

-----BEGIN PGP SIGNATURE-----

iD8DBQFBW16evJuQZxSWSsgRAoaVAJwJy/6UfZsqc4CBimYybiGyeCPA/QCdFqD3
zzKVQCcnPLyPN93f7GwiIFA=
=30J6
-----END PGP SIGNATURE-----



Re: Server unreliability

From
Richard Welty
Date:
On Wed, 29 Sep 2004 20:56:11 -0300 (ADT) "Marc G. Fournier" <scrappy@postgresql.org> wrote:
> As I mentioned in IRC, just so that everyone knows ... this is only for
> the archives, not the main site ... and there is no "contract that I have
> to put it up", so it can come down at a later date if we don't want it
> anymore ... but, there was no thought/intention of it for any of the other
> sites, only to tap into the archives site itself ...

i've used google adsense on one of my web sites since last january,
and in general the ads are well targeted, tasteful, and shouldn't tick
anyone off unless MySQL AB makes a big ad buy with PostgreSQL
as the targeted keyword.

richard
--
Richard Welty                                         rwelty@averillpark.net
Averill Park Networking                                         518-573-7592
    Java, PHP, PostgreSQL, Unix, Linux, IP Network Engineering, Security

Re: Server unreliability

From
Mitch Pirtle
Date:
On Thu, 30 Sep 2004 01:16:26 -0000, Greg Sabino Mullane
<greg@turnstep.com> wrote:
>
> > So far, I've had one person donate $10 ... in order to put a dedicated
> > server onto the network, I'd need alot more of those

Then perhaps a different approach needs to be taken?

> FWIW, I've been more than willing to put my money where my mouth is,
> but as long as we have banner ads all over out site, I'm not willing
> to provide money. I seem to recall that people offered to offset the
> costs of the banner ads earlier, but the discusion never got anywhere,
> mostly because nobody could produce the amount that the banner ads
> brought in.

I use Adblock for FireFox, and as a Gen Xer I don't see them anyway ;-)

> As far as suggestions, why not all chip in and get a nice dedicated server
> in a centralized place (e.g. Washington DC), get it up and running,
> and migrate things over one by one.

This was my thought initially, and servers themselves are actually
quite cheap (at ~$99 each) - that is why I asked about bandwidth
usage, as I believe this would be the cost factor.

For example, Server Beach (www.serverbeach.com) gives you a dedicated
machine for $99 monthly, and despite them being single processor
machines, we could have a dedicated webserver, dedicated database
server, and dedicated ftp/mail/whatever for ~$300...

I'm getting dual-xeons at aPlus.net for $200 or so, but I don't know
if PostgreSQL would need that kind of hardware if we could have
several servers.

Thoughts?  Discuss.  ;-)

Re: Server unreliability

From
"Marc G. Fournier"
Date:
On Wed, 29 Sep 2004, Richard Welty wrote:

> On Wed, 29 Sep 2004 20:56:11 -0300 (ADT) "Marc G. Fournier" <scrappy@postgresql.org> wrote:
>> As I mentioned in IRC, just so that everyone knows ... this is only for
>> the archives, not the main site ... and there is no "contract that I have
>> to put it up", so it can come down at a later date if we don't want it
>> anymore ... but, there was no thought/intention of it for any of the other
>> sites, only to tap into the archives site itself ...
>
> i've used google adsense on one of my web sites since last january,
> and in general the ads are well targeted, tasteful, and shouldn't tick
> anyone off unless MySQL AB makes a big ad buy with PostgreSQL
> as the targeted keyword.

ya, and we're only putting it on the archives themselves, not the main
site ... and the main site itself, with the new redesign will have one
banner ad, on the right hand side (if i recall right the positioning) ...
similar to how php.net has their banner ...

archives itself is already down to the one banner at the top left and the
AdSense stuff ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: [pgsql-www] Server unreliability

From
Alvaro Herrera
Date:
On Wed, Sep 29, 2004 at 09:34:21PM -0300, Marc G. Fournier wrote:
> On Wed, 29 Sep 2004, Alvaro Herrera wrote:
>
> >On Wed, Sep 29, 2004 at 08:35:51PM -0300, Marc G. Fournier wrote:
> >>
> >>Fixed ...
> >
> >Thanks ... there is a typo though: on the upper menu, the combo box says
> >"German" where it should say "Spanish".
>
> Oops, sorry, was even looking for that and missed it :(
>
> Fixed now ...

Thanks.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Escucha y olvidarás; ve y recordarás; haz y entenderás" (Confucio)


Re: Server unreliability - Solutions

From
"Joshua D. Drake"
Date:
Hello,

>o  Archives
>
>The archives situation is a continual problem.  Again, maybe a dancing
>bear can help.  :-)
>
>
This  is being resolved by Command Prompt. Marc and I are in the process
of moving archives as we speak.
In fact DNS should be propagating by morning.

>Basically, with no money and no one offering servers, I don't see a good
>solution to any of these problems, but I think we need to recognize
>these are problems and that we will continue to suffer until they are
>addressed.
>
>
>
Command Prompt has offered servers and bandwidth on Multiple occassions.
We have top notch
redundant bandwidth, natural gas fired generators, and solid uptime.

>Are there any proposals, no matter how radical, to correct these?
>
>
>
Please see above.

Sincerely,

Joshua D. Drake





--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com
PostgreSQL Replicator -- production quality replication for PostgreSQL


Re: Server unreliability

From
Devrim GUNDUZ
Date:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi,

On Wed, 29 Sep 2004, Mark Harrison wrote:

> 1.  Move primary file downloads to sourceforge.  That should reduce the
>    traffic load a lot.
>
> 2.  Move the mailing lists to a mailing list server (e.g. yahoo).

Are you kidding?

> 4.  For the website, concentrate on simple static pages that can
>    be served efficiently.

That's what we are doing now...

Regards,
- --
Devrim GUNDUZ
devrim~gunduz.org                devrim.gunduz~linux.org.tr
             http://www.tdmsoft.com
             http://www.gunduz.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD4DBQFBW7Nytl86P3SPfQ4RAtdFAJdpoosF8oW3c4nCyK/Gc2AcibFBAKCIkFGK
9d3qctBlMKOfH8vXu3xhiQ==
=18UW
-----END PGP SIGNATURE-----

Re: [pgsql-www] Server unreliability

From
Alvaro Herrera
Date:
On Wed, Sep 29, 2004 at 08:24:31PM -0300, Marc G. Fournier wrote:
> On Wed, 29 Sep 2004, Alvaro Herrera wrote:
>
> >about pg_query not getting a good connection or similar problems.
> >Isn't normal advice to check return codes before even trying to use the
> >connections?  Why isn't this done in the main postgresql.org code, where
> >anyone can see it, is beyond me.  Of course the solution to the
> >underlying problem is to restart the Postgres server, but why should we
> >inform the user that Postgres' own database server is down, in the worst
> >possible way?
>
> Just curious here, but when/where?

I don't remember an exact date or place, just that I've seen that happen
more than three times and I've always thought it was bad programming
practice.  It has been very annoying, mainly because it means casual
visitors could see those and possible think Postgres reliability sucked.
Perhaps it has been fixed since then.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"The important things in the world are problems with society that we don't
understand at all. The machines will become more complicated but they won't
be more complicated than the societies that run them."    (Freeman Dyson)


Re: [pgsql-www] Server unreliability

From
Alvaro Herrera
Date:
On Wed, Sep 29, 2004 at 06:20:37PM -0400, Alvaro Herrera wrote:

> Now that people are talking about www and archives, please take a look
> at a problem I've pointed a couple of times in the past and had no
> answer: the pgsql-es-ayuda list is not on the left column.  It's also
> not listed on the " Search in: " combo box at the top either.

Oh, another one:  Here


http://search.postgresql.org/archives.search?cs=utf-8&fm=on&st=20&dt=back&q=llave+foranea&ul=http%3A%2F%2Farchives.postgresql.org%2Fpgsql-es-ayuda%2F%25&dp=0&o=0&ps=10&s=date

the pgsql-es-ayuda is not in the upper combo box.  This is the page
that's presented as result from a search (it was originally restricted
to pgsql-es-ayuda).

It seems really weird to have two lists of lists, one at the top when
you are looking at plain archives, and other when you are looking at
search results.  Most certainly they should be identical!  Why aren't
they one and the same?

Thanks,

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
www.google.com: interfaz de línea de comando para la web.


Re: [pgsql-www] Server unreliability

From
"Marc G. Fournier"
Date:
On Thu, 30 Sep 2004, Alvaro Herrera wrote:

> On Wed, Sep 29, 2004 at 06:20:37PM -0400, Alvaro Herrera wrote:
>
>> Now that people are talking about www and archives, please take a look
>> at a problem I've pointed a couple of times in the past and had no
>> answer: the pgsql-es-ayuda list is not on the left column.  It's also
>> not listed on the " Search in: " combo box at the top either.
>
> Oh, another one:  Here
>
>
http://search.postgresql.org/archives.search?cs=utf-8&fm=on&st=20&dt=back&q=llave+foranea&ul=http%3A%2F%2Farchives.postgresql.org%2Fpgsql-es-ayuda%2F%25&dp=0&o=0&ps=10&s=date
>
> the pgsql-es-ayuda is not in the upper combo box.  This is the page
> that's presented as result from a search (it was originally restricted
> to pgsql-es-ayuda).
>
> It seems really weird to have two lists of lists, one at the top when
> you are looking at plain archives, and other when you are looking at
> search results.  Most certainly they should be identical!  Why aren't
> they one and the same?

search is on a different server then archives ... we try and keep the
search box in sync, but John most likely didn't see your posting about it
yesterday, and I haven't had a chance to update him ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: [pgsql-www] Server unreliability

From
"Marc G. Fournier"
Date:
On Thu, 30 Sep 2004, Alvaro Herrera wrote:

> I don't remember an exact date or place, just that I've seen that happen
> more than three times and I've always thought it was bad programming
> practice.  It has been very annoying, mainly because it means casual
> visitors could see those and possible think Postgres reliability sucked.
> Perhaps it has been fixed since then.

If you see it, please report which URL you see it on ... there has been
alot of code cleanups happening, especially with Alexey's work on the
backend code, but its possible that something has been missed ... you are
right that it doesn't look good when visitors see it, but if nobody
reports it, its not goign to get fixed either ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: [pgsql-www] Server unreliability

From
Gavin Sherry
Date:
On Wed, 29 Sep 2004, Marc G. Fournier wrote:

> On Wed, 29 Sep 2004, Alvaro Herrera wrote:
>
> > about pg_query not getting a good connection or similar problems.
> > Isn't normal advice to check return codes before even trying to use the
> > connections?  Why isn't this done in the main postgresql.org code, where
> > anyone can see it, is beyond me.  Of course the solution to the
> > underlying problem is to restart the Postgres server, but why should we
> > inform the user that Postgres' own database server is down, in the worst
> > possible way?
>
> Just curious here, but when/where?  We haven't had a database issue in
> quite awhile that *I'm* aware of ... in fact, the whole web site is
> static, generated periodically from .php files, so that the mirrors can
> pick things up properly, so there should never be a 'cannot connect to
> database' issue, since there are no connections to the database being made

I know this isn't on the main site but:

http://www.alcove.com.au/~swm/gborg.png

Gavin

Re: Server unreliability

From
Andrew Sullivan
Date:
On Wed, Sep 29, 2004 at 04:56:25PM -0700, Mark Harrison wrote:
> 1.  Move primary file downloads to sourceforge.  That should reduce the
>     traffic load a lot.

I'm coming to this discussion late, but given the project's history
with respect to SourceForge, I feel _extremely_ uncomfortable
depending on them for anything.

A

--
Andrew Sullivan  | ajs@crankycanuck.ca
The fact that technology doesn't work is no bar to success in the marketplace.
        --Philip Greenspun