Thread: Gborg still down

Gborg still down

From
"Greg Sabino Mullane"
Date:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Can someone give an update as to what is happening on gborg?

And how long until it is working again? Even if not the
www portion, the cvs would be nice to have back up.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200604121228
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iD8DBQFEPSrOvJuQZxSWSsgRArqqAKD86I/zUGnsI3QKI10ojkQ/EEIHFwCg6cn9
BlHsYoXPsjhGf44D3Ud8R0Y=
=SSf/
-----END PGP SIGNATURE-----



Re: Gborg still down

From
Josh Berkus
Date:
Greg,

> Can someone give an update as to what is happening on gborg?
>
> And how long until it is working again? Even if not the
> www portion, the cvs would be nice to have back up.

This reminds me, I'm looking at doing the Final Migration off GBorg (as
commanded by Bruce) in May.

--Josh

Re: Gborg still down

From
"Dave Page"
Date:
-----Original Message-----
From: "Greg Sabino Mullane"<greg@turnstep.com>
Sent: 12/04/06 17:29:24
To: "pgsql-www@postgresql.org"<pgsql-www@postgresql.org>
Subject: [pgsql-www] Gborg still down

> Can someone give an update as to what is happening on gborg?

I cannot fix it now as I'm on a pda, but I haven't seen any reports of problems for weeks. What broke & when?

Regards, Dave

-----Unmodified Original Message-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Can someone give an update as to what is happening on gborg?

And how long until it is working again? Even if not the
www portion, the cvs would be nice to have back up.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200604121228
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iD8DBQFEPSrOvJuQZxSWSsgRArqqAKD86I/zUGnsI3QKI10ojkQ/EEIHFwCg6cn9
BlHsYoXPsjhGf44D3Ud8R0Y=
=SSf/
-----END PGP SIGNATURE-----



---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

               http://archives.postgresql.org


Re: Gborg still down

From
"Marc G. Fournier"
Date:
thanks for letting me know its down ... everything is back up now ...

On Wed, 12 Apr 2006, Greg Sabino Mullane wrote:

>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> Can someone give an update as to what is happening on gborg?
>
> And how long until it is working again? Even if not the
> www portion, the cvs would be nice to have back up.
>
> - --
> Greg Sabino Mullane greg@turnstep.com
> PGP Key: 0x14964AC8 200604121228
> http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
> -----BEGIN PGP SIGNATURE-----
>
> iD8DBQFEPSrOvJuQZxSWSsgRArqqAKD86I/zUGnsI3QKI10ojkQ/EEIHFwCg6cn9
> BlHsYoXPsjhGf44D3Ud8R0Y=
> =SSf/
> -----END PGP SIGNATURE-----
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
>               http://archives.postgresql.org
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Gborg still down

From
"Marc G. Fournier"
Date:
On Wed, 12 Apr 2006, Dave Page wrote:

>
> -----Original Message-----
> From: "Greg Sabino Mullane"<greg@turnstep.com>
> Sent: 12/04/06 17:29:24
> To: "pgsql-www@postgresql.org"<pgsql-www@postgresql.org>
> Subject: [pgsql-www] Gborg still down
>
>> Can someone give an update as to what is happening on gborg?
>
> I cannot fix it now as I'm on a pda, but I haven't seen any reports of
> problems for weeks. What broke & when?

neptune rebooted last night, and, for some reasons, some vServers didn't
restart, altho their IPs did get configured ... thought I had gotten all
of the ones that were down, but obviously I missed this one :(

  >
> Regards, Dave
>
> -----Unmodified Original Message-----
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> Can someone give an update as to what is happening on gborg?
>
> And how long until it is working again? Even if not the
> www portion, the cvs would be nice to have back up.
>
> - --
> Greg Sabino Mullane greg@turnstep.com
> PGP Key: 0x14964AC8 200604121228
> http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
> -----BEGIN PGP SIGNATURE-----
>
> iD8DBQFEPSrOvJuQZxSWSsgRArqqAKD86I/zUGnsI3QKI10ojkQ/EEIHFwCg6cn9
> BlHsYoXPsjhGf44D3Ud8R0Y=
> =SSf/
> -----END PGP SIGNATURE-----
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
>               http://archives.postgresql.org
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Gborg still down

From
"Jim C. Nasby"
Date:
On Wed, Apr 12, 2006 at 03:43:24PM -0300, Marc G. Fournier wrote:
> neptune rebooted last night, and, for some reasons, some vServers didn't
> restart, altho their IPs did get configured ... thought I had gotten all
> of the ones that were down, but obviously I missed this one :(

Do we not have anything that monitors the availability of our various
websites??
--
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461

Re: Gborg still down

From
"Joshua D. Drake"
Date:
Jim C. Nasby wrote:
> On Wed, Apr 12, 2006 at 03:43:24PM -0300, Marc G. Fournier wrote:
>> neptune rebooted last night, and, for some reasons, some vServers didn't
>> restart, altho their IPs did get configured ... thought I had gotten all
>> of the ones that were down, but obviously I missed this one :(
>
> Do we not have anything that monitors the availability of our various
> websites??

The websites that CMD hosts, are monitored. I have no idea about the rest.

Joshua D. Drake

--

            === The PostgreSQL Company: Command Prompt, Inc. ===
      Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
      Providing the most comprehensive  PostgreSQL solutions since 1997
                     http://www.commandprompt.com/



Re: Gborg still down

From
"Marc G. Fournier"
Date:
On Fri, 21 Apr 2006, Joshua D. Drake wrote:

> Jim C. Nasby wrote:
>> On Wed, Apr 12, 2006 at 03:43:24PM -0300, Marc G. Fournier wrote:
>>> neptune rebooted last night, and, for some reasons, some vServers didn't
>>> restart, altho their IPs did get configured ... thought I had gotten all
>>> of the ones that were down, but obviously I missed this one :(
>>
>> Do we not have anything that monitors the availability of our various
>> websites??
>
> The websites that CMD hosts, are monitored. I have no idea about the rest.

Dave has a central nagios server setup, last I heard, for monitoring
everything ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Gborg still down

From
"Joshua D. Drake"
Date:
>> The websites that CMD hosts, are monitored. I have no idea about the
>> rest.
>
> Dave has a central nagios server setup, last I heard, for monitoring
> everything ...

Well all due respect, but as hub "Hosts the project" Why doesn't Hub
have monitoring for everything.

Joshua D. Drake


>
> ----
> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend
>


--

            === The PostgreSQL Company: Command Prompt, Inc. ===
      Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
      Providing the most comprehensive  PostgreSQL solutions since 1997
                     http://www.commandprompt.com/



Re: Gborg still down

From
"Magnus Hagander"
Date:
> >> The websites that CMD hosts, are monitored. I have no idea
> about the
> >> rest.
> >
> > Dave has a central nagios server setup, last I heard, for
> monitoring
> > everything ...
>
> Well all due respect, but as hub "Hosts the project" Why
> doesn't Hub have monitoring for everything.

Because it makes more sense monitoring internet services from *outside*
the hosting network? ;-)

(The main reason we stuck the automirror system on a network where
*nothing* else pgsql related lives..)

//Magnus

Re: Gborg still down

From
"Joshua D. Drake"
Date:
> Because it makes more sense monitoring internet services from *outside*
> the hosting network? ;-)

Well of course :) That's what we do. We have a server that sits
somewhere else entirely that does our nagios.

My main point was it seems that this is something hub should be invovled
in sense a lot of the core servers are hosted by hub.


Joshua D. Drake

>
> (The main reason we stuck the automirror system on a network where
> *nothing* else pgsql related lives..)
>
> //Magnus
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>


--

            === The PostgreSQL Company: Command Prompt, Inc. ===
      Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
      Providing the most comprehensive  PostgreSQL solutions since 1997
                     http://www.commandprompt.com/



Re: Gborg still down

From
"Dave Page"
Date:

> -----Original Message-----
> From: Joshua D. Drake [mailto:jd@commandprompt.com]
> Sent: 21 April 2006 20:05
> To: Magnus Hagander
> Cc: Marc G. Fournier; Jim C. Nasby; Dave Page;
> greg@turnstep.com; pgsql-www@postgresql.org
> Subject: Re: [pgsql-www] Gborg still down
>
>
> > Because it makes more sense monitoring internet services from
> > *outside* the hosting network? ;-)
>
> Well of course :) That's what we do. We have a server that
> sits somewhere else entirely that does our nagios.
>
> My main point was it seems that this is something hub should
> be invovled in sense a lot of the core servers are hosted by hub.

Most of the servers are not hosted by hub these days - we have them
pretty well distributed around the world these days.

Regards, Dave

Re: Gborg still down

From
"Joshua D. Drake"
Date:
> Most of the servers are not hosted by hub these days - we have them
> pretty well distributed around the world these days.

I would be happy to add our nagios server to the monitoring mix. Just
let us know what you would like us to monitor and who should be notified.

Joshua D. Drake

>
> Regards, Dave
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faq
>


--

            === The PostgreSQL Company: Command Prompt, Inc. ===
      Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
      Providing the most comprehensive  PostgreSQL solutions since 1997
                     http://www.commandprompt.com/



Re: Gborg still down

From
"Marc G. Fournier"
Date:
On Fri, 21 Apr 2006, Joshua D. Drake wrote:

>
>> Most of the servers are not hosted by hub these days - we have them
>> pretty well distributed around the world these days.
>
> I would be happy to add our nagios server to the monitoring mix. Just let us
> know what you would like us to monitor and who should be notified.

Just please don't put me onto your auto-page list again ... the last time,
I spent more time looking at false positives ... actually, they were all
false positives :(

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664

Re: Gborg still down

From
"Joshua D. Drake"
Date:
Marc G. Fournier wrote:
> On Fri, 21 Apr 2006, Joshua D. Drake wrote:
>
>>
>>> Most of the servers are not hosted by hub these days - we have them
>>> pretty well distributed around the world these days.
>>
>> I would be happy to add our nagios server to the monitoring mix. Just
>> let us know what you would like us to monitor and who should be notified.
>
> Just please don't put me onto your auto-page list again ... the last
> time, I spent more time looking at false positives ... actually, they
> were all false positives :(

We don't do ICMP monitoring anymore, nor do we do monitoring off that
old network.

We only do service level monitoring and I was assuming they would get
sent to different individuals based on what server it was.

J


>
> ----
> Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
> Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664
>


--

            === The PostgreSQL Company: Command Prompt, Inc. ===
      Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
      Providing the most comprehensive  PostgreSQL solutions since 1997
                     http://www.commandprompt.com/



Re: Gborg still down

From
"Jim C. Nasby"
Date:
On Fri, Apr 21, 2006 at 08:41:00PM +0100, Dave Page wrote:
> Most of the servers are not hosted by hub these days - we have them
> pretty well distributed around the world these days.

Well, wherever they're hosted monitoring has been an issue for as long
as I can remember. Search not getting updated, list archives not
updated, sites down, etc.

If individuals hosting these various services aren't monitoring them
adequately then the project itself should be monitoring them. Plus
there's servers/services that I suspect won't get monitored very well by
their owners (mirrors come to mind...)
--
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461

Re: Gborg still down

From
"Joshua D. Drake"
Date:
Jim C. Nasby wrote:
> On Fri, Apr 21, 2006 at 08:41:00PM +0100, Dave Page wrote:
>> Most of the servers are not hosted by hub these days - we have them
>> pretty well distributed around the world these days.
>
> Well, wherever they're hosted monitoring has been an issue for as long
> as I can remember. Search not getting updated, list archives not
> updated, sites down, etc.

Well Search has been resolved as it was moved to us and has been kicking
arse since then. Archives has not had any server side problems in longer
then I can remember.

> If individuals hosting these various services aren't monitoring them
> adequately then the project itself should be monitoring them. Plus
> there's servers/services that I suspect won't get monitored very well by
> their owners (mirrors come to mind...)

It may be helpful to understand that a lot of the servers are mirrors
(like archives) and even if archives is doing just fine the source for
archives may be having problems (which has been the case).

Joshua D. Drake



--

            === The PostgreSQL Company: Command Prompt, Inc. ===
      Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
      Providing the most comprehensive  PostgreSQL solutions since 1997
                     http://www.commandprompt.com/



Re: Gborg still down

From
Dave Page
Date:


On 21/4/06 23:45, "Jim C. Nasby" <jnasby@pervasive.com> wrote:

> On Fri, Apr 21, 2006 at 08:41:00PM +0100, Dave Page wrote:
>> Most of the servers are not hosted by hub these days - we have them
>> pretty well distributed around the world these days.
>
> Well, wherever they're hosted monitoring has been an issue for as long
> as I can remember. Search not getting updated, list archives not
> updated, sites down, etc.
>
> If individuals hosting these various services aren't monitoring them
> adequately then the project itself should be monitoring them. Plus
> there's servers/services that I suspect won't get monitored very well by
> their owners (mirrors come to mind...)

As we already said, we do monitor them. However, we monitor that they are
running - ie. Do we get a response from Apache, ssh, rsyncd or whatever.
Checking whether the archives/search are serving up to date content would be
a whole other (difficult) problem altogether.

Regards, Dave


Re: Gborg still down

From
"Jim C. Nasby"
Date:
On Sat, Apr 22, 2006 at 09:01:48AM +0100, Dave Page wrote:
> As we already said, we do monitor them. However, we monitor that they are
> running - ie. Do we get a response from Apache, ssh, rsyncd or whatever.
> Checking whether the archives/search are serving up to date content would be
> a whole other (difficult) problem altogether.

ISTM it wouldn't be that difficult if whatever updated the archives
stuffed the current time in a file somewhere once it was finished...

BTW, lest it sound like I don't appreciate the work of everyone who
keeps the various sites up, I do. But it hurts our credibility everytime
someone can't get to one of our sites, or the archives are out of date,
or search doesn't work, etc., etc. I know there's been some big
improvements made recently; I'm just hoping that we can get monitoring
setup for everything that's currently not being monitored.
--
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461

Re: Gborg still down

From
"Dave Page"
Date:
-----Original Message-----
From: "Jim C. Nasby"<jnasby@pervasive.com>
Sent: 22/04/06 19:20:03
To: "Dave Page"<dpage@vale-housing.co.uk>
Cc: "Joshua D. Drake"<jd@commandprompt.com>, "Magnus Hagander"<mha@sollentuna.net>, "Marc G.
Fournier"<scrappy@postgresql.org>,"greg@turnstep.com"<greg@turnstep.com>,
"pgsql-www@postgresql.org"<pgsql-www@postgresql.org>
Subject: Re: [pgsql-www] Gborg still down

> ISTM it wouldn't be that difficult if whatever updated the archives
> stuffed the current time in a file somewhere once it was finished...

That would prove that the mhonarc script ran and the rsync to the webserver, but it wouldn't prove that messages on all
listswere actually getting there in the first place, or subsequently being indexed for searching. 

Still, it wouldn't hurt to start with the easier stuff...

Regards Dave

Re: Gborg still down

From
"Greg Sabino Mullane"
Date:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> As we already said, we do monitor them. However, we monitor that they are
> running - ie. Do we get a response from Apache, ssh, rsyncd or whatever.
> Checking whether the archives/search are serving up to date content would be
> a whole other (difficult) problem altogether.

Well, with regards to what started this whole thread, the cvs server on
gborg was completely down (e.g. not accepting connections) so that is one
service that should definitely be added. The web server itself was up, but
only displaying a "undergoing maintanence" message. It should be simple enough
to have the monitor script download the main page and make sure that a static
string is on the page (e.g. "Welcome to Gborg!"). I agree that the archive / search
is a whole 'nother matter. I'm willing to help out on any of it though if needed,
just point me in the right direction.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200604241106
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iD8DBQFETOnpvJuQZxSWSsgRAsSZAJwOwcE0qT9jFyxKIeIXXfXV+s4+MQCghXBX
iB+ygZEs/SXwU46yEBkB8BU=
=7mnY
-----END PGP SIGNATURE-----



Re: Gborg still down

From
"Dave Page"
Date:

> -----Original Message-----
> From: pgsql-www-owner@postgresql.org
> [mailto:pgsql-www-owner@postgresql.org] On Behalf Of Greg
> Sabino Mullane
> Sent: 24 April 2006 16:09
> To: pgsql-www@postgresql.org
> Subject: Re: [pgsql-www] Gborg still down
>
>
> > As we already said, we do monitor them. However, we monitor
> that they
> > are running - ie. Do we get a response from Apache, ssh,
> rsyncd or whatever.
> > Checking whether the archives/search are serving up to date content
> > would be a whole other (difficult) problem altogether.
>
> Well, with regards to what started this whole thread, the cvs
> server on gborg was completely down (e.g. not accepting
> connections) so that is one service that should definitely be
> added.

Done.

> The web server itself was up, but only displaying a
> "undergoing maintanence" message. It should be simple enough
> to have the monitor script download the main page and make
> sure that a static string is on the page (e.g. "Welcome to
> Gborg!"). I agree that the archive / search is a whole
> 'nother matter. I'm willing to help out on any of it though
> if needed, just point me in the right direction.

You're welcome to improve the nagios checks that we do at the moment if
you like. I can give you access to the server for that of course.

Regards, Dave.

Re: Gborg still down

From
"Greg Sabino Mullane"
Date:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> You're welcome to improve the nagios checks that we do at the moment if
> you like. I can give you access to the server for that of course.

Sure, I'd like to do that.

- --
Greg Sabino Mullane greg@turnstep.com
PGP Key: 0x14964AC8 200604251129
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-----BEGIN PGP SIGNATURE-----

iD8DBQFETkBwvJuQZxSWSsgRAv1qAJ9xCqIzczi8RNtMZvOSEn4VnPMI0ACeOZfN
EeuaZ4YGtNR2vWBzh7f2qgQ=
=Wen5
-----END PGP SIGNATURE-----