Thread: gitmaster server problem?
About 20 minutes ago I starting getting gitmaster pull errors:
$ git pull ssh://git@gitmaster.postgresql.org/postgresql.git
ssh: connect to host gitmaster.postgresql.org port 22: Connection timed
out
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Public git seems fine:
$ git pull https://git.postgresql.org/git/postgresql.git
From https://git.postgresql.org/git/postgresql
* branch HEAD -> FETCH_HEAD
Already up to date.
Can anyone confirm or report on the cause?
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
On Sat, 17 Aug 2024 at 18:41, Bruce Momjian <bruce@momjian.us> wrote: > > About 20 minutes ago I starting getting gitmaster pull errors: > > $ git pull ssh://git@gitmaster.postgresql.org/postgresql.git > ssh: connect to host gitmaster.postgresql.org port 22: Connection timed > out > fatal: Could not read from remote repository. > > Please make sure you have the correct access rights > and the repository exists. [...] > Can anyone confirm or report on the cause? Additional infra that doesn't seem to work right now: mailing list archives [0] seem to fail at a 503 produced by Varnish Cache server: Error 503 Backend fetch failed. Maybe more of infra is down, or otherwise under maintenance? Kind regards, Matthias van de Meent [0] https://www.postgresql.org/list/pgsql-hackers/2024-08/
Matthias van de Meent <boekewurm+postgres@gmail.com> writes:
> Additional infra that doesn't seem to work right now: mailing list
> archives [0] seem to fail at a 503 produced by Varnish Cache server:
> Error 503 Backend fetch failed. Maybe more of infra is down, or
> otherwise under maintenance?
Yeah, I'm seeing that too; not at the top level of the site but
when you try to drill down to individual messages.
commitfest.postgresql.org isn't responding either,
nor wiki.postgresql.org.
regards, tom lane
Matthias van de Meent <boekewurm+postgres@gmail.com> writes:
> Additional infra that doesn't seem to work right now: mailing list
> archives [0] seem to fail at a 503 produced by Varnish Cache server:
> Error 503 Backend fetch failed. Maybe more of infra is down, or
> otherwise under maintenance?
Yeah, I'm seeing that too; not at the top level of the site but
when you try to drill down to individual messages.
commitfest.postgresql.org isn't responding either,
nor wiki.postgresql.org.
regards, tom lane
On Sat, Aug 17, 2024 at 02:15:49PM -0400, Tom Lane wrote: > Matthias van de Meent <boekewurm+postgres@gmail.com> writes: > > Additional infra that doesn't seem to work right now: mailing list > > archives [0] seem to fail at a 503 produced by Varnish Cache server: > > Error 503 Backend fetch failed. Maybe more of infra is down, or > > otherwise under maintenance? > > Yeah, I'm seeing that too; not at the top level of the site but > when you try to drill down to individual messages. > > commitfest.postgresql.org isn't responding either, > nor wiki.postgresql.org. Okay, thanks for the confirmation. I emailed pginfra at the same time I posted the first email, so they are aware. I have not received a reply from them yet. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Sat, Aug 17, 2024 at 02:15:49PM -0400, Tom Lane wrote: > Matthias van de Meent <boekewurm+postgres@gmail.com> writes: > > Additional infra that doesn't seem to work right now: mailing list > > archives [0] seem to fail at a 503 produced by Varnish Cache server: > > Error 503 Backend fetch failed. Maybe more of infra is down, or > > otherwise under maintenance? > > Yeah, I'm seeing that too; not at the top level of the site but > when you try to drill down to individual messages. > > commitfest.postgresql.org isn't responding either, > nor wiki.postgresql.org. Okay, thanks for the confirmation. I emailed pginfra at the same time I posted the first email, so they are aware. I have not received a reply from them yet. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On 8/17/24 13:59, Matthias van de Meent wrote: > On Sat, 17 Aug 2024 at 18:41, Bruce Momjian <bruce@momjian.us> wrote: >> >> About 20 minutes ago I starting getting gitmaster pull errors: >> >> $ git pull ssh://git@gitmaster.postgresql.org/postgresql.git >> ssh: connect to host gitmaster.postgresql.org port 22: Connection timed >> out >> fatal: Could not read from remote repository. >> >> Please make sure you have the correct access rights >> and the repository exists. > [...] >> Can anyone confirm or report on the cause? > > Additional infra that doesn't seem to work right now: mailing list > archives [0] seem to fail at a 503 produced by Varnish Cache server: > Error 503 Backend fetch failed. Maybe more of infra is down, or > otherwise under maintenance? > > Kind regards, > > Matthias van de Meent > > [0] https://www.postgresql.org/list/pgsql-hackers/2024-08/ Just to close this out -- problem found and fixed about an hour ago. Sorry for the interruption. -- Joe Conway PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
On Sat, 17 Aug 2024 at 22:05, Joe Conway <mail@joeconway.com> wrote: > Just to close this out -- problem found and fixed about an hour ago. > Sorry for the interruption. Whatever the problem was it seems to have returned
On 8/19/24 05:04, Jelte Fennema-Nio wrote: > On Sat, 17 Aug 2024 at 22:05, Joe Conway <mail@joeconway.com> wrote: >> Just to close this out -- problem found and fixed about an hour ago. >> Sorry for the interruption. > > Whatever the problem was it seems to have returned What specifically? I am not seeing anything at the moment. -- Joe Conway PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
On Mon, 19 Aug 2024 at 13:39, Joe Conway <mail@joeconway.com> wrote: > What specifically? I am not seeing anything at the moment. It seems to be working again fine. But before https://git.postgresql.org was throwing 502 and 503 errors. Depending on the request either by Varnish or nginx.
On Mon, Aug 19, 2024 at 7:57 AM Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > On Mon, 19 Aug 2024 at 13:39, Joe Conway <mail@joeconway.com> wrote: > > What specifically? I am not seeing anything at the moment. > > It seems to be working again fine. But before > https://git.postgresql.org was throwing 502 and 503 errors. Depending > on the request either by Varnish or nginx. I'm still unable to access any https://git.postgresql.org/ URL. -- Robert Haas EDB: http://www.enterprisedb.com
On Mon, 19 Aug 2024 at 14:04, Robert Haas <robertmhaas@gmail.com> wrote: > I'm still unable to access any https://git.postgresql.org/ URL. Yeah it's broken for me again too.
On Mon, 19 Aug 2024 at 14:05, Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > > On Mon, 19 Aug 2024 at 14:04, Robert Haas <robertmhaas@gmail.com> wrote: > > I'm still unable to access any https://git.postgresql.org/ URL. > > Yeah it's broken for me again too. In case the actual error is helpful (although I doubt it): Error 503 Backend fetch failed Backend fetch failed Guru Meditation: XID: 1030914118 ________________________________ Varnish cache server
On 8/19/24 08:06, Jelte Fennema-Nio wrote: > On Mon, 19 Aug 2024 at 14:05, Jelte Fennema-Nio <postgres@jeltef.nl> wrote: >> >> On Mon, 19 Aug 2024 at 14:04, Robert Haas <robertmhaas@gmail.com> wrote: >> > I'm still unable to access any https://git.postgresql.org/ URL. >> >> Yeah it's broken for me again too. > > In case the actual error is helpful (although I doubt it): > > Error 503 Backend fetch failed > > Backend fetch failed > > Guru Meditation: > > XID: 1030914118 I am not seeing any errors here. nagios monitoring is showing an ipv6 error though. Not the same issue as yesterday. -- Joe Conway PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com
On Mon, Aug 19, 2024 at 9:56 PM Bruce Momjian <bruce@momjian.us> wrote: > as of now. cgit working https://git.postgresql.org/cgit/postgresql.git/ git not working ( 403 Forbidden nginx) https://git.postgresql.org/git/postgresql.git/
On Mon, Aug 19, 2024 at 9:56 PM Bruce Momjian <bruce@momjian.us> wrote: > as of now. cgit working https://git.postgresql.org/cgit/postgresql.git/ git not working ( 403 Forbidden nginx) https://git.postgresql.org/git/postgresql.git/
On Mon, 19 Aug 2024 at 14:56, Bruce Momjian <bruce@momjian.us> wrote:
On Sat, Aug 17, 2024 at 02:15:49PM -0400, Tom Lane wrote:
> Matthias van de Meent <boekewurm+postgres@gmail.com> writes:
> > Additional infra that doesn't seem to work right now: mailing list
> > archives [0] seem to fail at a 503 produced by Varnish Cache server:
> > Error 503 Backend fetch failed. Maybe more of infra is down, or
> > otherwise under maintenance?
>
> Yeah, I'm seeing that too; not at the top level of the site but
> when you try to drill down to individual messages.
>
> commitfest.postgresql.org isn't responding either,
> nor wiki.postgresql.org.
Okay, thanks for the confirmation. I emailed pginfra at the same time I
posted the first email, so they are aware. I have not received a reply
from them yet.
Joe replied.
I've just checked again myself and I don't see any problems with the sites mentioned from a user perspective, but there are some non-obvious issues being reported by Nagios on git.postgresql.org (which is not the same as gitmaster). We'll dig in some more.
Dave Page
pgAdmin: https://www.pgadmin.org
On Mon, 19 Aug 2024 at 14:56, Bruce Momjian <bruce@momjian.us> wrote:
On Sat, Aug 17, 2024 at 02:15:49PM -0400, Tom Lane wrote:
> Matthias van de Meent <boekewurm+postgres@gmail.com> writes:
> > Additional infra that doesn't seem to work right now: mailing list
> > archives [0] seem to fail at a 503 produced by Varnish Cache server:
> > Error 503 Backend fetch failed. Maybe more of infra is down, or
> > otherwise under maintenance?
>
> Yeah, I'm seeing that too; not at the top level of the site but
> when you try to drill down to individual messages.
>
> commitfest.postgresql.org isn't responding either,
> nor wiki.postgresql.org.
Okay, thanks for the confirmation. I emailed pginfra at the same time I
posted the first email, so they are aware. I have not received a reply
from them yet.
Joe replied.
I've just checked again myself and I don't see any problems with the sites mentioned from a user perspective, but there are some non-obvious issues being reported by Nagios on git.postgresql.org (which is not the same as gitmaster). We'll dig in some more.
Dave Page
pgAdmin: https://www.pgadmin.org
On Mon, Aug 19, 2024 at 3:56 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Matthias van de Meent <boekewurm+postgres@gmail.com> writes:
> Additional infra that doesn't seem to work right now: mailing list
> archives [0] seem to fail at a 503 produced by Varnish Cache server:
> Error 503 Backend fetch failed. Maybe more of infra is down, or
> otherwise under maintenance?
Yeah, I'm seeing that too; not at the top level of the site but
when you try to drill down to individual messages.
commitfest.postgresql.org isn't responding either,
nor wiki.postgresql.org.
Are you still seeing these issues? It works fine both from here and from our monitoring, but there could be something source-network-specific maybe....
The git.postgresql.org one specifically looks a lot like a DOS. It may not be an intentional DOS, but the git server side is very inefficient and can rapidly turn valid attempts into DOSes if they parallellize too much...
On Mon, Aug 19, 2024 at 3:56 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Matthias van de Meent <boekewurm+postgres@gmail.com> writes:
> Additional infra that doesn't seem to work right now: mailing list
> archives [0] seem to fail at a 503 produced by Varnish Cache server:
> Error 503 Backend fetch failed. Maybe more of infra is down, or
> otherwise under maintenance?
Yeah, I'm seeing that too; not at the top level of the site but
when you try to drill down to individual messages.
commitfest.postgresql.org isn't responding either,
nor wiki.postgresql.org.
Are you still seeing these issues? It works fine both from here and from our monitoring, but there could be something source-network-specific maybe....
The git.postgresql.org one specifically looks a lot like a DOS. It may not be an intentional DOS, but the git server side is very inefficient and can rapidly turn valid attempts into DOSes if they parallellize too much...
On Mon, Aug 19, 2024 at 08:11:36AM -0400, Joe Conway wrote: > On 8/19/24 08:06, Jelte Fennema-Nio wrote: > > On Mon, 19 Aug 2024 at 14:05, Jelte Fennema-Nio <postgres@jeltef.nl> wrote: > > > > > > On Mon, 19 Aug 2024 at 14:04, Robert Haas <robertmhaas@gmail.com> wrote: > > > > I'm still unable to access any https://git.postgresql.org/ URL. > > > > > > Yeah it's broken for me again too. > > > > In case the actual error is helpful (although I doubt it): > > > > Error 503 Backend fetch failed > > > > Backend fetch failed > > > > Guru Meditation: > > > > XID: 1030914118 > > I am not seeing any errors here. nagios monitoring is showing an ipv6 error > though. Not the same issue as yesterday. gitmaster.postgresql.org is working fine for me right now. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Magnus Hagander <magnus@hagander.net> writes: > On Mon, Aug 19, 2024 at 3:56 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Yeah, I'm seeing that too; not at the top level of the site but >> when you try to drill down to individual messages. >> commitfest.postgresql.org isn't responding either, >> nor wiki.postgresql.org. > Are you still seeing these issues? It works fine both from here and from > our monitoring, but there could be something source-network-specific > maybe.... Right at the moment, https://git.postgresql.org/gitweb/?p=postgresql.git is failing for me with "Backend fetch failed". However, the mail archives, commitfest.p.o, and wiki.p.o seem to be responding normally, as is gitmaster. regards, tom lane
Magnus Hagander <magnus@hagander.net> writes: > On Mon, Aug 19, 2024 at 3:56 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: >> Yeah, I'm seeing that too; not at the top level of the site but >> when you try to drill down to individual messages. >> commitfest.postgresql.org isn't responding either, >> nor wiki.postgresql.org. > Are you still seeing these issues? It works fine both from here and from > our monitoring, but there could be something source-network-specific > maybe.... Right at the moment, https://git.postgresql.org/gitweb/?p=postgresql.git is failing for me with "Backend fetch failed". However, the mail archives, commitfest.p.o, and wiki.p.o seem to be responding normally, as is gitmaster. regards, tom lane
On Mon, 19 Aug 2024 at 15:27, jian he <jian.universality@gmail.com> wrote:
On Mon, Aug 19, 2024 at 9:56 PM Bruce Momjian <bruce@momjian.us> wrote:
>
as of now.
cgit working
https://git.postgresql.org/cgit/postgresql.git/
Yes.
git not working
( 403 Forbidden nginx)
https://git.postgresql.org/git/postgresql.git/
From a browser, that's what I'd expect. You should be able to clone from it though:
git clone https://git.postgresql.org/git/postgresql.git
Cloning into 'postgresql'...
remote: Enumerating objects: 42814, done.
remote: Counting objects: 100% (42814/42814), done.
remote: Compressing objects: 100% (15585/15585), done.
remote: Total 1014138 (delta 34190), reused 33980 (delta 26978), pack-reused 971324
Receiving objects: 100% (1014138/1014138), 343.01 MiB | 15.49 MiB/s, done.
Resolving deltas: 100% (873517/873517), done.
Cloning into 'postgresql'...
remote: Enumerating objects: 42814, done.
remote: Counting objects: 100% (42814/42814), done.
remote: Compressing objects: 100% (15585/15585), done.
remote: Total 1014138 (delta 34190), reused 33980 (delta 26978), pack-reused 971324
Receiving objects: 100% (1014138/1014138), 343.01 MiB | 15.49 MiB/s, done.
Resolving deltas: 100% (873517/873517), done.
Dave Page
pgAdmin: https://www.pgadmin.org
On Mon, 19 Aug 2024 at 15:27, jian he <jian.universality@gmail.com> wrote:
On Mon, Aug 19, 2024 at 9:56 PM Bruce Momjian <bruce@momjian.us> wrote:
>
as of now.
cgit working
https://git.postgresql.org/cgit/postgresql.git/
Yes.
git not working
( 403 Forbidden nginx)
https://git.postgresql.org/git/postgresql.git/
From a browser, that's what I'd expect. You should be able to clone from it though:
git clone https://git.postgresql.org/git/postgresql.git
Cloning into 'postgresql'...
remote: Enumerating objects: 42814, done.
remote: Counting objects: 100% (42814/42814), done.
remote: Compressing objects: 100% (15585/15585), done.
remote: Total 1014138 (delta 34190), reused 33980 (delta 26978), pack-reused 971324
Receiving objects: 100% (1014138/1014138), 343.01 MiB | 15.49 MiB/s, done.
Resolving deltas: 100% (873517/873517), done.
Cloning into 'postgresql'...
remote: Enumerating objects: 42814, done.
remote: Counting objects: 100% (42814/42814), done.
remote: Compressing objects: 100% (15585/15585), done.
remote: Total 1014138 (delta 34190), reused 33980 (delta 26978), pack-reused 971324
Receiving objects: 100% (1014138/1014138), 343.01 MiB | 15.49 MiB/s, done.
Resolving deltas: 100% (873517/873517), done.
Dave Page
pgAdmin: https://www.pgadmin.org
On Mon, 19 Aug 2024 at 15:29, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Magnus Hagander <magnus@hagander.net> writes:
> On Mon, Aug 19, 2024 at 3:56 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Yeah, I'm seeing that too; not at the top level of the site but
>> when you try to drill down to individual messages.
>> commitfest.postgresql.org isn't responding either,
>> nor wiki.postgresql.org.
> Are you still seeing these issues? It works fine both from here and from
> our monitoring, but there could be something source-network-specific
> maybe....
Right at the moment,
https://git.postgresql.org/gitweb/?p=postgresql.git
is failing for me with "Backend fetch failed".
That's working fine for me now, however Magnus did just give the box some more CPUs.
We're also considering rate limiting by IP on that server, as it seems to be getting hammered with requests from China.
Dave Page
pgAdmin: https://www.pgadmin.org
On Mon, 19 Aug 2024 at 15:29, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Magnus Hagander <magnus@hagander.net> writes:
> On Mon, Aug 19, 2024 at 3:56 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Yeah, I'm seeing that too; not at the top level of the site but
>> when you try to drill down to individual messages.
>> commitfest.postgresql.org isn't responding either,
>> nor wiki.postgresql.org.
> Are you still seeing these issues? It works fine both from here and from
> our monitoring, but there could be something source-network-specific
> maybe....
Right at the moment,
https://git.postgresql.org/gitweb/?p=postgresql.git
is failing for me with "Backend fetch failed".
That's working fine for me now, however Magnus did just give the box some more CPUs.
We're also considering rate limiting by IP on that server, as it seems to be getting hammered with requests from China.
Dave Page
pgAdmin: https://www.pgadmin.org
On Mon, Aug 19, 2024 at 4:28 PM Dave Page <dpage@pgadmin.org> wrote:
On Mon, 19 Aug 2024 at 14:56, Bruce Momjian <bruce@momjian.us> wrote:On Sat, Aug 17, 2024 at 02:15:49PM -0400, Tom Lane wrote:
> Matthias van de Meent <boekewurm+postgres@gmail.com> writes:
> > Additional infra that doesn't seem to work right now: mailing list
> > archives [0] seem to fail at a 503 produced by Varnish Cache server:
> > Error 503 Backend fetch failed. Maybe more of infra is down, or
> > otherwise under maintenance?
>
> Yeah, I'm seeing that too; not at the top level of the site but
> when you try to drill down to individual messages.
>
> commitfest.postgresql.org isn't responding either,
> nor wiki.postgresql.org.
Okay, thanks for the confirmation. I emailed pginfra at the same time I
posted the first email, so they are aware. I have not received a reply
from them yet.Joe replied.
In the context of replying it's also worth noting that CCing the same email to both -hackers and -www ensures it gets caught in moderation, and will take longer to reach someone.
/Magnus
On Mon, Aug 19, 2024 at 4:28 PM Dave Page <dpage@pgadmin.org> wrote:
On Mon, 19 Aug 2024 at 14:56, Bruce Momjian <bruce@momjian.us> wrote:On Sat, Aug 17, 2024 at 02:15:49PM -0400, Tom Lane wrote:
> Matthias van de Meent <boekewurm+postgres@gmail.com> writes:
> > Additional infra that doesn't seem to work right now: mailing list
> > archives [0] seem to fail at a 503 produced by Varnish Cache server:
> > Error 503 Backend fetch failed. Maybe more of infra is down, or
> > otherwise under maintenance?
>
> Yeah, I'm seeing that too; not at the top level of the site but
> when you try to drill down to individual messages.
>
> commitfest.postgresql.org isn't responding either,
> nor wiki.postgresql.org.
Okay, thanks for the confirmation. I emailed pginfra at the same time I
posted the first email, so they are aware. I have not received a reply
from them yet.Joe replied.
In the context of replying it's also worth noting that CCing the same email to both -hackers and -www ensures it gets caught in moderation, and will take longer to reach someone.
/Magnus
On 8/19/24 07:27, Tom Lane wrote: > Magnus Hagander <magnus@hagander.net> writes: > Right at the moment, > > https://git.postgresql.org/gitweb/?p=postgresql.git > > is failing for me with "Backend fetch failed". However, the mail I just tried it and it worked. > archives, commitfest.p.o, and wiki.p.o seem to be responding normally, > as is gitmaster. > > regards, tom lane > > -- Adrian Klaver adrian.klaver@aklaver.com
On 8/19/24 07:27, Tom Lane wrote: > Magnus Hagander <magnus@hagander.net> writes: > Right at the moment, > > https://git.postgresql.org/gitweb/?p=postgresql.git > > is failing for me with "Backend fetch failed". However, the mail I just tried it and it worked. > archives, commitfest.p.o, and wiki.p.o seem to be responding normally, > as is gitmaster. > > regards, tom lane > > -- Adrian Klaver adrian.klaver@aklaver.com