Thread: 403 - Forbidden on gitweb pages
Every so often (a few percent of the time, I think) I get "403 - Forbidden" from following a link on http://git.postgresql.org/gitweb/. So far, a reload has always worked first time, but there's definitely something fishy in there. I don't recall exactly when this started happening, maybe a month or so back; it hasn't been there all along. Anybody else see this, or have an idea what broke? regards, tom lane
Tom Lane wrote: > Every so often (a few percent of the time, I think) I get "403 - > Forbidden" from following a link on http://git.postgresql.org/gitweb/. > So far, a reload has always worked first time, but there's definitely > something fishy in there. I don't recall exactly when this started > happening, maybe a month or so back; it hasn't been there all along. > > Anybody else see this, or have an idea what broke? Yes, I see this also. I reported it to Magnus, who muttered something about Varnish possibly misbehaving, but that was the end of it. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Thursday, February 14, 2013, Alvaro Herrera wrote:
Tom Lane wrote:
> Every so often (a few percent of the time, I think) I get "403 -
> Forbidden" from following a link on http://git.postgresql.org/gitweb/.
> So far, a reload has always worked first time, but there's definitely
> something fishy in there. I don't recall exactly when this started
> happening, maybe a month or so back; it hasn't been there all along.
>
> Anybody else see this, or have an idea what broke?
Yes, I see this also. I reported it to Magnus, who muttered something
about Varnish possibly misbehaving, but that was the end of it.
He told me it was intentional rate limiting, applied because gitweb is so inefficient.
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
<p dir="ltr"><br /> On Feb 14, 2013 1:35 AM, "Dave Page" <<a href="mailto:dpage@pgadmin.org">dpage@pgadmin.org</a>>wrote:<br /> ><br /> ><br /> ><br /> > On Thursday,February 14, 2013, Alvaro Herrera wrote:<br /> >><br /> >> Tom Lane wrote:<br /> >> > Everyso often (a few percent of the time, I think) I get "403 -<br /> >> > Forbidden" from following a link on <ahref="http://git.postgresql.org/gitweb/">http://git.postgresql.org/gitweb/</a>.<br /> >> > So far, a reload hasalways worked first time, but there's definitely<br /> >> > something fishy in there. I don't recall exactlywhen this started<br /> >> > happening, maybe a month or so back; it hasn't been there all along.<br /> >>><br /> >> > Anybody else see this, or have an idea what broke?<br /> >><br /> >> Yes, Isee this also. I reported it to Magnus, who muttered something<br /> >> about Varnish possibly misbehaving, but thatwas the end of it.<br /> >><br /> ><br /> > He told me it was intentional rate limiting, applied becausegitweb is so inefficient.<br /><p dir="ltr">Correct. And varnish is not involved at all. I think alvaro is confusingit with me saying that varnish might be a way for us to actually fix the problem. <p dir="ltr">The problem is basicallythat gitweb is a big perl cgi (yes, cgi only) which means that for every request it had to reload all of perl andreparse the whole script. And on top of that the actual implementation of several features is fairly slow in itself. <pdir="ltr">This caused easy overload - for example the whole site went down when a search engine spidered us... <br /><pdir="ltr">The bandaid solution was to rate limit access per ip. The error you hit is when you're rate limited. <p dir="ltr">Now,I was supposed to have on my to do list to make a better fix. But as this one was implemented a long time ago(measured in years, really, I have to admit I forgot... I'll put bit back on the list. <p dir="ltr">/Magnus
Magnus Hagander <magnus@hagander.net> writes:
> The bandaid solution was to rate limit access per ip. The error you hit is
> when you're rate limited.
[ raised eyebrow... ] I'm fairly sure I've seen it more than once when
performing a *single* page fetch. In any case, the probability of
failure increased by a couple orders of magnitude sometime in the past
month or so, because I'd never seen it before that.
regards, tom lane
<p dir="ltr"><br /> On Feb 14, 2013 5:31 AM, "Tom Lane" <<a href="mailto:tgl@sss.pgh.pa.us">tgl@sss.pgh.pa.us</a>>wrote:<br /> ><br /> > Magnus Hagander <<a href="mailto:magnus@hagander.net">magnus@hagander.net</a>>writes:<br /> > > The bandaid solution was to rate limitaccess per ip. The error you hit is<br /> > > when you're rate limited.<br /> ><br /> > [ raised eyebrow...] I'm fairly sure I've seen it more than once when<br /> > performing a *single* page fetch. In any case,the probability of<br /> > failure increased by a couple orders of magnitude sometime in the past<br /> > monthor so, because I'd never seen it before that.<p dir="ltr">In that case you may be hitting a different issue, and we'rewriting it off as the rare limiting by mistake. Because nothing about that one has changed for quite some time, andshould never hit on a single page. Will have to investigate that further then. <p dir="ltr">/Magnus
Magnus Hagander <magnus@hagander.net> writes: > The problem is basically that gitweb is a big perl cgi (yes, cgi only) > which means that for every request it had to reload all of perl and reparse > the whole script. And on top of that the actual implementation of several > features is fairly slow in itself. What about finding a more efficient replacement? http://gitlabhq.com/ http://hjemli.net/git/cgit/ We need to check that GitLab can be setup to not offer all of its features, and the CGIT way of displaying branches looks very much suited to our project back branches support. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
On 02/14/2013 05:31 AM, Tom Lane wrote: > Magnus Hagander <magnus@hagander.net> writes: >> The bandaid solution was to rate limit access per ip. The error you hit is >> when you're rate limited. > > [ raised eyebrow... ] I'm fairly sure I've seen it more than once when > performing a *single* page fetch. In any case, the probability of > failure increased by a couple orders of magnitude sometime in the past > month or so, because I'd never seen it before that. any chance you moved to a different browser (or a new version of it) in that timeframe? I can trivially reproduce that issue here now because my browser is employing fairly agressive prefetching techniques that the currently rate-limiting system is not prepared to deal with, and from looking at the logs this is actually a fairly common issue :( Stefan
Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes:
> On 02/14/2013 05:31 AM, Tom Lane wrote:
>> [ raised eyebrow... ] I'm fairly sure I've seen it more than once when
>> performing a *single* page fetch. In any case, the probability of
>> failure increased by a couple orders of magnitude sometime in the past
>> month or so, because I'd never seen it before that.
> any chance you moved to a different browser (or a new version of it) in
> that timeframe?
> I can trivially reproduce that issue here now because my browser is
> employing fairly agressive prefetching techniques that the currently
> rate-limiting system is not prepared to deal with, and from looking at
> the logs this is actually a fairly common issue :(
Hm. I usually use Apple's Safari, which is currently at 6.0.2, and
it looks like I installed that update at the beginning of November.
It's possible they instituted aggressive prefetching in the 6.0.1 to
6.0.2 update, but somehow I doubt that.
Would it make sense to just back off the rate limiting a bit?
regards, tom lane
On 02/15/2013 12:28 AM, Tom Lane wrote: > Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes: >> On 02/14/2013 05:31 AM, Tom Lane wrote: >>> [ raised eyebrow... ] I'm fairly sure I've seen it more than once when >>> performing a *single* page fetch. In any case, the probability of >>> failure increased by a couple orders of magnitude sometime in the past >>> month or so, because I'd never seen it before that. > >> any chance you moved to a different browser (or a new version of it) in >> that timeframe? >> I can trivially reproduce that issue here now because my browser is >> employing fairly agressive prefetching techniques that the currently >> rate-limiting system is not prepared to deal with, and from looking at >> the logs this is actually a fairly common issue :( > > Hm. I usually use Apple's Safari, which is currently at 6.0.2, and > it looks like I installed that update at the beginning of November. > It's possible they instituted aggressive prefetching in the 6.0.1 to > 6.0.2 update, but somehow I doubt that. hm ok... > > Would it make sense to just back off the rate limiting a bit? done that for now until we have a better solution - lets see if it behaves better now. Stefan
On Fri, Feb 15, 2013 at 3:33 PM, Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> wrote: > On 02/15/2013 12:28 AM, Tom Lane wrote: >> Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes: >>> On 02/14/2013 05:31 AM, Tom Lane wrote: >>>> [ raised eyebrow... ] I'm fairly sure I've seen it more than once when >>>> performing a *single* page fetch. In any case, the probability of >>>> failure increased by a couple orders of magnitude sometime in the past >>>> month or so, because I'd never seen it before that. >> >>> any chance you moved to a different browser (or a new version of it) in >>> that timeframe? >>> I can trivially reproduce that issue here now because my browser is >>> employing fairly agressive prefetching techniques that the currently >>> rate-limiting system is not prepared to deal with, and from looking at >>> the logs this is actually a fairly common issue :( >> >> Hm. I usually use Apple's Safari, which is currently at 6.0.2, and >> it looks like I installed that update at the beginning of November. >> It's possible they instituted aggressive prefetching in the 6.0.1 to >> 6.0.2 update, but somehow I doubt that. > > hm ok... > >> >> Would it make sense to just back off the rate limiting a bit? > > done that for now until we have a better solution - lets see if it > behaves better now. I've deployed what I think is a better fix for this now - it's now limited to two parallel *active* connections from wherever, but any further requests are placed in a queue rather than being rejected iwth 403. As a bonus, we now also have caching. That makes particularly the frontpage quite a lot faster for most people - not likely to have a big effect on details pages, since there are just too many of them to get efficient caching. -- Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/