Thread: Re: No easy way to join discussion in existing thread when not subscribed

On 09/30/2015 09:53 AM, Stefan Kaltenbrunner wrote:
> On 09/30/2015 03:27 AM, Amir Rohan wrote:
>> On 09/29/2015 10:51 PM, Stefan Kaltenbrunner wrote:
>>> On 09/29/2015 09:34 PM, Amir Rohan wrote:
>>>
>>> for most accesses to the archives the string for the basic auth reply
>>> quotes the "archives" and "password" strings with ' - see
>>
>> Fixed.
>
> I think you missed at least one spot in the code you added and also at
> least one occurance in existing code.
>

Hmmm, then that is a 100% miss rate when "antispam" greps precisely
twice in views.py. But, I double checked the patch (V3) and both are fixed.

Did you notice that the file contains multiple patches?

>>
>> You're right to be concerned, I raised the issue myself to begin with.
>> We can solve any particular problem, but how to optimize depends too
>> much on particulars I don't have.
>>
>> If you have both cpu and memory shortage, we could trade storage.
>> You already serve monthly mbox's, having per thread mboxes which are
>> updated in batch (say hourly) could be managable, and that code
>> is practically written already. Serving static is as cheap as it gets
>> on noth cpu and memory.
>
> yeah that is what I was thinking - though I dont think we want hourly.
> Went went a long way to actually get the current system to be "almost
> instant" in terms of having the archives in sync with the lists
>

I'm pretty sure the monthly mboxes are lagging by at least a few hours,
if they are not actually nightlies.

This reads like a rejection for this whole "generate from db" approach.
And I can't help implement the static solution, as that's cron/root
stuff.

For closure only, here is the final V4, which:
1) Batches fetches from the DB (but not as far as one network roundtrip
per message), with a new settings.THREAD_MBOX_DB_BATCH_SIZE.
2) Bails out with 403, as soon as a batch exceeds mbox size hard limit.
3) Sets Content-Disposition so the user gets a descriptive filename
instead of a hashy Message ID.
4) returns 200 ("this feature disabled") instead of 403 as previously,
Since disabling downloads shouldn't log requests as errors in your
monitoring infra.

Cheers,
Amir

On 09/30/2015 09:33 AM, Amir Rohan wrote:
> On 09/30/2015 09:53 AM, Stefan Kaltenbrunner wrote:
>> On 09/30/2015 03:27 AM, Amir Rohan wrote:
>>> On 09/29/2015 10:51 PM, Stefan Kaltenbrunner wrote:
>>>> On 09/29/2015 09:34 PM, Amir Rohan wrote:
>>>>
>>>> for most accesses to the archives the string for the basic auth reply
>>>> quotes the "archives" and "password" strings with ' - see
>>>
>>> Fixed.
>>
>> I think you missed at least one spot in the code you added and also at
>> least one occurance in existing code.
>>
> 
> Hmmm, then that is a 100% miss rate when "antispam" greps precisely
> twice in views.py. But, I double checked the patch (V3) and both are fixed.
> 
> Did you notice that the file contains multiple patches?

no missed that - makes reading it without applying not exactly easier :)


> 
>>>
>>> You're right to be concerned, I raised the issue myself to begin with.
>>> We can solve any particular problem, but how to optimize depends too
>>> much on particulars I don't have.
>>>
>>> If you have both cpu and memory shortage, we could trade storage.
>>> You already serve monthly mbox's, having per thread mboxes which are
>>> updated in batch (say hourly) could be managable, and that code
>>> is practically written already. Serving static is as cheap as it gets
>>> on noth cpu and memory.
>>
>> yeah that is what I was thinking - though I dont think we want hourly.
>> Went went a long way to actually get the current system to be "almost
>> instant" in terms of having the archives in sync with the lists
>>
> 
> I'm pretty sure the monthly mboxes are lagging by at least a few hours,
> if they are not actually nightlies.

they are rsynced over from the main mailinglist server where mj2
generates them.

> 
> This reads like a rejection for this whole "generate from db" approach.
> And I can't help implement the static solution, as that's cron/root
> stuff.

no -  that was not what I was trying to say - my proposal was to add the
per-thread mbox generation (and maybe even the monthly ones longer term)
as an option to
http://git.postgresql.org/gitweb/?p=pgarchives.git;a=blob;f=loader/load_message.py
(which is basically called by the MTA) - that way the mbox file would be
built at the same time the archive webfrontend also has it. We might
want to rename the tool at that time as well to reduce confusion or add
a seperate script that does that and gets called by the other (or the mta)



Stefan