Thread: Authentication prompt for mbox downloads

On Thu, Mar 26, 2020 at 1:53 PM Dave Page <dpage@pgadmin.org> wrote:
>
> Some, maybe all of the major browsers no longer display the security realm on login prompts, which was previously
usedto inform the user of the anti-spam username and password used to protect the mailbox archives. This means that the
onlyway to get it now is either to go find it in the source code for the website, or look at the response headers in
thebrowsers developer tools. 

FWIW, Firefox still shows it as expected. There's at least one browser
out there that's not broken... But yeah, that's not enough, and it
seems the others no longer care about this problem (which they said
years ago they were going to fix, but since they haven't..)


> The attached patch adds a note to the page instead.

My feeling is that's a very prominent location (and space usage) for
something that in the end very few people use. And it also misses the
usecase of people downloading thread-as-mbox, as well as the
functionality to view raw message, which both use the same protection,
so should be equally informed.

Perhaps put it in a title attribute on the link, so it shows as a
popup? That would work in all the use cases, I think?

//Magnus

On 3/26/20 8:53 AM, Dave Page wrote:
> Some, maybe all of the major browsers no longer display the security
> realm on login prompts, which was previously used to inform the user of
> the anti-spam username and password used to protect the mailbox
> archives.

I think it's mainly broken in Chrome, though I just checked and this now
extends to Safari. It works fine in Firefox.

> This means that the only way to get it now is either to go
> find it in the source code for the website, or look at the response
> headers in the browsers developer tools.
>
> The attached patch adds a note to the page instead.

Syntax-wise please switch the "<i>" to "<em>". Should we go down this
patch, we'd also want to place that message on any page where one can
download an archive.

I do wonder if by placing the text on the site like that, we make it a
bit easier to defeat the original purpose of the prompt. Some other ideas:

1. We have a JavaScript snippet that executes when the page loads to
render the text in place. Not fool proof, but it's around the same level
as the current solution (though this would likely expose the credentials
in the JavaScript source).

2. We render the username/password using images. Similarly, not
foolproof, but requires a nontrivial effort.

And yes, it's fairly easy to get the credentials now and script it (at
least if you use Firefox, or browse our git repos), but I figure we
could make it a slight burden.

Jonathan

On Thu, Mar 26, 2020 at 2:14 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> On 3/26/20 8:53 AM, Dave Page wrote:
> > Some, maybe all of the major browsers no longer display the security
> > realm on login prompts, which was previously used to inform the user of
> > the anti-spam username and password used to protect the mailbox
> > archives.
>
> I think it's mainly broken in Chrome, though I just checked and this now
> extends to Safari. It works fine in Firefox.
>
> > This means that the only way to get it now is either to go
> > find it in the source code for the website, or look at the response
> > headers in the browsers developer tools.
> >
> > The attached patch adds a note to the page instead.
>
> Syntax-wise please switch the "<i>" to "<em>". Should we go down this
> patch, we'd also want to place that message on any page where one can
> download an archive.
>
> I do wonder if by placing the text on the site like that, we make it a
> bit easier to defeat the original purpose of the prompt. Some other ideas:
>
> 1. We have a JavaScript snippet that executes when the page loads to
> render the text in place. Not fool proof, but it's around the same level
> as the current solution (though this would likely expose the credentials
> in the JavaScript source).
>
> 2. We render the username/password using images. Similarly, not
> foolproof, but requires a nontrivial effort.

I don't think either of those make any actual difference. We already
give the instructions in the actual prompt sent back, which is the
very first things that scripts will see. If we want to defeat those
things, we need to go to something like a captcha for example. Which
will add a fair amount of friction for those that *do* know it
already.

Probably the majority of people who are downloading these have done so
at least once before, and thus do *not* need the instructions. We
should try to avoid making it worse for them. And in particular, 99%
of the visitors to our archives are not interested in mboxes at all,
and we should *definitely* try to avoid making it worse for them.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

On 3/26/20 9:23 AM, Magnus Hagander wrote:
> On Thu, Mar 26, 2020 at 2:14 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>>
>> On 3/26/20 8:53 AM, Dave Page wrote:
>>> Some, maybe all of the major browsers no longer display the security
>>> realm on login prompts, which was previously used to inform the user of
>>> the anti-spam username and password used to protect the mailbox
>>> archives.
>>
>> I think it's mainly broken in Chrome, though I just checked and this now
>> extends to Safari. It works fine in Firefox.
>>
>>> This means that the only way to get it now is either to go
>>> find it in the source code for the website, or look at the response
>>> headers in the browsers developer tools.
>>>
>>> The attached patch adds a note to the page instead.
>>
>> Syntax-wise please switch the "<i>" to "<em>". Should we go down this
>> patch, we'd also want to place that message on any page where one can
>> download an archive.
>>
>> I do wonder if by placing the text on the site like that, we make it a
>> bit easier to defeat the original purpose of the prompt. Some other ideas:
>>
>> 1. We have a JavaScript snippet that executes when the page loads to
>> render the text in place. Not fool proof, but it's around the same level
>> as the current solution (though this would likely expose the credentials
>> in the JavaScript source).
>>
>> 2. We render the username/password using images. Similarly, not
>> foolproof, but requires a nontrivial effort.
>
> I don't think either of those make any actual difference. We already
> give the instructions in the actual prompt sent back, which is the
> very first things that scripts will see.

Yes...that's what I said (perhaps not clearly) in the part of my
response you cut out.

> If we want to defeat those
> things, we need to go to something like a captcha for example. Which
> will add a fair amount of friction for those that *do* know it
> already.

I'm not suggesting we disable the Basic Auth mechanism. I'm just making
suggestions around displaying the credentials.

If someone needs to look up the credentials, the captcha is not a bad idea.

> Probably the majority of people who are downloading these have done so
> at least once before, and thus do *not* need the instructions.

I don't know how frequently people use this feature (perhaps you have
the stats?). Whenever I do, I know I have to look up the instructions
every time because I don't remember the credentials, which leads to the
poor user experience that Dave describes.

> We
> should try to avoid making it worse for them. And in particular, 99%
> of the visitors to our archives are not interested in mboxes at all,

I would not dispute that the number of people downloading the mboxes is
way smaller than the other usage of the archives, but it would be good
to know the actual proportions. GA does not provide the mbox stats.

> and we should *definitely* try to avoid making it worse for them.

I don't see how any of the above, including Dave's patch, make things
worse. Again I was just suggesting on how to display the credentials,
not adding more steps to downloading the mbox.

Thanks,

Jonathan

On Thu, Mar 26, 2020 at 2:33 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>
> On 3/26/20 9:23 AM, Magnus Hagander wrote:
> > On Thu, Mar 26, 2020 at 2:14 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
> >>
> >> On 3/26/20 8:53 AM, Dave Page wrote:
> >>> Some, maybe all of the major browsers no longer display the security
> >>> realm on login prompts, which was previously used to inform the user of
> >>> the anti-spam username and password used to protect the mailbox
> >>> archives.
> >>
> >> I think it's mainly broken in Chrome, though I just checked and this now
> >> extends to Safari. It works fine in Firefox.
> >>
> >>> This means that the only way to get it now is either to go
> >>> find it in the source code for the website, or look at the response
> >>> headers in the browsers developer tools.
> >>>
> >>> The attached patch adds a note to the page instead.
> >>
> >> Syntax-wise please switch the "<i>" to "<em>". Should we go down this
> >> patch, we'd also want to place that message on any page where one can
> >> download an archive.
> >>
> >> I do wonder if by placing the text on the site like that, we make it a
> >> bit easier to defeat the original purpose of the prompt. Some other ideas:
> >>
> >> 1. We have a JavaScript snippet that executes when the page loads to
> >> render the text in place. Not fool proof, but it's around the same level
> >> as the current solution (though this would likely expose the credentials
> >> in the JavaScript source).
> >>
> >> 2. We render the username/password using images. Similarly, not
> >> foolproof, but requires a nontrivial effort.
> >
> > I don't think either of those make any actual difference. We already
> > give the instructions in the actual prompt sent back, which is the
> > very first things that scripts will see.
>
> Yes...that's what I said (perhaps not clearly) in the part of my
> response you cut out.
>
> > If we want to defeat those
> > things, we need to go to something like a captcha for example. Which
> > will add a fair amount of friction for those that *do* know it
> > already.
>
> I'm not suggesting we disable the Basic Auth mechanism. I'm just making
> suggestions around displaying the credentials.
>
> If someone needs to look up the credentials, the captcha is not a bad idea.
>
> > Probably the majority of people who are downloading these have done so
> > at least once before, and thus do *not* need the instructions.
>
> I don't know how frequently people use this feature (perhaps you have
> the stats?). Whenever I do, I know I have to look up the instructions
> every time because I don't remember the credentials, which leads to the
> poor user experience that Dave describes.

We seem to average 2 downloads of mbox files per day.

And about 1300 hits to the mbox urls that return the 401 code to
require authorization.

It's hard to directly compare to "actual accesses to messages",
because those are very heavily cached. But we're looking at
approximately 75,000 cache *misses* on emails every day. So I would
expect at least a few million downloads (for "recent messages" the
cache rate is something in excess of 90%, but a fair number of the
other hits might be a bot that's picking up old ones).

Bottom line is, the number of people downloading the mboxes are *very*
few in comparison.


> > We
> > should try to avoid making it worse for them. And in particular, 99%
> > of the visitors to our archives are not interested in mboxes at all,
>
> I would not dispute that the number of people downloading the mboxes is
> way smaller than the other usage of the archives, but it would be good
> to know the actual proportions. GA does not provide the mbox stats.
>
> > and we should *definitely* try to avoid making it worse for them.
>
> I don't see how any of the above, including Dave's patch, make things
> worse. Again I was just suggesting on how to display the credentials,
> not adding more steps to downloading the mbox.

Well, Daves patch doesn't cover 2 out of 3 cases, that's a start :)

And it makes it worse in that it goes in the very most "valuable
space" on the screen, for something that's focused on a tiny portion
of our users.

I'm certainly not against making the information better, I'm just not
sure that's the best way.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

On 3/26/20 9:51 AM, Magnus Hagander wrote:
> On Thu, Mar 26, 2020 at 2:33 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>>
>> On 3/26/20 9:23 AM, Magnus Hagander wrote:
>>> On Thu, Mar 26, 2020 at 2:14 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>>>>
>>>> On 3/26/20 8:53 AM, Dave Page wrote:
>>>>> Some, maybe all of the major browsers no longer display the security
>>>>> realm on login prompts, which was previously used to inform the user of
>>>>> the anti-spam username and password used to protect the mailbox
>>>>> archives.
>>>>
>>>> I think it's mainly broken in Chrome, though I just checked and this now
>>>> extends to Safari. It works fine in Firefox.
>>>>
>>>>> This means that the only way to get it now is either to go
>>>>> find it in the source code for the website, or look at the response
>>>>> headers in the browsers developer tools.
>>>>>
>>>>> The attached patch adds a note to the page instead.
>>>>
>>>> Syntax-wise please switch the "<i>" to "<em>". Should we go down this
>>>> patch, we'd also want to place that message on any page where one can
>>>> download an archive.
>>>>
>>>> I do wonder if by placing the text on the site like that, we make it a
>>>> bit easier to defeat the original purpose of the prompt. Some other ideas:
>>>>
>>>> 1. We have a JavaScript snippet that executes when the page loads to
>>>> render the text in place. Not fool proof, but it's around the same level
>>>> as the current solution (though this would likely expose the credentials
>>>> in the JavaScript source).
>>>>
>>>> 2. We render the username/password using images. Similarly, not
>>>> foolproof, but requires a nontrivial effort.
>>>
>>> I don't think either of those make any actual difference. We already
>>> give the instructions in the actual prompt sent back, which is the
>>> very first things that scripts will see.
>>
>> Yes...that's what I said (perhaps not clearly) in the part of my
>> response you cut out.
>>
>>> If we want to defeat those
>>> things, we need to go to something like a captcha for example. Which
>>> will add a fair amount of friction for those that *do* know it
>>> already.
>>
>> I'm not suggesting we disable the Basic Auth mechanism. I'm just making
>> suggestions around displaying the credentials.
>>
>> If someone needs to look up the credentials, the captcha is not a bad idea.
>>
>>> Probably the majority of people who are downloading these have done so
>>> at least once before, and thus do *not* need the instructions.
>>
>> I don't know how frequently people use this feature (perhaps you have
>> the stats?). Whenever I do, I know I have to look up the instructions
>> every time because I don't remember the credentials, which leads to the
>> poor user experience that Dave describes.
>
> We seem to average 2 downloads of mbox files per day.
>
> And about 1300 hits to the mbox urls that return the 401 code to
> require authorization.
>
> It's hard to directly compare to "actual accesses to messages",
> because those are very heavily cached. But we're looking at
> approximately 75,000 cache *misses* on emails every day. So I would
> expect at least a few million downloads (for "recent messages" the
> cache rate is something in excess of 90%, but a fair number of the
> other hits might be a bot that's picking up old ones).
>
> Bottom line is, the number of people downloading the mboxes are *very*
> few in comparison.

*nods* thanks for confirming!

>>> We
>>> should try to avoid making it worse for them. And in particular, 99%
>>> of the visitors to our archives are not interested in mboxes at all,
>>
>> I would not dispute that the number of people downloading the mboxes is
>> way smaller than the other usage of the archives, but it would be good
>> to know the actual proportions. GA does not provide the mbox stats.
>>
>>> and we should *definitely* try to avoid making it worse for them.
>>
>> I don't see how any of the above, including Dave's patch, make things
>> worse. Again I was just suggesting on how to display the credentials,
>> not adding more steps to downloading the mbox.
>
> Well, Daves patch doesn't cover 2 out of 3 cases, that's a start :)
>
> And it makes it worse in that it goes in the very most "valuable
> space" on the screen, for something that's focused on a tiny portion
> of our users.
>
> I'm certainly not against making the information better, I'm just not
> sure that's the best way.

We could move it to the bottom of the pages. Or add it to this page[1]
and call it a day.

Jonathan

[1] https://www.postgresql.org/list/

Magnus Hagander <magnus@hagander.net> writes:
> I'm certainly not against making the information better, I'm just not
> sure that's the best way.

So I tried this with the latest version of Safari, and AFAICT it
just worked --- I got a download without any fiddle-faddle.
I assume what this means is that I'm already logged in, but
if so it was quite some time ago, because I rarely use this.

I'd think this means that for most of the people who actually use
this feature there's not going to be a problem.  So I agree that
taking up valuable screen real estate is not a great solution.

            regards, tom lane

On 3/26/20 7:37 AM, Dave Page wrote:
> 
> 
> On Thu, Mar 26, 2020 at 2:03 PM Jonathan S. Katz <jkatz@postgresql.org 
> <mailto:jkatz@postgresql.org>> wrote:
> 
>     On 3/26/20 9:51 AM, Magnus Hagander wrote:
>      >
>      >> I don't see how any of the above, including Dave's patch, make
>     things
>      >> worse. Again I was just suggesting on how to display the
>     credentials,
>      >> not adding more steps to downloading the mbox.
>      >
>      > Well, Daves patch doesn't cover 2 out of 3 cases, that's a start :)
> 
> 
> I'm not clear what cases you're referring to, even after re-reading the 
> thread.

https://www.postgresql.org/message-id/CABUevEyiUpQun8X2fQj6m3n2J_2CHisTDX0vUXFwq%2BJR9JdZHA%40mail.gmail.com

"And it also misses the
usecase of people downloading thread-as-mbox, as well as the
functionality to view raw message, which both use the same protection,
so should be equally informed."

> 
>      > And it makes it worse in that it goes in the very most "valuable
>      > space" on the screen, for something that's focused on a tiny portion
>      > of our users.
>      >
>      > I'm certainly not against making the information better, I'm just not
>      > sure that's the best way.
> 
>     We could move it to the bottom of the pages. Or add it to this page[1]
>     and call it a day.
> 
> 
> Either would largely guarantee that users don't see it.
> 
> I was going to suggest putting a help icon with popup hint on click next 
> to the table header for the column, but then I noticed we don't actually 
> have column headers.
> -- 
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
> 
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company


-- 
Adrian Klaver
adrian.klaver@aklaver.com

On Thu, Mar 26, 2020 at 3:38 PM Dave Page <dpage@pgadmin.org> wrote:
>
>
>
> On Thu, Mar 26, 2020 at 2:03 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>>
>> On 3/26/20 9:51 AM, Magnus Hagander wrote:
>> >
>> >> I don't see how any of the above, including Dave's patch, make things
>> >> worse. Again I was just suggesting on how to display the credentials,
>> >> not adding more steps to downloading the mbox.
>> >
>> > Well, Daves patch doesn't cover 2 out of 3 cases, that's a start :)
>
>
> I'm not clear what cases you're referring to, even after re-reading the thread.

The downloading of thread mailboxes, and the viewing of raw messages.
It was in my first response :)


>> > And it makes it worse in that it goes in the very most "valuable
>> > space" on the screen, for something that's focused on a tiny portion
>> > of our users.
>> >
>> > I'm certainly not against making the information better, I'm just not
>> > sure that's the best way.
>>
>> We could move it to the bottom of the pages. Or add it to this page[1]
>> and call it a day.
>
>
> Either would largely guarantee that users don't see it.
>
> I was going to suggest putting a help icon with popup hint on click next to the table header for the column, but then
Inoticed we don't actually have column headers.
 

We could put it just to the right of the download link though? There
seems to be enough space for that with a large margin at least on
desktops.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

Dave Page <dpage@pgadmin.org> writes:
> On Thu, Mar 26, 2020 at 2:44 PM Magnus Hagander <magnus@hagander.net> wrote:
>> We could put it just to the right of the download link though? There
>> seems to be enough space for that with a large margin at least on
>> desktops.

> We could. I was hoping for a way to avoid putting it on every row -
> obviously technically we can do that, but it might look weird.

If you put a "help" link to the right of the existing buttons,
it wouldn't look too weird or take up too much space.

BTW, Safari swears up and down that I have no saved passwords, so
I remain unclear on why it "just works" for me.

            regards, tom lane

On Thu, Mar 26, 2020 at 4:50 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Dave Page <dpage@pgadmin.org> writes:
> > On Thu, Mar 26, 2020 at 2:44 PM Magnus Hagander <magnus@hagander.net> wrote:
> >> We could put it just to the right of the download link though? There
> >> seems to be enough space for that with a large margin at least on
> >> desktops.
>
> > We could. I was hoping for a way to avoid putting it on every row -
> > obviously technically we can do that, but it might look weird.
>
> If you put a "help" link to the right of the existing buttons,
> it wouldn't look too weird or take up too much space.
>
> BTW, Safari swears up and down that I have no saved passwords, so
> I remain unclear on why it "just works" for me.

Can you send me (offlist) the exact link you're downloading -- there
is always the risk that there is a bug somewhere :)


-- 
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

On 3/26/20 3:44 PM, Magnus Hagander wrote:
> On Thu, Mar 26, 2020 at 3:38 PM Dave Page <dpage@pgadmin.org> wrote:
>>
>>
>>
>> On Thu, Mar 26, 2020 at 2:03 PM Jonathan S. Katz <jkatz@postgresql.org> wrote:
>>>
>>> On 3/26/20 9:51 AM, Magnus Hagander wrote:
>>>>
>>>>> I don't see how any of the above, including Dave's patch, make things
>>>>> worse. Again I was just suggesting on how to display the credentials,
>>>>> not adding more steps to downloading the mbox.
>>>>
>>>> Well, Daves patch doesn't cover 2 out of 3 cases, that's a start :)
>>
>>
>> I'm not clear what cases you're referring to, even after re-reading the thread.
> 
> The downloading of thread mailboxes, and the viewing of raw messages.
> It was in my first response :)
> 
> 
>>>> And it makes it worse in that it goes in the very most "valuable
>>>> space" on the screen, for something that's focused on a tiny portion
>>>> of our users.
>>>>
>>>> I'm certainly not against making the information better, I'm just not
>>>> sure that's the best way.
>>>
>>> We could move it to the bottom of the pages. Or add it to this page[1]
>>> and call it a day.
>>
>>
>> Either would largely guarantee that users don't see it.
>>
>> I was going to suggest putting a help icon with popup hint on click next to the table header for the column, but
thenI noticed we don't actually have column headers.
 
> 
> We could put it just to the right of the download link though? There
> seems to be enough space for that with a large margin at least on
> desktops.

what about doing something completely different - like hiding those
links behind community auth and call it a day(especially with so few users)?

I dont see a problem with requiring community auth for downloading those
links as well as raw messages..



Stefan

Greetings,

* Stefan Kaltenbrunner (stefan@kaltenbrunner.cc) wrote:
> On 3/26/20 3:44 PM, Magnus Hagander wrote:
> > We could put it just to the right of the download link though? There
> > seems to be enough space for that with a large margin at least on
> > desktops.
>
> what about doing something completely different - like hiding those
> links behind community auth and call it a day(especially with so few users)?
>
> I dont see a problem with requiring community auth for downloading those
> links as well as raw messages..

I guess the question there is- how hard would it be for someone to
script the community auth process, so that they can automatically
download the mbox's each month, if they wish to..?

I'm guessing it's probably not that hard, but that's the one thing that
comes to mind regarding this proposal..  Otherwise I am generally in
support of requiring community auth for this instead of using the basic
auth method we have today.

Thanks,

Stephen

On Sun, Mar 29, 2020 at 9:08 PM Stephen Frost <sfrost@snowman.net> wrote:
>
> Greetings,
>
> * Stefan Kaltenbrunner (stefan@kaltenbrunner.cc) wrote:
> > On 3/26/20 3:44 PM, Magnus Hagander wrote:
> > > We could put it just to the right of the download link though? There
> > > seems to be enough space for that with a large margin at least on
> > > desktops.
> >
> > what about doing something completely different - like hiding those
> > links behind community auth and call it a day(especially with so few users)?
> >
> > I dont see a problem with requiring community auth for downloading those
> > links as well as raw messages..
>
> I guess the question there is- how hard would it be for someone to
> script the community auth process, so that they can automatically
> download the mbox's each month, if they wish to..?

Complicated enough that a process like that is going to be *really* annoying.


> I'm guessing it's probably not that hard, but that's the one thing that
> comes to mind regarding this proposal..  Otherwise I am generally in
> support of requiring community auth for this instead of using the basic
> auth method we have today.

One of the original usecases for "download thread as mbox" was you, I
believe , which was basically about opening a mailbox directly from
inside mutt or something? Isn't that also going to be a lot more
annoying from something like this?

-- 
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

On 3/29/20 9:24 PM, Magnus Hagander wrote:
> On Sun, Mar 29, 2020 at 9:08 PM Stephen Frost <sfrost@snowman.net> wrote:
>>
>> Greetings,
>>
>> * Stefan Kaltenbrunner (stefan@kaltenbrunner.cc) wrote:
>>> On 3/26/20 3:44 PM, Magnus Hagander wrote:
>>>> We could put it just to the right of the download link though? There
>>>> seems to be enough space for that with a large margin at least on
>>>> desktops.
>>>
>>> what about doing something completely different - like hiding those
>>> links behind community auth and call it a day(especially with so few users)?
>>>
>>> I dont see a problem with requiring community auth for downloading those
>>> links as well as raw messages..
>>
>> I guess the question there is- how hard would it be for someone to
>> script the community auth process, so that they can automatically
>> download the mbox's each month, if they wish to..?
> 
> Complicated enough that a process like that is going to be *really* annoying.

if we are just talking about providing a monthly mbox file to download
we could probably just provide a completely different way to provide
that data that does not involve dynamically generating it in real-time...

> 
> 
>> I'm guessing it's probably not that hard, but that's the one thing that
>> comes to mind regarding this proposal..  Otherwise I am generally in
>> support of requiring community auth for this instead of using the basic
>> auth method we have today.
> 
> One of the original usecases for "download thread as mbox" was you, I
> believe , which was basically about opening a mailbox directly from
> inside mutt or something? Isn't that also going to be a lot more
> annoying from something like this?

I dont understand that argument - so mutt directly does http and http
basic auth or what is the usecase? If it is just clicking in your
browser and have that link fed into mutt by means of mime-type or file
extension detection that would just work as before...



Stefan

Greetings,

* Stefan Kaltenbrunner (stefan@kaltenbrunner.cc) wrote:
> On 3/29/20 9:24 PM, Magnus Hagander wrote:
> > On Sun, Mar 29, 2020 at 9:08 PM Stephen Frost <sfrost@snowman.net> wrote:
> >> * Stefan Kaltenbrunner (stefan@kaltenbrunner.cc) wrote:
> >>> On 3/26/20 3:44 PM, Magnus Hagander wrote:
> >>>> We could put it just to the right of the download link though? There
> >>>> seems to be enough space for that with a large margin at least on
> >>>> desktops.
> >>>
> >>> what about doing something completely different - like hiding those
> >>> links behind community auth and call it a day(especially with so few users)?
> >>>
> >>> I dont see a problem with requiring community auth for downloading those
> >>> links as well as raw messages..
> >>
> >> I guess the question there is- how hard would it be for someone to
> >> script the community auth process, so that they can automatically
> >> download the mbox's each month, if they wish to..?
> >
> > Complicated enough that a process like that is going to be *really* annoying.

Hrmpf, really?

> if we are just talking about providing a monthly mbox file to download
> we could probably just provide a completely different way to provide
> that data that does not involve dynamically generating it in real-time...

Being able to download just a single thread as an mbox is really awful
handy.

> >> I'm guessing it's probably not that hard, but that's the one thing that
> >> comes to mind regarding this proposal..  Otherwise I am generally in
> >> support of requiring community auth for this instead of using the basic
> >> auth method we have today.
> >
> > One of the original usecases for "download thread as mbox" was you, I
> > believe , which was basically about opening a mailbox directly from
> > inside mutt or something? Isn't that also going to be a lot more
> > annoying from something like this?
>
> I dont understand that argument - so mutt directly does http and http
> basic auth or what is the usecase? If it is just clicking in your
> browser and have that link fed into mutt by means of mime-type or file
> extension detection that would just work as before...

The use-case is something with wget/curl to actually pull down the mbox
of the month or of the thread.  If it's actually hard to script one of
those to work with community auth then that would definitely be an issue
for at least some of the users of it.

I suppose there's another thought in here around having different
end-points to the archives, et al.  That is- keep the existing
end-points with basic auth but don't link to that from the web page
archive and instead have it linked from some other page that talks about
how to use basic auth to pull down the mboxes.  Then, on the web
archive, have a link to an end-point that uses community auth.

Thanks,

Stephen

On Sun, Mar 29, 2020 at 9:35 PM Stephen Frost <sfrost@snowman.net> wrote:
>
> Greetings,
>
> * Stefan Kaltenbrunner (stefan@kaltenbrunner.cc) wrote:
> > On 3/29/20 9:24 PM, Magnus Hagander wrote:
> > > On Sun, Mar 29, 2020 at 9:08 PM Stephen Frost <sfrost@snowman.net> wrote:
> > >> * Stefan Kaltenbrunner (stefan@kaltenbrunner.cc) wrote:
> > >>> On 3/26/20 3:44 PM, Magnus Hagander wrote:
> > >>>> We could put it just to the right of the download link though? There
> > >>>> seems to be enough space for that with a large margin at least on
> > >>>> desktops.
> > >>>
> > >>> what about doing something completely different - like hiding those
> > >>> links behind community auth and call it a day(especially with so few users)?
> > >>>
> > >>> I dont see a problem with requiring community auth for downloading those
> > >>> links as well as raw messages..
> > >>
> > >> I guess the question there is- how hard would it be for someone to
> > >> script the community auth process, so that they can automatically
> > >> download the mbox's each month, if they wish to..?
> > >
> > > Complicated enough that a process like that is going to be *really* annoying.
>
> Hrmpf, really?

It does become a multi-step process.

You have to go fetch the login page, then submit your login
credentials and get a cookie. Then you have to go to the download
page, follow the redirect from there up to the login system, provide
the cookie, follow the next redirect over to the receive page, get the
cookie from there, follow the next redirect back to the download url
and provide this new cookie.

So yeah, doable, but not exactly convenient.

And if you want to automate this it now means you are storing your
community auth password in a script somewhere, and not just a
throwaway password.


> > if we are just talking about providing a monthly mbox file to download
> > we could probably just provide a completely different way to provide
> > that data that does not involve dynamically generating it in real-time...
>
> Being able to download just a single thread as an mbox is really awful
> handy.

And regardless of where we provide them we still have the same
problem. The *reason* we have that password there is to prevent
automated scrapers from grabbing all that data. And that *does* work,
if we look at the logs of downloads of them. Yes, custom made scrapers
that want to grab that down (which is what it's designed for)
sometimes do, but we get *very* few downloads from automated scrapers
and crawlers.


> > >> I'm guessing it's probably not that hard, but that's the one thing that
> > >> comes to mind regarding this proposal..  Otherwise I am generally in
> > >> support of requiring community auth for this instead of using the basic
> > >> auth method we have today.
> > >
> > > One of the original usecases for "download thread as mbox" was you, I
> > > believe , which was basically about opening a mailbox directly from
> > > inside mutt or something? Isn't that also going to be a lot more
> > > annoying from something like this?
> >
> > I dont understand that argument - so mutt directly does http and http
> > basic auth or what is the usecase? If it is just clicking in your
> > browser and have that link fed into mutt by means of mime-type or file
> > extension detection that would just work as before...
>
> The use-case is something with wget/curl to actually pull down the mbox
> of the month or of the thread.  If it's actually hard to script one of
> those to work with community auth then that would definitely be an issue
> for at least some of the users of it.

Once you've reached the point of scripting it, though, you have
already read the docs :) (and in fact the code behind it all)


> I suppose there's another thought in here around having different
> end-points to the archives, et al.  That is- keep the existing
> end-points with basic auth but don't link to that from the web page
> archive and instead have it linked from some other page that talks about
> how to use basic auth to pull down the mboxes.  Then, on the web
> archive, have a link to an end-point that uses community auth.

We could take it one more step and just make it an intermediate page.
That is, the link that now goes to the mbox, would go to an
*individual* page for that mbox file that says "here are the
credentials, now press this big button below and input those
credentials". That button would point directly to the predictable URL
for that mbox, meaning that those who automate this somewhere can just
go directly to that URL, whereas those who are manually downloading it
for the first time gets to go to a page with instructions.

I think the only downside in the process there is those that manually
download it regularly and *do* remember the instructions -- they have
to click twice. But it's still just clicks, and very straight forward
clicks.

-- 
 Magnus Hagander
 Me: https://www.hagander.net/
 Work: https://www.redpill-linpro.com/

On 2020-03-30 17:27, Magnus Hagander wrote:
> And regardless of where we provide them we still have the same
> problem. The*reason*  we have that password there is to prevent
> automated scrapers from grabbing all that data. And that*does*  work,
> if we look at the logs of downloads of them. Yes, custom made scrapers
> that want to grab that down (which is what it's designed for)
> sometimes do, but we get*very*  few downloads from automated scrapers
> and crawlers.

Yeah, this thread is turning a literally one-line patch to refine an 
otherwise perfectly working system into a major reengineering project.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services