Thread: Re: No easy way to join discussion in existing thread when not subscribed

Re: No easy way to join discussion in existing thread when not subscribed

From
"Amir Rohan"
Date:
On 10/01/2015 04:29 PM, Stephen Frost wrote:
> * Amir Rohan (amir.rohan@mail.com) wrote:
>> I added a blurb to the wiki about this. What about the "Mail me this
>> message" proposed earlier? I'd be glad to help make that happen.
> 
> I don't think anyone's really against that idea, except perhaps that we
> need to try and avoid letting it be abused.
> 
> Of course, someone needs to implement it.


On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote:
>
> yeah - as Stephen said upthread I think that would be a very useful
> feature...
>

Great, here's a spec:

1) If the user is not logged in, error as the mbox downloads does.
2) If the user is logged in, retrieve the raw message from the db (like
the "raw" link) does and send it via email (the system is already setup
to do this) to the registered email address for the logged-in user.

Threats:
a1) Abusing the system to send lots of email to one victim.
a2) Abusing the system to send one email to lots of victims.
a3) DOS on the server through overuse by legitimate users.
a4) DOS on the server through overuse by malicious users, possibly
involving many accounts.

To mitigate these, we:
b1) Require a community login which involves an email verification step.
mitigates (a1) and (a2).

If a3 and a4 are concerns in practice:

b2) Implement per-user rate limits.
b3) Implement a global rate limit, to cap the size of mailqueue and
reduce the damage which can be done by registering many accounts.

If there's a kv store available (redis/memcached), we'll use that.
Otherwise,

CREATE TABLE rate_limit_mail_per_user( userid integer PRIMARY KEY, cnt integer NOTNULL, last_request_date timestamp NOT
NULL
);

CREATE INDEX on rate_limit_1(userid);

Saturate cnt on settings.RATE_LIMIT_THRESHOLD, and reset cnt when a
request comes in that is at least settings.RATE_LIMIT_DELAY seconds
after the last one.
The table size is bounded by the number of registered users, and it can
be truncated daily.

Comments?
Amir




Re: No easy way to join discussion in existing thread when not subscribed

From
"Amir Rohan"
Date:
> From: "Amir Rohan" <amir.rohan@mail.com>
>
> CREATE TABLE rate_limit_mail_per_user(

sorry, 

CREATE UNLOGGED TABLE rate_limit_mail_per_user(

>   userid integer PRIMARY KEY,
>   cnt integer NOTNULL,
>   last_request_date timestamp NOT NULL
> );



Re: No easy way to join discussion in existing thread when not subscribed

From
Stephen Frost
Date:
Amir,

* Amir Rohan (amir.rohan@mail.com) wrote:
> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote:
> > yeah - as Stephen said upthread I think that would be a very useful
> > feature...
>
> Great, here's a spec:
>
> 1) If the user is not logged in, error as the mbox downloads does.
> 2) If the user is logged in, retrieve the raw message from the db (like
> the "raw" link) does and send it via email (the system is already setup
> to do this) to the registered email address for the logged-in user.
>
> Threats:
> a1) Abusing the system to send lots of email to one victim.
> a2) Abusing the system to send one email to lots of victims.
> a3) DOS on the server through overuse by legitimate users.
> a4) DOS on the server through overuse by malicious users, possibly
> involving many accounts.
>
> To mitigate these, we:
> b1) Require a community login which involves an email verification step.
> mitigates (a1) and (a2).

Works for me.

> If a3 and a4 are concerns in practice:

I don't see that being the case here and so I don't believe we need any
particular safeguards for those cases.

Further, if we do, they can always be added later and don't need to
complicate the initial implementation.

Thanks!

Stephen

Re: No easy way to join discussion in existing thread when not subscribed

From
Stefan Kaltenbrunner
Date:
On 10/02/2015 02:45 PM, Stephen Frost wrote:
> Amir,
>
> * Amir Rohan (amir.rohan@mail.com) wrote:
>> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote:
>>> yeah - as Stephen said upthread I think that would be a very useful
>>> feature...
>>
>> Great, here's a spec:
>>
>> 1) If the user is not logged in, error as the mbox downloads does.
>> 2) If the user is logged in, retrieve the raw message from the db (like
>> the "raw" link) does and send it via email (the system is already setup
>> to do this) to the registered email address for the logged-in user.
>>
>> Threats:
>> a1) Abusing the system to send lots of email to one victim.
>> a2) Abusing the system to send one email to lots of victims.
>> a3) DOS on the server through overuse by legitimate users.
>> a4) DOS on the server through overuse by malicious users, possibly
>> involving many accounts.
>>
>> To mitigate these, we:
>> b1) Require a community login which involves an email verification step.
>> mitigates (a1) and (a2).
>
> Works for me.

+1

>
>> If a3 and a4 are concerns in practice:
>
> I don't see that being the case here and so I don't believe we need any
> particular safeguards for those cases.
>
> Further, if we do, they can always be added later and don't need to
> complicate the initial implementation.


I agree there - we probably have other issues if somebody ends up 
creating thousends or more community accounts and if we need to 
ratelimit mail we can handle that on the MTA side as well...



Stefan



Re: No easy way to join discussion in existing thread when not subscribed

From
Amir Rohan
Date:
On 10/02/2015 03:48 PM, Stefan Kaltenbrunner wrote:
> On 10/02/2015 02:45 PM, Stephen Frost wrote:
>> * Amir Rohan (amir.rohan@mail.com) wrote:
>>> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote:
>>>> yeah - as Stephen said upthread I think that would be a very useful
>>>> feature...
>>>
>>> Great, here's a spec:
>>>
>>> 1) If the user is not logged in, error as the mbox downloads does.
>>> 2) If the user is logged in, retrieve the raw message from the db (like
>>> the "raw" link) does and send it via email (the system is already setup
>>> to do this) to the registered email address for the logged-in user.
>>>
>>> Threats:
>>> a1) Abusing the system to send lots of email to one victim.
>>> a2) Abusing the system to send one email to lots of victims.
>>> a3) DOS on the server through overuse by legitimate users.
>>> a4) DOS on the server through overuse by malicious users, possibly
>>> involving many accounts.
>>>
>>> To mitigate these, we:
>>> b1) Require a community login which involves an email verification step.
>>> mitigates (a1) and (a2).
>>
>> Works for me.
> 
> +1

Tosay I sepent some time looking more closely at this. I'd like your
input on a few decisions that needs to be made and some points
that need to be verified if this is to work.

pgarchives and pgweb are separate apps, each using its own db:
- pgweb has the MailQueue model, archives does not.
- pgarchives has the Message model, pgweb does not.

To implement a "mail me this message", one way or another we need
to iinteract with both.

We can either:
1) Upon request, have archives redirect to a new endpoint on pgweb,
which calls the archives "raw" endpoint to fetch the message,
parses it and push a job to the mail queue.
2) Upon request, have archives POST the message (and user email,etc')
to a new endpoint on pgweb. This needs to be inacccesible outside
the LAN, because it's basically a spam service.
3) Add an a Mailqueue model to pgarchives, and have it mail things
independently. This requires another cron job to drain the new queue.

Which do you prefer?

Next point, the Mailqueue is drained send_queued_mail.py,
would sending these messages, with an envelope-to: header
either through the local MTA or the relay be enough to
get it delivered to the user, or will this trigger any
anti-spam mechanisms along the way? If we use the local MTA,
is it setup to rewrite any headers before delivery?

Amir




Re: No easy way to join discussion in existing thread when not subscribed

From
Stefan Kaltenbrunner
Date:
On 10/05/2015 08:05 PM, Amir Rohan wrote:
> On 10/02/2015 03:48 PM, Stefan Kaltenbrunner wrote:
>> On 10/02/2015 02:45 PM, Stephen Frost wrote:
>>> * Amir Rohan (amir.rohan@mail.com) wrote:
>>>> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote:
>>>>> yeah - as Stephen said upthread I think that would be a very useful
>>>>> feature...
>>>>
>>>> Great, here's a spec:
>>>>
>>>> 1) If the user is not logged in, error as the mbox downloads does.
>>>> 2) If the user is logged in, retrieve the raw message from the db (like
>>>> the "raw" link) does and send it via email (the system is already setup
>>>> to do this) to the registered email address for the logged-in user.
>>>>
>>>> Threats:
>>>> a1) Abusing the system to send lots of email to one victim.
>>>> a2) Abusing the system to send one email to lots of victims.
>>>> a3) DOS on the server through overuse by legitimate users.
>>>> a4) DOS on the server through overuse by malicious users, possibly
>>>> involving many accounts.
>>>>
>>>> To mitigate these, we:
>>>> b1) Require a community login which involves an email verification step.
>>>> mitigates (a1) and (a2).
>>>
>>> Works for me.
>>
>> +1
> 
> Tosay I sepent some time looking more closely at this. I'd like your
> input on a few decisions that needs to be made and some points
> that need to be verified if this is to work.
> 
> pgarchives and pgweb are separate apps, each using its own db:
> - pgweb has the MailQueue model, archives does not.
> - pgarchives has the Message model, pgweb does not.
> 
> To implement a "mail me this message", one way or another we need
> to iinteract with both.
> 
> We can either:
> 1) Upon request, have archives redirect to a new endpoint on pgweb,
> which calls the archives "raw" endpoint to fetch the message,
> parses it and push a job to the mail queue.
> 2) Upon request, have archives POST the message (and user email,etc')
> to a new endpoint on pgweb. This needs to be inacccesible outside
> the LAN, because it's basically a spam service.
> 3) Add an a Mailqueue model to pgarchives, and have it mail things
> independently. This requires another cron job to drain the new queue.
> 
> Which do you prefer?

I dont think we want or need an endpoint on pgweb so my vote would be to
simply send out the mails from the archives box, having a job there to
drain the queue can easily be done so 3) looks pretty good to me

> 
> Next point, the Mailqueue is drained send_queued_mail.py,
> would sending these messages, with an envelope-to: header
> either through the local MTA or the relay be enough to
> get it delivered to the user, or will this trigger any
> anti-spam mechanisms along the way? If we use the local MTA,
> is it setup to rewrite any headers before delivery?

it can use the local MTA (which in turn will automatically use an
upstream relay but that is fine).

As a general notice - our infrastructure is heavily distributed (see
http://www.postgresql.org/about/servers/ for a list of virtual machine
hosts - we have 60+ VMs on top) between europe and north america and in
the general sense there is no "LAN" between those as in communication
between those goes by default over an untrusted channel (ie internet).



Stefan



Re: No easy way to join discussion in existing thread when not subscribed

From
Amir Rohan
Date:
On 10/05/2015 09:32 PM, Stefan Kaltenbrunner wrote:
> On 10/05/2015 08:05 PM, Amir Rohan wrote:
>> On 10/02/2015 03:48 PM, Stefan Kaltenbrunner wrote:
>>> On 10/02/2015 02:45 PM, Stephen Frost wrote:
>>>> * Amir Rohan (amir.rohan@mail.com) wrote:
>>>>> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote:
>>>>>> yeah - as Stephen said upthread I think that would be a very useful
>>>>>> feature...
>>>>>
>>>>> Great, here's a spec:
>>>>>
>>>>> 1) If the user is not logged in, error as the mbox downloads does.
>>>>> 2) If the user is logged in, retrieve the raw message from the db (like
>>>>> the "raw" link) does and send it via email (the system is already setup
>>>>> to do this) to the registered email address for the logged-in user.
>>>>>
>>>>> Threats:
>>>>> a1) Abusing the system to send lots of email to one victim.
>>>>> a2) Abusing the system to send one email to lots of victims.
>>>>> a3) DOS on the server through overuse by legitimate users.
>>>>> a4) DOS on the server through overuse by malicious users, possibly
>>>>> involving many accounts.
>>>>>
>>>>> To mitigate these, we:
>>>>> b1) Require a community login which involves an email verification step.
>>>>> mitigates (a1) and (a2).
>>>>
>>>> Works for me.
>>>
>>> +1
>>
>> Today I sepent some time looking more closely at this. I'd like your
>> input on a few decisions that needs to be made and some points
>> that need to be verified if this is to work.
>>
>> pgarchives and pgweb are separate apps, each using its own db:
>> - pgweb has the MailQueue model, archives does not.
>> - pgarchives has the Message model, pgweb does not.
>>
>> To implement a "mail me this message", one way or another we need
>> to interact with both.
>>
>> We can either:
>> 1) Upon request, have archives redirect to a new endpoint on pgweb,
>> which calls the archives "raw" endpoint to fetch the message,
>> parses it and push a job to the mail queue.
>> 2) Upon request, have archives POST the message (and user email,etc')
>> to a new endpoint on pgweb. This needs to be inaccessible outside
>> the LAN, because it's basically a spam service.
>> 3) Add an a Mailqueue model to pgarchives, and have it mail things
>> independently. This requires another cron job to drain the new queue.
>>
>> Which do you prefer?
> 
> I don't think we want or need an endpoint on pgweb so my vote would be to
> simply send out the mails from the archives box, having a job there to
> drain the queue can easily be done so 3) looks pretty good to me
> 
>>
>> Next point, the Mailqueue is drained send_queued_mail.py,
>> would sending these messages, with an envelope-to: header
>> either through the local MTA or the relay be enough to
>> get it delivered to the user, or will this trigger any
>> anti-spam mechanisms along the way? If we use the local MTA,
>> is it setup to rewrite any headers before delivery?
> 
> it can use the local MTA (which in turn will automatically use an
> upstream relay but that is fine).
> 
> As a general notice - our infrastructure is heavily distributed (see
> http://www.postgresql.org/about/servers/ for a list of virtual machine
> hosts - we have 60+ VMs on top) between europe and north america and in
> the general sense there is no "LAN" between those as in communication
> between those goes by default over an untrusted channel (ie internet).
> 
> 
> 
> Stefan
> 

Fine by me, that simplifies things. That only leaves the community
login bit. pgarchives isn't setup to use sessions, and afaict
pgweb uses the defaults setup which means sessions are stored
in a table in its own database, and is not available to archives.
So although archives sits under the same frontend and sees the
session cookies, it can't get at the session data.
I've read authentication.rst and the full "community auth" flow
you provide is rather complex, would require creating a new
key for use by archives, etc'. Is there a simpler way? that was
the biggest benefit of opting for (1)/(2).

Amir





Re: No easy way to join discussion in existing thread when not subscribed

From
Amir Rohan
Date:
On 10/05/2015 10:14 PM, Amir Rohan wrote:
> On 10/05/2015 09:32 PM, Stefan Kaltenbrunner wrote:
>> On 10/05/2015 08:05 PM, Amir Rohan wrote:
>>> On 10/02/2015 03:48 PM, Stefan Kaltenbrunner wrote:
>>>> On 10/02/2015 02:45 PM, Stephen Frost wrote:
>>>>> * Amir Rohan (amir.rohan@mail.com) wrote:
>>>>>> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote:
>>>>>>> yeah - as Stephen said upthread I think that would be a very useful
>>>>>>> feature...
>>>>>>
>>>>>> Great, here's a spec:
>>>>>>
>>>>>> 1) If the user is not logged in, error as the mbox downloads does.
>>>>>> 2) If the user is logged in, retrieve the raw message from the db (like
>>>>>> the "raw" link) does and send it via email (the system is already setup
>>>>>> to do this) to the registered email address for the logged-in user.
>>>>>>
>>>>>> Threats:
>>>>>> a1) Abusing the system to send lots of email to one victim.
>>>>>> a2) Abusing the system to send one email to lots of victims.
>>>>>> a3) DOS on the server through overuse by legitimate users.
>>>>>> a4) DOS on the server through overuse by malicious users, possibly
>>>>>> involving many accounts.
>>>>>>
>>>>>> To mitigate these, we:
>>>>>> b1) Require a community login which involves an email verification step.
>>>>>> mitigates (a1) and (a2).
>>>>>
>>>>> Works for me.
>>>>
>>>> +1
>>>
>>> Today I sepent some time looking more closely at this. I'd like your
>>> input on a few decisions that needs to be made and some points
>>> that need to be verified if this is to work.
>>>
>>> pgarchives and pgweb are separate apps, each using its own db:
>>> - pgweb has the MailQueue model, archives does not.
>>> - pgarchives has the Message model, pgweb does not.
>>>
>>> To implement a "mail me this message", one way or another we need
>>> to interact with both.
>>>
>>> We can either:
>>> 1) Upon request, have archives redirect to a new endpoint on pgweb,
>>> which calls the archives "raw" endpoint to fetch the message,
>>> parses it and push a job to the mail queue.
>>> 2) Upon request, have archives POST the message (and user email,etc')
>>> to a new endpoint on pgweb. This needs to be inaccessible outside
>>> the LAN, because it's basically a spam service.
>>> 3) Add an a Mailqueue model to pgarchives, and have it mail things
>>> independently. This requires another cron job to drain the new queue.
>>>
>>> Which do you prefer?
>>
>> I don't think we want or need an endpoint on pgweb so my vote would be to
>> simply send out the mails from the archives box, having a job there to
>> drain the queue can easily be done so 3) looks pretty good to me
>>
>>>
>>> Next point, the Mailqueue is drained send_queued_mail.py,
>>> would sending these messages, with an envelope-to: header
>>> either through the local MTA or the relay be enough to
>>> get it delivered to the user, or will this trigger any
>>> anti-spam mechanisms along the way? If we use the local MTA,
>>> is it setup to rewrite any headers before delivery?
>>
>> it can use the local MTA (which in turn will automatically use an
>> upstream relay but that is fine).
>>
>> As a general notice - our infrastructure is heavily distributed (see
>> http://www.postgresql.org/about/servers/ for a list of virtual machine
>> hosts - we have 60+ VMs on top) between europe and north america and in
>> the general sense there is no "LAN" between those as in communication
>> between those goes by default over an untrusted channel (ie internet).
>>
>> Stefan
>>
> 
> Fine by me, that simplifies things. That only leaves the community
> login bit. pgarchives isn't setup to use sessions, and afaict
> pgweb uses the defaults setup which means sessions are stored
> in a table in its own database, and is not available to archives.
> So although archives sits under the same frontend and sees the
> session cookies, it can't get at the session data.
> I've read authentication.rst and the full "community auth" flow
> you provide is rather complex, would require creating a new
> key for use by archives, etc'. Is there a simpler way? that was
> the biggest benefit of opting for (1)/(2).
> 
> Amir 

Learned something new about SMTP today and I've now got this feature
working (mod community login).

I've added a "via email" link beside the "raw" link. Clicking it POSTs
the msgid to a new view which fetches the raw message and puts it in the
new Mailqueue
together with user's login email (mocked, now) for the worker to process.
The view itself displays an "expect an email, returning you to..."
message and meta-refreshes back to the message view after a
few seconds. It Seems to work quite well.

To make use of community login I'll either need a way to get at
the session from pgweb's db, or whatever is involved in establishing the
archives app as an auth consumer from pgweb. Both unfortunately depend
on information I don't have.

Regards,
Amir




PATCH: Add "mail me this message" feature to archive viewer

From
Amir Rohan
Date:
Hi,

See attached patch implementing this feature, as discussed in the
"No easy way to join existing discussion when not subscribed" thread
<56138D2D.1000304@zoho.com>.
As my troublesome mail client turned that thread into a mess, this
should server as a fresh start.

The patch adds a "via email" link beside "[raw] or [whole thread]"
in the message view. A Mailqueue implementation was copied verbatim
from pgweb and so the mail worker at
`tools/mailqueue/send_queued_mail.py` needs to run periodically to drain
the queue.


It was agreed that restricting this feature to logged-in community users
was necessary and sufficient, so I have added the necessary decorators
to the view. Because the archive app isn't currently
hooked in to the community login system, and because I cannot fix that
without aid from the web team, they are commented out currently.

Optional patch 0002 uses django's "flash" messages extension to provide
visual feedback to the user using a very soothing shade of green.

Screenshot: http://bit.ly/1R71P6U

Regards,
Amir


Attachment

Re: PATCH: Add "mail me this message" feature to archive viewer

From
Amir Rohan
Date:
On 10/12/2015 04:56 PM, Amir Rohan wrote:
> Hi,
> 
> See attached patch implementing this feature, as discussed in the
> "No easy way to join existing discussion when not subscribed" thread
> <56138D2D.1000304@zoho.com>.
> As my troublesome mail client turned that thread into a mess, this
> should server as a fresh start.
> 
> The patch adds a "via email" link beside "[raw] or [whole thread]"
> in the message view. A Mailqueue implementation was copied verbatim
> from pgweb and so the mail worker at
> `tools/mailqueue/send_queued_mail.py` needs to run periodically to drain
> the queue.
> 
> 
> It was agreed that restricting this feature to logged-in community users
> was necessary and sufficient, so I have added the necessary decorators
> to the view. Because the archive app isn't currently
> hooked in to the community login system, and because I cannot fix that
> without aid from the web team, they are commented out currently.
> 
> Optional patch 0002 uses django's "flash" messages extension to provide
> visual feedback to the user using a very soothing shade of green.
> 
> Screenshot: http://bit.ly/1R71P6U
> 
> Regards,
> Amir
> 

Bump. Is there something more I can do to get this live on the archives?
I've wanted to actually make use of this several times since
implementing it.

Regards,
Amir




Re: PATCH: Add "mail me this message" feature to archive viewer

From
Stephen Frost
Date:
Amir,

* Amir Rohan (amir.rohan@zoho.com) wrote:
> Bump. Is there something more I can do to get this live on the archives?
> I've wanted to actually make use of this several times since
> implementing it.

Thanks for bumping it.

I'm afraid that we're currently waiting on Magnus to have time to
formally review the changes, which is unlikely to happen this week or
next as he's heavily involved in the PGConf.EU conference.

I agree that it'd be quite nice to have the feature, as I believe others
do, and hopefully Magnus will have time to review it after PGConf.EU.

Thanks!

Stephen

Re: PATCH: Add "mail me this message" feature to archive viewer

From
Amir Rohan
Date:
On 10/21/2015 05:21 PM, Stephen Frost wrote:
> Amir,
> 
> * Amir Rohan (amir.rohan@zoho.com) wrote:
>> Bump. Is there something more I can do to get this live on the archives?
>> I've wanted to actually make use of this several times since
>> implementing it.
> 
> Thanks for bumping it.
> 
> I'm afraid that we're currently waiting on Magnus to have time to
> formally review the changes, which is unlikely to happen this week or
> next as he's heavily involved in the PGConf.EU conference.
> 
> I agree that it'd be quite nice to have the feature, as I believe others
> do, and hopefully Magnus will have time to review it after PGConf.EU.
> 
> Thanks!
> 
> Stephen
> 

Ah, that explains it. Thanks for letting me know.

Amir




Re: PATCH: Add "mail me this message" feature to archive viewer

From
Amir Rohan
Date:
On 10/12/2015 04:56 PM, Amir Rohan wrote:
> Hi,
> 
> See attached patch implementing this feature, as discussed in the
> "No easy way to join existing discussion when not subscribed" thread
> <56138D2D.1000304@zoho.com>.
> As my troublesome mail client turned that thread into a mess, this
> should server as a fresh start.
> 
> The patch adds a "via email" link beside "[raw] or [whole thread]"
> in the message view. A Mailqueue implementation was copied verbatim
> from pgweb and so the mail worker at
> `tools/mailqueue/send_queued_mail.py` needs to run periodically to drain
> the queue.
> 
> 
> It was agreed that restricting this feature to logged-in community users
> was necessary and sufficient, so I have added the necessary decorators
> to the view. Because the archive app isn't currently
> hooked in to the community login system, and because I cannot fix that
> without aid from the web team, they are currently commented out.
> 
> Optional patch 0002 uses django's "flash" messages extension to provide
> visual feedback to the user using a very soothing shade of green.
> 
> Screenshot: http://bit.ly/1R71P6U
> 

On 10/21/2015 04:51 PM, Amir Rohan wrote:
> Bump. Is there something more I can do to get this live on the
> archives?
> I've wanted to actually make use of this several times since
> implementing it.

On 10/21/2015 05:21 PM, Stephen Frost wrote:
> I'm afraid that we're currently waiting on Magnus to have time to
> formally review the changes, which is unlikely to happen this week or
> next as he's heavily involved in the PGConf.EU conference.

Bump. Still waiting for help from the web team to implement community
auth in the archives app in order to get this on the website.

Amir