Thread: Re: No easy way to join discussion in existing thread when not subscribed
On 10/01/2015 04:29 PM, Stephen Frost wrote: > * Amir Rohan (amir.rohan@mail.com) wrote: >> I added a blurb to the wiki about this. What about the "Mail me this >> message" proposed earlier? I'd be glad to help make that happen. > > I don't think anyone's really against that idea, except perhaps that we > need to try and avoid letting it be abused. > > Of course, someone needs to implement it. On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote: > > yeah - as Stephen said upthread I think that would be a very useful > feature... > Great, here's a spec: 1) If the user is not logged in, error as the mbox downloads does. 2) If the user is logged in, retrieve the raw message from the db (like the "raw" link) does and send it via email (the system is already setup to do this) to the registered email address for the logged-in user. Threats: a1) Abusing the system to send lots of email to one victim. a2) Abusing the system to send one email to lots of victims. a3) DOS on the server through overuse by legitimate users. a4) DOS on the server through overuse by malicious users, possibly involving many accounts. To mitigate these, we: b1) Require a community login which involves an email verification step. mitigates (a1) and (a2). If a3 and a4 are concerns in practice: b2) Implement per-user rate limits. b3) Implement a global rate limit, to cap the size of mailqueue and reduce the damage which can be done by registering many accounts. If there's a kv store available (redis/memcached), we'll use that. Otherwise, CREATE TABLE rate_limit_mail_per_user( userid integer PRIMARY KEY, cnt integer NOTNULL, last_request_date timestamp NOT NULL ); CREATE INDEX on rate_limit_1(userid); Saturate cnt on settings.RATE_LIMIT_THRESHOLD, and reset cnt when a request comes in that is at least settings.RATE_LIMIT_DELAY seconds after the last one. The table size is bounded by the number of registered users, and it can be truncated daily. Comments? Amir
> From: "Amir Rohan" <amir.rohan@mail.com> > > CREATE TABLE rate_limit_mail_per_user( sorry, CREATE UNLOGGED TABLE rate_limit_mail_per_user( > userid integer PRIMARY KEY, > cnt integer NOTNULL, > last_request_date timestamp NOT NULL > );
Amir, * Amir Rohan (amir.rohan@mail.com) wrote: > On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote: > > yeah - as Stephen said upthread I think that would be a very useful > > feature... > > Great, here's a spec: > > 1) If the user is not logged in, error as the mbox downloads does. > 2) If the user is logged in, retrieve the raw message from the db (like > the "raw" link) does and send it via email (the system is already setup > to do this) to the registered email address for the logged-in user. > > Threats: > a1) Abusing the system to send lots of email to one victim. > a2) Abusing the system to send one email to lots of victims. > a3) DOS on the server through overuse by legitimate users. > a4) DOS on the server through overuse by malicious users, possibly > involving many accounts. > > To mitigate these, we: > b1) Require a community login which involves an email verification step. > mitigates (a1) and (a2). Works for me. > If a3 and a4 are concerns in practice: I don't see that being the case here and so I don't believe we need any particular safeguards for those cases. Further, if we do, they can always be added later and don't need to complicate the initial implementation. Thanks! Stephen
Re: No easy way to join discussion in existing thread when not subscribed
From
Stefan Kaltenbrunner
Date:
On 10/02/2015 02:45 PM, Stephen Frost wrote: > Amir, > > * Amir Rohan (amir.rohan@mail.com) wrote: >> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote: >>> yeah - as Stephen said upthread I think that would be a very useful >>> feature... >> >> Great, here's a spec: >> >> 1) If the user is not logged in, error as the mbox downloads does. >> 2) If the user is logged in, retrieve the raw message from the db (like >> the "raw" link) does and send it via email (the system is already setup >> to do this) to the registered email address for the logged-in user. >> >> Threats: >> a1) Abusing the system to send lots of email to one victim. >> a2) Abusing the system to send one email to lots of victims. >> a3) DOS on the server through overuse by legitimate users. >> a4) DOS on the server through overuse by malicious users, possibly >> involving many accounts. >> >> To mitigate these, we: >> b1) Require a community login which involves an email verification step. >> mitigates (a1) and (a2). > > Works for me. +1 > >> If a3 and a4 are concerns in practice: > > I don't see that being the case here and so I don't believe we need any > particular safeguards for those cases. > > Further, if we do, they can always be added later and don't need to > complicate the initial implementation. I agree there - we probably have other issues if somebody ends up creating thousends or more community accounts and if we need to ratelimit mail we can handle that on the MTA side as well... Stefan
On 10/02/2015 03:48 PM, Stefan Kaltenbrunner wrote: > On 10/02/2015 02:45 PM, Stephen Frost wrote: >> * Amir Rohan (amir.rohan@mail.com) wrote: >>> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote: >>>> yeah - as Stephen said upthread I think that would be a very useful >>>> feature... >>> >>> Great, here's a spec: >>> >>> 1) If the user is not logged in, error as the mbox downloads does. >>> 2) If the user is logged in, retrieve the raw message from the db (like >>> the "raw" link) does and send it via email (the system is already setup >>> to do this) to the registered email address for the logged-in user. >>> >>> Threats: >>> a1) Abusing the system to send lots of email to one victim. >>> a2) Abusing the system to send one email to lots of victims. >>> a3) DOS on the server through overuse by legitimate users. >>> a4) DOS on the server through overuse by malicious users, possibly >>> involving many accounts. >>> >>> To mitigate these, we: >>> b1) Require a community login which involves an email verification step. >>> mitigates (a1) and (a2). >> >> Works for me. > > +1 Tosay I sepent some time looking more closely at this. I'd like your input on a few decisions that needs to be made and some points that need to be verified if this is to work. pgarchives and pgweb are separate apps, each using its own db: - pgweb has the MailQueue model, archives does not. - pgarchives has the Message model, pgweb does not. To implement a "mail me this message", one way or another we need to iinteract with both. We can either: 1) Upon request, have archives redirect to a new endpoint on pgweb, which calls the archives "raw" endpoint to fetch the message, parses it and push a job to the mail queue. 2) Upon request, have archives POST the message (and user email,etc') to a new endpoint on pgweb. This needs to be inacccesible outside the LAN, because it's basically a spam service. 3) Add an a Mailqueue model to pgarchives, and have it mail things independently. This requires another cron job to drain the new queue. Which do you prefer? Next point, the Mailqueue is drained send_queued_mail.py, would sending these messages, with an envelope-to: header either through the local MTA or the relay be enough to get it delivered to the user, or will this trigger any anti-spam mechanisms along the way? If we use the local MTA, is it setup to rewrite any headers before delivery? Amir
Re: No easy way to join discussion in existing thread when not subscribed
From
Stefan Kaltenbrunner
Date:
On 10/05/2015 08:05 PM, Amir Rohan wrote: > On 10/02/2015 03:48 PM, Stefan Kaltenbrunner wrote: >> On 10/02/2015 02:45 PM, Stephen Frost wrote: >>> * Amir Rohan (amir.rohan@mail.com) wrote: >>>> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote: >>>>> yeah - as Stephen said upthread I think that would be a very useful >>>>> feature... >>>> >>>> Great, here's a spec: >>>> >>>> 1) If the user is not logged in, error as the mbox downloads does. >>>> 2) If the user is logged in, retrieve the raw message from the db (like >>>> the "raw" link) does and send it via email (the system is already setup >>>> to do this) to the registered email address for the logged-in user. >>>> >>>> Threats: >>>> a1) Abusing the system to send lots of email to one victim. >>>> a2) Abusing the system to send one email to lots of victims. >>>> a3) DOS on the server through overuse by legitimate users. >>>> a4) DOS on the server through overuse by malicious users, possibly >>>> involving many accounts. >>>> >>>> To mitigate these, we: >>>> b1) Require a community login which involves an email verification step. >>>> mitigates (a1) and (a2). >>> >>> Works for me. >> >> +1 > > Tosay I sepent some time looking more closely at this. I'd like your > input on a few decisions that needs to be made and some points > that need to be verified if this is to work. > > pgarchives and pgweb are separate apps, each using its own db: > - pgweb has the MailQueue model, archives does not. > - pgarchives has the Message model, pgweb does not. > > To implement a "mail me this message", one way or another we need > to iinteract with both. > > We can either: > 1) Upon request, have archives redirect to a new endpoint on pgweb, > which calls the archives "raw" endpoint to fetch the message, > parses it and push a job to the mail queue. > 2) Upon request, have archives POST the message (and user email,etc') > to a new endpoint on pgweb. This needs to be inacccesible outside > the LAN, because it's basically a spam service. > 3) Add an a Mailqueue model to pgarchives, and have it mail things > independently. This requires another cron job to drain the new queue. > > Which do you prefer? I dont think we want or need an endpoint on pgweb so my vote would be to simply send out the mails from the archives box, having a job there to drain the queue can easily be done so 3) looks pretty good to me > > Next point, the Mailqueue is drained send_queued_mail.py, > would sending these messages, with an envelope-to: header > either through the local MTA or the relay be enough to > get it delivered to the user, or will this trigger any > anti-spam mechanisms along the way? If we use the local MTA, > is it setup to rewrite any headers before delivery? it can use the local MTA (which in turn will automatically use an upstream relay but that is fine). As a general notice - our infrastructure is heavily distributed (see http://www.postgresql.org/about/servers/ for a list of virtual machine hosts - we have 60+ VMs on top) between europe and north america and in the general sense there is no "LAN" between those as in communication between those goes by default over an untrusted channel (ie internet). Stefan
On 10/05/2015 09:32 PM, Stefan Kaltenbrunner wrote: > On 10/05/2015 08:05 PM, Amir Rohan wrote: >> On 10/02/2015 03:48 PM, Stefan Kaltenbrunner wrote: >>> On 10/02/2015 02:45 PM, Stephen Frost wrote: >>>> * Amir Rohan (amir.rohan@mail.com) wrote: >>>>> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote: >>>>>> yeah - as Stephen said upthread I think that would be a very useful >>>>>> feature... >>>>> >>>>> Great, here's a spec: >>>>> >>>>> 1) If the user is not logged in, error as the mbox downloads does. >>>>> 2) If the user is logged in, retrieve the raw message from the db (like >>>>> the "raw" link) does and send it via email (the system is already setup >>>>> to do this) to the registered email address for the logged-in user. >>>>> >>>>> Threats: >>>>> a1) Abusing the system to send lots of email to one victim. >>>>> a2) Abusing the system to send one email to lots of victims. >>>>> a3) DOS on the server through overuse by legitimate users. >>>>> a4) DOS on the server through overuse by malicious users, possibly >>>>> involving many accounts. >>>>> >>>>> To mitigate these, we: >>>>> b1) Require a community login which involves an email verification step. >>>>> mitigates (a1) and (a2). >>>> >>>> Works for me. >>> >>> +1 >> >> Today I sepent some time looking more closely at this. I'd like your >> input on a few decisions that needs to be made and some points >> that need to be verified if this is to work. >> >> pgarchives and pgweb are separate apps, each using its own db: >> - pgweb has the MailQueue model, archives does not. >> - pgarchives has the Message model, pgweb does not. >> >> To implement a "mail me this message", one way or another we need >> to interact with both. >> >> We can either: >> 1) Upon request, have archives redirect to a new endpoint on pgweb, >> which calls the archives "raw" endpoint to fetch the message, >> parses it and push a job to the mail queue. >> 2) Upon request, have archives POST the message (and user email,etc') >> to a new endpoint on pgweb. This needs to be inaccessible outside >> the LAN, because it's basically a spam service. >> 3) Add an a Mailqueue model to pgarchives, and have it mail things >> independently. This requires another cron job to drain the new queue. >> >> Which do you prefer? > > I don't think we want or need an endpoint on pgweb so my vote would be to > simply send out the mails from the archives box, having a job there to > drain the queue can easily be done so 3) looks pretty good to me > >> >> Next point, the Mailqueue is drained send_queued_mail.py, >> would sending these messages, with an envelope-to: header >> either through the local MTA or the relay be enough to >> get it delivered to the user, or will this trigger any >> anti-spam mechanisms along the way? If we use the local MTA, >> is it setup to rewrite any headers before delivery? > > it can use the local MTA (which in turn will automatically use an > upstream relay but that is fine). > > As a general notice - our infrastructure is heavily distributed (see > http://www.postgresql.org/about/servers/ for a list of virtual machine > hosts - we have 60+ VMs on top) between europe and north america and in > the general sense there is no "LAN" between those as in communication > between those goes by default over an untrusted channel (ie internet). > > > > Stefan > Fine by me, that simplifies things. That only leaves the community login bit. pgarchives isn't setup to use sessions, and afaict pgweb uses the defaults setup which means sessions are stored in a table in its own database, and is not available to archives. So although archives sits under the same frontend and sees the session cookies, it can't get at the session data. I've read authentication.rst and the full "community auth" flow you provide is rather complex, would require creating a new key for use by archives, etc'. Is there a simpler way? that was the biggest benefit of opting for (1)/(2). Amir
On 10/05/2015 10:14 PM, Amir Rohan wrote: > On 10/05/2015 09:32 PM, Stefan Kaltenbrunner wrote: >> On 10/05/2015 08:05 PM, Amir Rohan wrote: >>> On 10/02/2015 03:48 PM, Stefan Kaltenbrunner wrote: >>>> On 10/02/2015 02:45 PM, Stephen Frost wrote: >>>>> * Amir Rohan (amir.rohan@mail.com) wrote: >>>>>> On 10/01/2015 09:18 PM, Stefan Kaltenbrunner wrote: >>>>>>> yeah - as Stephen said upthread I think that would be a very useful >>>>>>> feature... >>>>>> >>>>>> Great, here's a spec: >>>>>> >>>>>> 1) If the user is not logged in, error as the mbox downloads does. >>>>>> 2) If the user is logged in, retrieve the raw message from the db (like >>>>>> the "raw" link) does and send it via email (the system is already setup >>>>>> to do this) to the registered email address for the logged-in user. >>>>>> >>>>>> Threats: >>>>>> a1) Abusing the system to send lots of email to one victim. >>>>>> a2) Abusing the system to send one email to lots of victims. >>>>>> a3) DOS on the server through overuse by legitimate users. >>>>>> a4) DOS on the server through overuse by malicious users, possibly >>>>>> involving many accounts. >>>>>> >>>>>> To mitigate these, we: >>>>>> b1) Require a community login which involves an email verification step. >>>>>> mitigates (a1) and (a2). >>>>> >>>>> Works for me. >>>> >>>> +1 >>> >>> Today I sepent some time looking more closely at this. I'd like your >>> input on a few decisions that needs to be made and some points >>> that need to be verified if this is to work. >>> >>> pgarchives and pgweb are separate apps, each using its own db: >>> - pgweb has the MailQueue model, archives does not. >>> - pgarchives has the Message model, pgweb does not. >>> >>> To implement a "mail me this message", one way or another we need >>> to interact with both. >>> >>> We can either: >>> 1) Upon request, have archives redirect to a new endpoint on pgweb, >>> which calls the archives "raw" endpoint to fetch the message, >>> parses it and push a job to the mail queue. >>> 2) Upon request, have archives POST the message (and user email,etc') >>> to a new endpoint on pgweb. This needs to be inaccessible outside >>> the LAN, because it's basically a spam service. >>> 3) Add an a Mailqueue model to pgarchives, and have it mail things >>> independently. This requires another cron job to drain the new queue. >>> >>> Which do you prefer? >> >> I don't think we want or need an endpoint on pgweb so my vote would be to >> simply send out the mails from the archives box, having a job there to >> drain the queue can easily be done so 3) looks pretty good to me >> >>> >>> Next point, the Mailqueue is drained send_queued_mail.py, >>> would sending these messages, with an envelope-to: header >>> either through the local MTA or the relay be enough to >>> get it delivered to the user, or will this trigger any >>> anti-spam mechanisms along the way? If we use the local MTA, >>> is it setup to rewrite any headers before delivery? >> >> it can use the local MTA (which in turn will automatically use an >> upstream relay but that is fine). >> >> As a general notice - our infrastructure is heavily distributed (see >> http://www.postgresql.org/about/servers/ for a list of virtual machine >> hosts - we have 60+ VMs on top) between europe and north america and in >> the general sense there is no "LAN" between those as in communication >> between those goes by default over an untrusted channel (ie internet). >> >> Stefan >> > > Fine by me, that simplifies things. That only leaves the community > login bit. pgarchives isn't setup to use sessions, and afaict > pgweb uses the defaults setup which means sessions are stored > in a table in its own database, and is not available to archives. > So although archives sits under the same frontend and sees the > session cookies, it can't get at the session data. > I've read authentication.rst and the full "community auth" flow > you provide is rather complex, would require creating a new > key for use by archives, etc'. Is there a simpler way? that was > the biggest benefit of opting for (1)/(2). > > Amir Learned something new about SMTP today and I've now got this feature working (mod community login). I've added a "via email" link beside the "raw" link. Clicking it POSTs the msgid to a new view which fetches the raw message and puts it in the new Mailqueue together with user's login email (mocked, now) for the worker to process. The view itself displays an "expect an email, returning you to..." message and meta-refreshes back to the message view after a few seconds. It Seems to work quite well. To make use of community login I'll either need a way to get at the session from pgweb's db, or whatever is involved in establishing the archives app as an auth consumer from pgweb. Both unfortunately depend on information I don't have. Regards, Amir
Hi, See attached patch implementing this feature, as discussed in the "No easy way to join existing discussion when not subscribed" thread <56138D2D.1000304@zoho.com>. As my troublesome mail client turned that thread into a mess, this should server as a fresh start. The patch adds a "via email" link beside "[raw] or [whole thread]" in the message view. A Mailqueue implementation was copied verbatim from pgweb and so the mail worker at `tools/mailqueue/send_queued_mail.py` needs to run periodically to drain the queue. It was agreed that restricting this feature to logged-in community users was necessary and sufficient, so I have added the necessary decorators to the view. Because the archive app isn't currently hooked in to the community login system, and because I cannot fix that without aid from the web team, they are commented out currently. Optional patch 0002 uses django's "flash" messages extension to provide visual feedback to the user using a very soothing shade of green. Screenshot: http://bit.ly/1R71P6U Regards, Amir
Attachment
On 10/12/2015 04:56 PM, Amir Rohan wrote: > Hi, > > See attached patch implementing this feature, as discussed in the > "No easy way to join existing discussion when not subscribed" thread > <56138D2D.1000304@zoho.com>. > As my troublesome mail client turned that thread into a mess, this > should server as a fresh start. > > The patch adds a "via email" link beside "[raw] or [whole thread]" > in the message view. A Mailqueue implementation was copied verbatim > from pgweb and so the mail worker at > `tools/mailqueue/send_queued_mail.py` needs to run periodically to drain > the queue. > > > It was agreed that restricting this feature to logged-in community users > was necessary and sufficient, so I have added the necessary decorators > to the view. Because the archive app isn't currently > hooked in to the community login system, and because I cannot fix that > without aid from the web team, they are commented out currently. > > Optional patch 0002 uses django's "flash" messages extension to provide > visual feedback to the user using a very soothing shade of green. > > Screenshot: http://bit.ly/1R71P6U > > Regards, > Amir > Bump. Is there something more I can do to get this live on the archives? I've wanted to actually make use of this several times since implementing it. Regards, Amir
Amir, * Amir Rohan (amir.rohan@zoho.com) wrote: > Bump. Is there something more I can do to get this live on the archives? > I've wanted to actually make use of this several times since > implementing it. Thanks for bumping it. I'm afraid that we're currently waiting on Magnus to have time to formally review the changes, which is unlikely to happen this week or next as he's heavily involved in the PGConf.EU conference. I agree that it'd be quite nice to have the feature, as I believe others do, and hopefully Magnus will have time to review it after PGConf.EU. Thanks! Stephen
On 10/21/2015 05:21 PM, Stephen Frost wrote: > Amir, > > * Amir Rohan (amir.rohan@zoho.com) wrote: >> Bump. Is there something more I can do to get this live on the archives? >> I've wanted to actually make use of this several times since >> implementing it. > > Thanks for bumping it. > > I'm afraid that we're currently waiting on Magnus to have time to > formally review the changes, which is unlikely to happen this week or > next as he's heavily involved in the PGConf.EU conference. > > I agree that it'd be quite nice to have the feature, as I believe others > do, and hopefully Magnus will have time to review it after PGConf.EU. > > Thanks! > > Stephen > Ah, that explains it. Thanks for letting me know. Amir
On 10/12/2015 04:56 PM, Amir Rohan wrote: > Hi, > > See attached patch implementing this feature, as discussed in the > "No easy way to join existing discussion when not subscribed" thread > <56138D2D.1000304@zoho.com>. > As my troublesome mail client turned that thread into a mess, this > should server as a fresh start. > > The patch adds a "via email" link beside "[raw] or [whole thread]" > in the message view. A Mailqueue implementation was copied verbatim > from pgweb and so the mail worker at > `tools/mailqueue/send_queued_mail.py` needs to run periodically to drain > the queue. > > > It was agreed that restricting this feature to logged-in community users > was necessary and sufficient, so I have added the necessary decorators > to the view. Because the archive app isn't currently > hooked in to the community login system, and because I cannot fix that > without aid from the web team, they are currently commented out. > > Optional patch 0002 uses django's "flash" messages extension to provide > visual feedback to the user using a very soothing shade of green. > > Screenshot: http://bit.ly/1R71P6U > On 10/21/2015 04:51 PM, Amir Rohan wrote: > Bump. Is there something more I can do to get this live on the > archives? > I've wanted to actually make use of this several times since > implementing it. On 10/21/2015 05:21 PM, Stephen Frost wrote: > I'm afraid that we're currently waiting on Magnus to have time to > formally review the changes, which is unlikely to happen this week or > next as he's heavily involved in the PGConf.EU conference. Bump. Still waiting for help from the web team to implement community auth in the archives app in order to get this on the website. Amir