Thread: New email address
Yahoo recently changed their DMARC policy, and after some investigation and a support case with Yahoo, it is now clear that their email systems can no longer be used with the postgresql.org lists. I've migrated from kgrittn@ymail.com to kgrittn@gmail.com. In the process I noticed that some people have been sending mail intended for my attention to the kgrittn@mail.com address that I migrated from years ago. Because of MajorDomo elimination of duplicates, replying to a COMMITTERS message and adding my old email address caused it to go to the email equivalent of /dev/null. Apologies for what I missed due to that. I've dropped old addresses from the MajorDomo aliases list, which should help prevent a recurrence. You may want to to purge the old addresses from any address book entries for me. After a little settling time I will set up auto-reply messages to point anyone who sends to the addresses to the current address. -- Kevin Grittner EDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Tue, Nov 24, 2015 at 3:41 AM, Kevin Grittner <kgrittn@gmail.com> wrote: > Yahoo recently changed their DMARC policy, and after some > investigation and a support case with Yahoo, it is now clear that > their email systems can no longer be used with the postgresql.org > lists. I've migrated from kgrittn@ymail.com to kgrittn@gmail.com. Something to be aware of as well: I noticed that sometimes your emails coming from @ymail.com were flagged as spam by gmail. People be careful of that if you use it. -- Michael
<p dir="ltr">Le 24 nov. 2015 01:05, "Michael Paquier" <<a href="mailto:michael.paquier@gmail.com">michael.paquier@gmail.com</a>>a écrit :<br /> ><br /> > On Tue, Nov 24,2015 at 3:41 AM, Kevin Grittner <<a href="mailto:kgrittn@gmail.com">kgrittn@gmail.com</a>> wrote:<br /> > >Yahoo recently changed their DMARC policy, and after some<br /> > > investigation and a support case with Yahoo,it is now clear that<br /> > > their email systems can no longer be used with the <a href="http://postgresql.org">postgresql.org</a><br/> > > lists. I've migrated from <a href="mailto:kgrittn@ymail.com">kgrittn@ymail.com</a>to <a href="mailto:kgrittn@gmail.com">kgrittn@gmail.com</a>.<br /> ><br/> > Something to be aware of as well: I noticed that sometimes your emails<br /> > coming from @<a href="http://ymail.com">ymail.com</a>were flagged as spam by gmail. People be<br /> > careful of that if you use it.<pdir="ltr">+1, happened a lot actually.
<p dir="ltr"><br /> On Nov 24, 2015 01:05, "Michael Paquier" <<a href="mailto:michael.paquier@gmail.com">michael.paquier@gmail.com</a>>wrote:<br /> ><br /> > On Tue, Nov 24, 2015at 3:41 AM, Kevin Grittner <<a href="mailto:kgrittn@gmail.com">kgrittn@gmail.com</a>> wrote:<br /> > > Yahoorecently changed their DMARC policy, and after some<br /> > > investigation and a support case with Yahoo, itis now clear that<br /> > > their email systems can no longer be used with the <a href="http://postgresql.org">postgresql.org</a><br/> > > lists. I've migrated from <a href="mailto:kgrittn@ymail.com">kgrittn@ymail.com</a>to <a href="mailto:kgrittn@gmail.com">kgrittn@gmail.com</a>.<br /> ><br/> > Something to be aware of as well: I noticed that sometimes your emails<br /> > coming from @<a href="http://ymail.com">ymail.com</a>were flagged as spam by gmail. People be<br /> > careful of that if you use it.<br/><p dir="ltr">That's a direct effect of the dmarc policy change. Yahoo no longer supports their customers using mailinglists. They changed their policies for such emails to hard reject, which makes Gmail (and presumably others) stickthem in spam.. It would happen to all the emails except the ones where you are on direct cc. <p dir="ltr">/Magnus <br/>
Magnus Hagander wrote: > That's a direct effect of the dmarc policy change. Yahoo no longer supports > their customers using mailing lists. They changed their policies for such > emails to hard reject, which makes Gmail (and presumably others) stick them > in spam.. It would happen to all the emails except the ones where you are > on direct cc. FWIW we've been rejecting posts coming from @yahoo.com addresses for a long time now, since DMARC was first introduced. We didn't get around to blocking other domains owned by Yahoo such as ymail.com or national yahoo subdomains, but I assume (without checking) that those will cause trouble too and we will have to block them out in order not to fill our queues with useless bounces. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Tue, Nov 24, 2015 at 12:58 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
Magnus Hagander wrote:
> That's a direct effect of the dmarc policy change. Yahoo no longer supports
> their customers using mailing lists. They changed their policies for such
> emails to hard reject, which makes Gmail (and presumably others) stick them
> in spam.. It would happen to all the emails except the ones where you are
> on direct cc.
FWIW we've been rejecting posts coming from @yahoo.com addresses for a
long time now, since DMARC was first introduced. We didn't get around
to blocking other domains owned by Yahoo such as ymail.com or national
yahoo subdomains, but I assume (without checking) that those will cause
trouble too and we will have to block them out in order not to fill our
queues with useless bounces.
Yes. The difference is they changed it from a soft fail to a hard reject, AIUI. From all of their domains (Kevin forwarded me the responses from Yahoo support). So the problem got worse, but it's the same basic one.
Alvaro Herrera <alvherre@2ndquadrant.com> writes: > Magnus Hagander wrote: >> That's a direct effect of the dmarc policy change. Yahoo no longer supports >> their customers using mailing lists. They changed their policies for such >> emails to hard reject, which makes Gmail (and presumably others) stick them >> in spam.. It would happen to all the emails except the ones where you are >> on direct cc. > FWIW we've been rejecting posts coming from @yahoo.com addresses for a > long time now, since DMARC was first introduced. We didn't get around > to blocking other domains owned by Yahoo such as ymail.com or national > yahoo subdomains, but I assume (without checking) that those will cause > trouble too and we will have to block them out in order not to fill our > queues with useless bounces. FWIW, my neighborhood's mailing list just recently implemented some changes that were supposed to allow the list to work again for Yahoo and other DMARC-affected users, after quite some time without service. I don't know how successful they were at that, nor how difficult the changes were ... but I do know the list server was offline for more than a day while the changes went in, so it was less than trivial. The only real change I can detect from looking at mail headers is that it seems the list may now be attaching its own DKIM-Signature header to emails that had one upon arrival. If anyone thinks we might be motivated to become DMARC compliant, I can inquire for more details. But I won't bother unless there's real interest. regards, tom lane
On Tue, Nov 24, 2015 at 4:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Magnus Hagander wrote:
>> That's a direct effect of the dmarc policy change. Yahoo no longer supports
>> their customers using mailing lists. They changed their policies for such
>> emails to hard reject, which makes Gmail (and presumably others) stick them
>> in spam.. It would happen to all the emails except the ones where you are
>> on direct cc.
> FWIW we've been rejecting posts coming from @yahoo.com addresses for a
> long time now, since DMARC was first introduced. We didn't get around
> to blocking other domains owned by Yahoo such as ymail.com or national
> yahoo subdomains, but I assume (without checking) that those will cause
> trouble too and we will have to block them out in order not to fill our
> queues with useless bounces.
FWIW, my neighborhood's mailing list just recently implemented some
changes that were supposed to allow the list to work again for Yahoo
and other DMARC-affected users, after quite some time without service.
I don't know how successful they were at that, nor how difficult the
changes were ... but I do know the list server was offline for more
than a day while the changes went in, so it was less than trivial.
The only real change I can detect from looking at mail headers is that
it seems the list may now be attaching its own DKIM-Signature header
to emails that had one upon arrival.
If anyone thinks we might be motivated to become DMARC compliant,
I can inquire for more details. But I won't bother unless there's
real interest.
I'd definitely be interested at least in what they're doing. Whether we'd actually implement it would depend on the implications of course, but if they've actually figured out how to do it, it could be useful.
We've discussed just forcibly stripping the DKIM headers of those emails, but that's unlikely to help - I'm sure large mail providers will "know" that yahoo mail is supposed to carry DKIM and thus fail. The whole point of DKIM is to prevent changing the headers after all - and we do change the headers.
Yahoo has a page on it (can't find the ref this moment) where they simply say "there is no mailinglist software supporting this. There are some experimental patches for an old version of mailman that people will perhaps consider merging at some time in the future."
Magnus Hagander <magnus@hagander.net> writes: > On Tue, Nov 24, 2015 at 4:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> If anyone thinks we might be motivated to become DMARC compliant, >> I can inquire for more details. But I won't bother unless there's >> real interest. > I'd definitely be interested at least in what they're doing. Whether we'd > actually implement it would depend on the implications of course, but if > they've actually figured out how to do it, it could be useful. OK, will ask. regards, tom lane
Magnus Hagander <magnus@hagander.net> writes: > On Tue, Nov 24, 2015 at 4:00 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> If anyone thinks we might be motivated to become DMARC compliant, >> I can inquire for more details. But I won't bother unless there's >> real interest. > I'd definitely be interested at least in what they're doing. Whether we'd > actually implement it would depend on the implications of course, but if > they've actually figured out how to do it, it could be useful. Forwarded with Rudy's permission ... regards, tom lane ------- Forwarded Message Date: Tue, 24 Nov 2015 10:34:45 -0500 From: "Rudolph T. Maceyko" <rm55@pobox.com> To: Tom Lane <tgl@sss.pgh.pa.us> Subject: Re: How did you fix HP list for DMARC compliance, exactly? Hi Tom, The basic changes since Yahoo implemented their p=reject DMARC policy last year (and others followed) were: * make NO CHANGES to the body of the message--no headers, footers, etc. * make NO CHANGES to the subject header of the message--no more "[Highland Park]" * when mail comes to the list from a domain that uses a p=reject DMARC policy, CHANGE THE FROM HEADER so that it comes from the list. Otherwise, when that message would be verified by any site that checks DMARC, it would fail (and probably would not be delivered, or would be considered spam). That last point was the big one, and is something that Mailman supports (in recent versions). It provides the option either to wrap the "offending" message in an attachment, or to do what we do now, which is to change the From header (and add a Reply-To, so replies still work). You could also elect to change *every* message that way, which seems like overkill (at least it does today). All of that is necessary in order to avoid DMARC problems. Of course, you CAN ban subscribers from these domains, but the list of p=reject DMARC policy sites will only grow. I read that Google is considering implementing it next year. Anyway, in addition to all of that, I've implemented DKIM verification/signing and our own DMARC policy (but NOT p=reject), as well as greylisting. These are just ways to elevate our anti-spam profile (and reputation). I've also set up feedback loops for AOL and Yahoo (and I'm trying for Comcast) so they don't ding us just because one of their subscribers "accidentally" marks a message that came through the list as spam. Running a mail server is hard these days... -Rudy On 2015-11-24 10:19, Tom Lane wrote: > I'm curious about what changes you made for this. And, of course: did > it work? > > I'm inquiring on behalf of an open-source project I'm involved in, > who might be interested in fixing their lists similarly: > http://www.postgresql.org/message-id/flat/CACjxUsPCjAFU81izZ0VcmK78EtEQ4_EjgCJK402WwwXvEZRhZA@mail.gmail.com [1] > > I don't believe they run the same list software you do, so exact > instructions probably aren't useful, but a functional spec for what > needs to happen would be very valuable. > > Thanks! > > regards, tom lane ------- End of Forwarded Message
I wrote: > "Rudolph T. Maceyko" <rm55@pobox.com> writes: >> The basic changes since Yahoo implemented their p=reject DMARC policy >> last year (and others followed) were: >> * make NO CHANGES to the body of the message--no headers, footers, etc. >> * make NO CHANGES to the subject header of the message--no more >> "[Highland Park]" >> * when mail comes to the list from a domain that uses a p=reject DMARC >> policy, CHANGE THE FROM HEADER so that it comes from the list. After further off-list discussion with Rudy, I'm not entirely convinced by his reasoning for dropping Subject-munging and footer-addition; it seems like that might be at least in part a limitation of his mailman-based infrastructure. The clearly critical thing, though, is that when forwarding a message from a person at a DMARC-using domain, we would have to replace the From: line with something @postgresql.org. This is what gets it out from under the original domain's DMARC policy. The other stuff Rudy did, including adding the list's own DKIM-Signatures and publishing DMARC and SPF policy for the list domain, is not technically necessary (yet) but it makes the list traffic less likely to get tagged as spam by antispam heuristics. And, as he noted, there are feedback loops that mean once some traffic gets tagged as spam it becomes more likely that future traffic will be. If Rudy's right that Gmail is likely to start using p=reject DMARC policy, we are going to have to do something about this before that; we have too many people on gmail. I'm not exactly in love with replacing From: headers but there may be little alternative. We could do something likeFrom: Persons Real Name <nobody@postgresql.org>Reply-To:... so that at least the person's name would still be readable in MUA displays. We'd have to figure out whether we want the Reply-To: to be the original author or the list; as I recall, neither of those are fully satisfactory. regards, tom lane
On Tue, Nov 24, 2015 at 11:10 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Forwarded with Rudy's permission ... > From: "Rudolph T. Maceyko" <rm55@pobox.com> > * when mail comes to the list from a domain that uses a p=reject DMARC > policy, CHANGE THE FROM HEADER so that it comes from the list. > Otherwise, when that message would be verified by any site that checks > DMARC, it would fail (and probably would not be delivered, or would be > considered spam). > change the From header (and add a Reply-To, so replies still work). If this were done, would the other steps (not changing the subject or body of the email) be necessary? -- Kevin Grittner EDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Kevin Grittner <kgrittn@gmail.com> writes: > On Tue, Nov 24, 2015 at 11:10 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>> change the From header (and add a Reply-To, so replies still work). > If this were done, would the other steps (not changing the subject > or body of the email) be necessary? See my followup: I think it's probably true that we could skip those changes. But Rudy commented that there's a lot of underdocumented subtlety here. There might be reasons I'm missing why we'd need to stop doing those things. It doesn't seem like DMARC as such would force that, but perhaps those things trigger some popular antispam heuristics. regards, tom lane
On 2015-11-24 13:11, Tom Lane wrote: > Kevin Grittner <kgrittn@gmail.com> writes: >> On Tue, Nov 24, 2015 at 11:10 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>>> change the From header (and add a Reply-To, so replies still work). > >> If this were done, would the other steps (not changing the subject >> or body of the email) be necessary? > > See my followup: I think it's probably true that we could skip those > changes. But Rudy commented that there's a lot of underdocumented > subtlety here. There might be reasons I'm missing why we'd need to > stop doing those things. It doesn't seem like DMARC as such would > force that, but perhaps those things trigger some popular antispam > heuristics. > > regards, tom lane Any Header or Body changes will invalidate most, if not all, DKIM signatures. Since DKIM is used as part of DMARC, it's a problem. Not sure what MajorDomo2 will allow you to do. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 7011 W Parmer Ln, Apt 1115, Austin, TX 78729-6961
Larry Rosenman wrote: > On 2015-11-24 13:11, Tom Lane wrote: > >Kevin Grittner <kgrittn@gmail.com> writes: > >>On Tue, Nov 24, 2015 at 11:10 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > >>>>change the From header (and add a Reply-To, so replies still work). > > > >>If this were done, would the other steps (not changing the subject > >>or body of the email) be necessary? > > > >See my followup: I think it's probably true that we could skip those > >changes. But Rudy commented that there's a lot of underdocumented > >subtlety here. There might be reasons I'm missing why we'd need to > >stop doing those things. It doesn't seem like DMARC as such would > >force that, but perhaps those things trigger some popular antispam > >heuristics. > Any Header or Body changes will invalidate most, if not all, DKIM > signatures. Since DKIM is used as part > of DMARC, it's a problem. > > Not sure what MajorDomo2 will allow you to do. We can turn off header and body changes easily enough, but changing the "From:" address requires patching the Majordomo2 source code; and, as you may already be aware, the maintainers gave up on Mj2 a decade ago, so we'd be on our own for that. I cannot promise any sort of timeline for getting that done; and since that's an essential part of the recipe, I don't see any point in doing the other changes either for the time being. I think the breakage of DKIM signatures is already causing some pain (though nowhere near the level of DMARC). Of course, removing all the "List-" headers *and* our custom footers is a huge step backwards in terms of mailing list functionality :-( Also, removing the [HACKERS] etc tags will annoy some people, for sure. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2015-11-24 13:43, Alvaro Herrera wrote: > Larry Rosenman wrote: >> On 2015-11-24 13:11, Tom Lane wrote: >> >Kevin Grittner <kgrittn@gmail.com> writes: >> >>On Tue, Nov 24, 2015 at 11:10 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> >>>>change the From header (and add a Reply-To, so replies still work). >> > >> >>If this were done, would the other steps (not changing the subject >> >>or body of the email) be necessary? >> > >> >See my followup: I think it's probably true that we could skip those >> >changes. But Rudy commented that there's a lot of underdocumented >> >subtlety here. There might be reasons I'm missing why we'd need to >> >stop doing those things. It doesn't seem like DMARC as such would >> >force that, but perhaps those things trigger some popular antispam >> >heuristics. > >> Any Header or Body changes will invalidate most, if not all, DKIM >> signatures. Since DKIM is used as part >> of DMARC, it's a problem. >> >> Not sure what MajorDomo2 will allow you to do. > > We can turn off header and body changes easily enough, but changing the > "From:" address requires patching the Majordomo2 source code; and, as > you may already be aware, the maintainers gave up on Mj2 a decade ago, > so we'd be on our own for that. I cannot promise any sort of timeline > for getting that done; and since that's an essential part of the > recipe, > I don't see any point in doing the other changes either for the time > being. > > I think the breakage of DKIM signatures is already causing some pain > (though nowhere near the level of DMARC). > > Of course, removing all the "List-" headers *and* our custom footers is > a huge step backwards in terms of mailing list functionality :-( Also, > removing the [HACKERS] etc tags will annoy some people, for sure. You don't have to remove the List- headers. DKIM says what headers it's using. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 7011 W Parmer Ln, Apt 1115, Austin, TX 78729-6961
Larry Rosenman <ler@lerctr.org> writes: > On 2015-11-24 13:11, Tom Lane wrote: >> Kevin Grittner <kgrittn@gmail.com> writes: >>> If this were done, would the other steps (not changing the subject >>> or body of the email) be necessary? >> >> See my followup: I think it's probably true that we could skip those >> changes. > Any Header or Body changes will invalidate most, if not all, DKIM > signatures. Since DKIM is used as part of DMARC, it's a problem. Yeah, but SPF is also used as part of DMARC, which means that merely forwarding somebody's email out of our listserv is probably going to look like spam, even if we didn't change anything at all about the message contents. Also, some sources sign Reply-To: and/or Sender: (Yahoo, at least, does the former) which means you can't replace those headers either without breaking the DKIM signature. The only fix for that is to rewrite From:, and once you do that I don't see a convincing argument why you can't also rewrite Subject: and add a footer if you feel like it. (For context, Rudy had implemented the no-header-change and no-footers changes on our neighborhood list quite some time ago; the From: changes and local DKIM-Signature: were the things that went in last week. I'm of the opinion that he could revert the former now that he's done the latter; but he seems to think not, for reasons he was not able to explain to me convincingly. It seems to me that the first two changes were meant to allow the source's DKIM-Signature to still apply, and now that he's given up on that strategy, I don't see what they're buying.) regards, tom lane
Larry Rosenman <ler@lerctr.org> writes: > On 2015-11-24 13:43, Alvaro Herrera wrote: >> Of course, removing all the "List-" headers *and* our custom footers is >> a huge step backwards in terms of mailing list functionality :-( Also, >> removing the [HACKERS] etc tags will annoy some people, for sure. > You don't have to remove the List- headers. DKIM says what headers it's > using. Yeah. RFC 6376 is worth a quick look if you want to opine knowledgeably about this. Basically, the DKIM crypto hash covers the message body plus those header fields enumerated in the DKIM-Signature header, and 6376 gives this advice: The From header field MUST be signed (that is, included in the "h=" tag of the resulting DKIM-Signature header field). Signers SHOULD NOT sign an existing header field likely to be legitimately modified or removed in transit. Inparticular, [RFC5321] explicitly permits modification or removal of the Return-Path header field in transit. SignersMAY include any other header fields present at the time of signing at the discretion of the Signer. INFORMATIVE OPERATIONS NOTE: The choice of which header fields to sign is non-obvious. One strategy is to signall existing, non- repeatable header fields. An alternative strategy is to sign only header fields that arelikely to be displayed to or otherwise be likely to affect the processing of the message at the receiver. A thirdstrategy is to sign only "well-known" headers. Note that Verifiers may treat unsigned header fields with extreme skepticism, including refusing to display them to the end user or even ignoring the signature if it does notcover certain header fields. For this reason, signing fields present in the message such as Date, Subject, Reply-To,Sender, and all MIME header fields are highly advised. I think the advice to sign Reply-To and Sender is rather ill-advised, particularly the latter, as signing that absolutely would break mailing list forwarding. regards, tom lane
On 11/24/2015 07:55 PM, Tom Lane wrote: > I wrote: >> "Rudolph T. Maceyko" <rm55@pobox.com> writes: >>> The basic changes since Yahoo implemented their p=reject DMARC policy >>> last year (and others followed) were: >>> * make NO CHANGES to the body of the message--no headers, footers, etc. >>> * make NO CHANGES to the subject header of the message--no more >>> "[Highland Park]" >>> * when mail comes to the list from a domain that uses a p=reject DMARC >>> policy, CHANGE THE FROM HEADER so that it comes from the list. > > After further off-list discussion with Rudy, I'm not entirely convinced > by his reasoning for dropping Subject-munging and footer-addition; it > seems like that might be at least in part a limitation of his > mailman-based infrastructure. > > The clearly critical thing, though, is that when forwarding a message from > a person at a DMARC-using domain, we would have to replace the From: line > with something @postgresql.org. This is what gets it out from under the > original domain's DMARC policy. exactly > > The other stuff Rudy did, including adding the list's own DKIM-Signatures > and publishing DMARC and SPF policy for the list domain, is not > technically necessary (yet) but it makes the list traffic less likely to > get tagged as spam by antispam heuristics. And, as he noted, there are > feedback loops that mean once some traffic gets tagged as spam it becomes > more likely that future traffic will be. well the purpose of the feedback loops is for the receiving ISP to feed back information about (mostly) user tagged "I dont want this email" (which is what the work on - not actual heuristics triggering) stuff - from historical experience that works very very poor for a mailing list like ours because in almost all cases subscribers are simply using the "this is spam" feature to declare an email as "unwanted" and using it as a shortcut to actually unsubscribing (and not thinking about any further impact). > > If Rudy's right that Gmail is likely to start using p=reject DMARC policy, > we are going to have to do something about this before that; we have too > many people on gmail. I'm not exactly in love with replacing From: > headers but there may be little alternative. We could do something like > From: Persons Real Name <nobody@postgresql.org> > Reply-To: ... > so that at least the person's name would still be readable in MUA > displays. > > We'd have to figure out whether we want the Reply-To: to be the original > author or the list; as I recall, neither of those are fully satisfactory. well this is basically what it boils down to - we will absolutely have to do is replacing "From:" (assuming the gmail rumour is true which I'm not entirely convinced though) but what are we prepared to replace the current system with and are we accepting that the lists are going to work differently. Stefan
On 11/24/2015 07:55 PM, Tom Lane wrote: > [snip] > The clearly critical thing, though, is that when forwarding a message from > a person at a DMARC-using domain, we would have to replace the From: line > with something @postgresql.org. This is what gets it out from under the > original domain's DMARC policy. One possibility that comes to mind: - Remove the sender's DMARC headers+signature **after thoroughly checking it** (to minimize the amount of UBE/UCE/junk going in) - Replace the sender's (i.e. 'From:' header) with list-sender+munched-email@postgresql.org (VERP-ified address) - Add the required headers, footers, change the subject line, etc - DKIM-sign the resulting message with postgresql.org's keys before sending it > [snip] > > If Rudy's right that Gmail is likely to start using p=reject DMARC policy, > we are going to have to do something about this before that; we have too > many people on gmail. I'm not exactly in love with replacing From: > headers but there may be little alternative. We could do something like > From: Persons Real Name <nobody@postgresql.org> > Reply-To: ... > so that at least the person's name would still be readable in MUA > displays. Yup > We'd have to figure out whether we want the Reply-To: to be the original > author or the list; as I recall, neither of those are fully satisfactory. Or just strip it, though that trump the sender's explicit preference (expressed by setting the header) I might be able to help a bit with implementation if needed. / J.L.
On Tue, Nov 24, 2015 at 10:03 PM, José Luis Tallón <jltallon@adv-solutions.net> wrote: >> From: Persons Real Name <nobody@postgresql.org> >> Reply-To: ... >> so that at least the person's name would still be readable in MUA >> displays. > > Yup It'll still mess up everyone's contact book which will fill up with these fake email addresses. And the Reply-To will mean private responses will go to the list. Fwiw I'm all for dropping the footer and the [HACKERS] which are both ill-advised imho. But modifying the From: header seems really broken. -- greg
Greg Stark <stark@mit.edu> writes: > It'll still mess up everyone's contact book which will fill up with > these fake email addresses. And the Reply-To will mean private > responses will go to the list. Yeah, it's not pretty. But I'm not sure we're gonna have much choice if Gmail changes their policy. > Fwiw I'm all for dropping the footer and the [HACKERS] which are both > ill-advised imho. But modifying the From: header seems really broken. IMO the footer is a *very* good idea; when we started using the current form of that, it greatly reduced the amount of "how do I unsubscribe" noise. But having said that, it probably wouldn't need to be on every message to be effective. I personally like the subject-munging but could live without it. [ thinks for a bit... ] I wonder whether we could do something like this: * Leave the From: and Reply-To: alone. * Add the footer only if the message isn't DKIM-signed. * Give up Subject-munging. (Munging only non-signed messages would be way too confusing.) I think that would put us in a situation where DKIM signatures would still pass, at least unless the source insisted on signing Sender: too. We might still have some issues with SPF checks, but not breaking DKIM would be a step forward. If things change to the point where only a small minority of messages get the footers because most people are using DKIM, then we might have to reconsider that part. But that seems far away yet. regards, tom lane
On Wed, Nov 25, 2015 at 2:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > I think that would put us in a situation where DKIM signatures would still > pass, at least unless the source insisted on signing Sender: too. Incidentally I'm confused about your concern about Sender. Sender has almost no significance for email afaik. It has specified behaviour for NNTP from whence it was copied but has no behaviour specified for mail. It's supposed to be the actual account that generated the email independent of how they want to appear. It's not clear how it's useful but if it's useful for anything it's reporting abuse to webmail providers so signing it seems sensible to me and I don't see any reason for list software to be concerned with it at all. Afaik MUAs have very little behaviour affected by them. Gmail displays it (which generally only confuses people). Yahoo uses it to authenticate mailing list management emails which prevents me from unsubscribing from lists there. Older versions of Exchange actually displayed it in preference to the From header whictih *really* confused people but afaik that's no longer the case. At least I haven't received any emails from Exchange users to my gmail account directly in a long time. -- greg
Greg Stark <stark@mit.edu> writes: > On Wed, Nov 25, 2015 at 2:16 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> I think that would put us in a situation where DKIM signatures would still >> pass, at least unless the source insisted on signing Sender: too. > Incidentally I'm confused about your concern about Sender. Sender has > almost no significance for email afaik. The PG lists do not think so; for example in -hackers traffic you will find Sender: pgsql-hackers-owner@postgresql.org which matches the envelope From address. Every other mailing list I'm on behaves similarly. Now, I'm not an email standards guru so I have no idea whether that's actually necessary or not, but I kinda doubt that you're right and all the mailing list software authors are wrong. regards, tom lane
It's a reasonable idea for mailing list software to put the list in Sender given that it's the "agent responsible for the actual transmission of the message" as RFC2822 specifies. But you could just as easily argue that the list is just relaying the message and the agent actually transmitting the message is the webmail provider the user used to compose the message. In a case like mine where I send it from gmail but use my .edu email address that's a sensible interpretation. But my point was that while the RFC says what to put there there's absolutely no reference anywhere for when the information should cause any MUA or MTA to behave differently. Responses are governed entirely by From and Reply-To, bounces don't use it, and so on. It's a bit of extra information that can optionally be added but there's no indication of how it should be used. Using it for anything aside from abuse investigations is going to pretty much always be wrong. RFC2822: The originator fields indicate the mailbox(es) of the source of the message. The "From:" field specifies the author(s)of the message, that is, the mailbox(es) of the person(s) or system(s) responsible for the writing of the message. The "Sender:" field specifies the mailbox of the agent responsible for the actual transmission of the message. For example, if a secretary were to send a message for another person, the mailbox of the secretary would appearin the "Sender:" field and the mailbox of the actual author would appear in the "From:" field. If the originatorof the message can be indicated by a single mailbox and the author and transmitter are identical, the "Sender:"field SHOULD NOT be used. Otherwise, both fields SHOULD appear. The originator fields also provide the information required when replying to a message. When the "Reply-To:" field ispresent, it indicates the mailbox(es) to which the author of the message suggests that replies be sent. In the absenceof the "Reply-To:" field, replies SHOULD by default be sent to the mailbox(es) specified in the "From:" field unlessotherwise specified by the person composing the reply.
Greg Stark <stark@mit.edu> writes: > It's a reasonable idea for mailing list software to put the list in > Sender given that it's the "agent responsible for the actual > transmission of the message" as RFC2822 specifies. Right. > But my point was that while the RFC says what to put there there's > absolutely no reference anywhere for when the information should cause > any MUA or MTA to behave differently. Agreed. To my mind that's a reason why Sender should not be DKIM-signed. Unfortunately, RFC 6376 explicitly suggests doing so ... and it looks like some people are taking that advice. I did a bit of grepping in my PG-lists inbox, and what I find is that about 19000 messages out of the 55000 I have saved since 2012 have DKIM-Signature lines (a lot more than I would have guessed, actually), and about 1400 of those list Sender as one of the lines to be signed. So if we override Sender we'll be breaking signatures on those. On the other hand, it looks like a nontrivial fraction of these things are broken anyway. I counted header field mentions in these lines: 1 DKIM-Signature <- explicitly forbidden by DKIM RFC 1 X-QQ-BUSINESS-ORIGIN 1 X-QQ-FEAT 1 X-QQ-MIME 1 X-QQ-Mailer 1 X-QQ-SENDSIZE 1 X-QQ-SSF 1 X-QQ-STYLE 1 X-QQ-WAPMAIL 1 X-QQ-mid 1 in-response-to 1 newsgroups 2 Content-Description 2 List-Archive <- guaranteed to break any mailing list 2 List-Help 2 List-Id 2 List-Owner 2 List-Post 2 List-Subscribe 2 List-Unsubscribe 2 Resent-Cc 2 Resent-Date 2 Resent-Message-ID 2 Resent-Sender 2 Resent-To 2 x-forwarded-message-id 3Resent-From 3 importance 4 Content-ID 5 X-Originating-IP 6 X-Yahoo-Newman-Id 7 thread-topic 8 Disposition-Notification-To 9 mail-followup-to 11 X-Yahoo-Newman-Property 11 X-Yahoo-SMTP 12 organization 13 x-enigmail-version 20 Thread-Index 26 x-mimeole 26 x-msmail-priority 27 X-Priority 51 accept-language 68 Content-Language 230 x-google-sender-auth 278 X-Rocket-MIMEInfo 300 X-YMail-OSG 401 X-Mailer 433 Received <- guaranteed to fail validation at recipient 546 x-received <- guaranteed to fail validation at recipient (there is no overlap in the above two groups) 598 content-disposition 1313 Reply-To 1432 User-Agent 1444 Sender 1466 x-sasl-enc 3159 Content-Transfer-Encoding 16405 CC 17530 In-Reply-To 17532 References 19070 MIME-Version19075 Message-ID 19090 Content-Type 19234 To 19419 Date 19635 Subject 19651 From So the number of people signing Sender is only marginally more than the number of people who are clueless enough to sign Received lines. Maybe we can write off the Sender signers as people who might get a clue eventually. I don't have an easy way to identify which sources are sending that, though. regards, tom lane
On 11/24/2015 11:03 PM, José Luis Tallón wrote: > On 11/24/2015 07:55 PM, Tom Lane wrote: >> [snip] >> The clearly critical thing, though, is that when forwarding a message >> from >> a person at a DMARC-using domain, we would have to replace the From: line >> with something @postgresql.org. This is what gets it out from under the >> original domain's DMARC policy. > > One possibility that comes to mind: > > - Remove the sender's DMARC headers+signature **after thoroughly > checking it** (to minimize the amount of UBE/UCE/junk going in) > - Replace the sender's (i.e. 'From:' header) with > list-sender+munched-email@postgresql.org (VERP-ified address) > > - Add the required headers, footers, change the subject line, etc > > - DKIM-sign the resulting message with postgresql.org's keys before > sending it that seems entirely doable with our current infrastructure (and even with minimal-to-no hackery on mj2) - but it still carries the "changes From:" issue :/ >> [snip] >> >> If Rudy's right that Gmail is likely to start using p=reject DMARC >> policy, >> we are going to have to do something about this before that; we have too >> many people on gmail. I'm not exactly in love with replacing From: >> headers but there may be little alternative. We could do something like >> From: Persons Real Name <nobody@postgresql.org> >> Reply-To: ... >> so that at least the person's name would still be readable in MUA >> displays. > Yup > >> We'd have to figure out whether we want the Reply-To: to be the original >> author or the list; as I recall, neither of those are fully satisfactory. > Or just strip it, though that trump the sender's explicit preference > (expressed by setting the header) > > > I might be able to help a bit with implementation if needed. the MTA side of things is fairly easy/straightforward(including using exim for some of the heavy lifting like doing the inbound dkim checking and "hinting" the downstream ML-boxes with the results) - however the main mailinglist infrastructure is still mj2 and that is aeons old perl - still interested in helping with implementation? ;) Stefan
Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes: > that seems entirely doable with our current infrastructure (and even > with minimal-to-no hackery on mj2) - but it still carries the "changes > From:" issue :/ Yeah. What do you think of the other approach of trying to preserve validity of the incoming DKIM-Signature (in particular, by not changing the Subject or message body)? regards, tom lane
On Wed, Nov 25, 2015 at 6:55 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > > But my point was that while the RFC says what to put there there's > > absolutely no reference anywhere for when the information should cause > > any MUA or MTA to behave differently. > > Agreed. To my mind that's a reason why Sender should not be DKIM-signed. > Unfortunately, RFC 6376 explicitly suggests doing so ... and it looks like > some people are taking that advice. Hm, I see it as a reason why signing Sender is reasonable. If it were a functional header then there might be a reason it would have to be changed. But if it's purely informational and the receiving MUA is going to display to the user (which is a bad idea imho but Gmail and Exchange both do it) then it makes sense to expect some authentication for it. I think the thinking is basically "sign everything we're going to present to the user phishers can't claim to be someone they're not". In which case it's fairly important that things like Sender be signed. Or that everyone agree it's just a useless header and stop sending or displaying it. I don't think we should base any action on guesses of what Gmail does. Google may do something we don't expect that's more complex to work around the problem. For one thing you can have email addresses at Google from a number of domains so they may well be able to have more than one policy for different users. I would suggest we stop doing things that are obviously incompatible with DKIM -- header and body munging for example. And I suspect we can stop touching Sender without any ill effects too. One idea might be to add a script to check a user's domain for p=reject and send them a warning when subscribing (or sending mail to the list?) warning them of the problem. -- greg
On Tue, Nov 24, 2015 at 7:50 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Yeah, but SPF is also used as part of DMARC, which means that merely > forwarding somebody's email out of our listserv is probably going to look > like spam, even if we didn't change anything at all about the message > contents. Also, some sources sign Reply-To: and/or Sender: (Yahoo, > at least, does the former) which means you can't replace those headers > either without breaking the DKIM signature. The only fix for that is to > rewrite From:, and once you do that I don't see a convincing argument why > you can't also rewrite Subject: and add a footer if you feel like it. From what I'm reading in DMARC if the DKIM signature is valid then the email should be accepted regardless of SPF. Many of the sources I'm reading say that the p=reject Yahoo change only affected lists that break DKIM signatures. If that's the case then I would say avoiding breaking DKIM is by far the preferable solution. Fwiw I think people are by far underestimating the breakage that rewriting From headers will produce. It breaks assumptions in lots of places. Just think, you can no longer search for or filter emails from someone reliably, autoresponders will now respond to the list, if you Cc the mail then which From header the recipient sees is random, and so on and so on. I can't imagine the mayhem that cross-posting will produce. It's just a terrible terrible idea. -- greg
<div class="moz-cite-prefix">On 11/26/2015 09:12 PM, Greg Stark wrote:<br /></div><blockquote cite="mid:CAM-w4HPNfQFAxYQhA=T7DJDoiyWEB+jwx1y2VCEvuvmk6dX7kA@mail.gmail.com"type="cite"><pre wrap="">On Wed, Nov 25, 2015at 6:55 PM, Tom Lane <a class="moz-txt-link-rfc2396E" href="mailto:tgl@sss.pgh.pa.us"><tgl@sss.pgh.pa.us></a> wrote: </pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">But my point was that while the RFC says what to put therethere's absolutely no reference anywhere for when the information should cause any MUA or MTA to behave differently. </pre></blockquote><pre wrap=""> Agreed. To my mind that's a reason why Sender should not be DKIM-signed. Unfortunately, RFC 6376 explicitly suggests doing so ... and it looks like some people are taking that advice. </pre></blockquote><pre wrap=""> Hm, I see it as a reason why signing Sender is reasonable. If it were a functional header then there might be a reason it would have to be changed. But if it's purely informational and the receiving MUA is going to display to the user (which is a bad idea imho but Gmail and Exchange both do it) then it makes sense to expect some authentication for it. I think the thinking is basically "sign everything we're going to present to the user phishers can't claim to be someone they're not". In which case it's fairly important that things like Sender be signed. Or that everyone agree it's just a useless header and stop sending or displaying it.</pre></blockquote><br /> From DMARC.org's Wiki:<br /><span class="comment"><span class="c00"><<<2 Add an "Original Authentication Results" header to indicate you have <br /> performed the authenticationand you are validating it <br /> 3 Take ownership of the email, by removing the DKIM signature and puttingyour own <br /> as well as changing the from header in the email to contain an email address <br /> within your mailinglist domain.</span></span> >>><br /><br /><br /> Read elsewhere: "To allow for forwarding scenarios, DMARCalso allows the <strong>Display From</strong> to be cryptographically signed by DKIM, and if any unauthorized spammeror phisher were to attempt to assume that identity, the encryption would fail."<br /><br /><blockquote cite="mid:CAM-w4HPNfQFAxYQhA=T7DJDoiyWEB+jwx1y2VCEvuvmk6dX7kA@mail.gmail.com"type="cite"><pre wrap="">I don't think we shouldbase any action on guesses of what Gmail does. Google may do something we don't expect that's more complex to work around the problem. For one thing you can have email addresses at Google from a number of domains so they may well be able to have more than one policy for different users.</pre></blockquote> Yep<br /><blockquote cite="mid:CAM-w4HPNfQFAxYQhA=T7DJDoiyWEB+jwx1y2VCEvuvmk6dX7kA@mail.gmail.com"type="cite"><pre wrap="">I would suggest westop doing things that are obviously incompatible with DKIM -- header and body munging for example. And I suspect we can stop touching Sender without any ill effects too. One idea might be to add a script to check a user's domain for p=reject and send them a warning when subscribing (or sending mail to the list?) warning them of the problem. </pre></blockquote> Definitively worth the effort, unless an almost perfect solution is found :S<br /><br /><br /> /J.L.<br /><br />
On Thu, Nov 26, 2015 at 11:13 PM, José Luis Tallón <jltallon@adv-solutions.net> wrote: > From DMARC.org's Wiki: > <<< 2 Add an "Original Authentication Results" header to indicate you have > performed the authentication and you are validating it > 3 Take ownership of the email, by removing the DKIM signature and putting > your own > as well as changing the from header in the email to contain an email address > within your mailing list domain. >>> But you missed option #1: 1. Operate strictly as a "forwarder," where the RFC5321.RcptTo field is changed to send the message to list members, but the RFC5322 message headers and body are not altered. Pros: Receiving systems can validate the DKIM signature of the message author, if one was present. Cons: Senders that depend solely on SPF for authentication will still fail. Precludes many customary features of mailing lists, such as "Subject:" tags, list footers/disclaimers, etc. -- greg
Greg Stark wrote: > Hm, I see it as a reason why signing Sender is reasonable. If it were > a functional header then there might be a reason it would have to be > changed. But if it's purely informational and the receiving MUA is > going to display to the user (which is a bad idea imho but Gmail and > Exchange both do it) then it makes sense to expect some authentication > for it. I think the thinking is basically "sign everything we're going > to present to the user phishers can't claim to be someone they're > not". In which case it's fairly important that things like Sender be > signed. Or that everyone agree it's just a useless header and stop > sending or displaying it. As the recipient of most -owner addresses, I would be glad to stop munging Sender. For some reason, some mailers record that as the address of the mailing list in the user's addressbook; so if in the future they send emails to the list, they end up in my mailbox instead of posted. > One idea might be to add a script to check a user's domain for > p=reject and send them a warning when subscribing (or sending mail to > the list?) warning them of the problem. I don't think that's going to be anything but unwelcome noise. What would they do if they became aware of the issue? They could switch providers, but that only works for so long. As soon as Gmail switches to p=reject, we've lost. We got away with doing it for Yahoo because there's not a lot of people using that -- not on these lists anyway. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 11/26/2015 09:10 PM, Tom Lane wrote: > Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> writes: >> that seems entirely doable with our current infrastructure (and even >> with minimal-to-no hackery on mj2) - but it still carries the "changes >> From:" issue :/ > > Yeah. What do you think of the other approach of trying to preserve > validity of the incoming DKIM-Signature (in particular, by not changing > the Subject or message body)? well not changing the subject seems like something we could do without fuss - not changing the body would likely mean we would (again) get a number of people asking "how do I unsubscribe", but maybe we will have to live with that. As for google/gmail - it seems they are indeed moving towards p=reject based on: https://dmarc.org/2015/10/global-mailbox-providers-deploying-dmarc-to-protect-users/ https://wordtothewise.com/2015/10/dmarc-news-gmail-preject-and-arc/ so we have to do "something" anyway (before June 2016) - I have not actually studied the referenced ietf drafts mentioned in the second post yet so maybe there is something in there to help with our usecase... Stefan
On Thu, Nov 26, 2015 at 11:26 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > I don't think that's going to be anything but unwelcome noise. What > would they do if they became aware of the issue? They could switch > providers, but that only works for so long. As soon as Gmail switches > to p=reject, we've lost. We got away with doing it for Yahoo because > there's not a lot of people using that -- not on these lists anyway. On further thought I think Gmail going p=reject is the wrong thing to worry about. The thing we need to check is how major mail providers like Gmail and Yahoo handle SPF failures *today*. There are plenty of domains we probably don't want to miss emails from that *already* have p=reject. For example if a Google employee mails us from @google.com [*] today that domain has p=reject so will everyone reading the list on Gmail or Yahoo miss the email? I bet other major companies have p=reject on their corporate domains as well. I'm hoping that as long as the DKIM signature succeeds those mails will still get through to Gmail, Yahoo, etc. There may be some people who miss it due to supporting SPF but not DKIM but if major providers fall into that camp then we're done for regardless of whether they have p=reject for their own domains. [*] This finally explains to me why there was a push to get employees to use an alternate domain for their free software mailing list posts instead of their @google.com address. -- greg
On Fri, Nov 27, 2015 at 10:36 AM, Greg Stark <stark@mit.edu> wrote:
-- On Thu, Nov 26, 2015 at 11:26 PM, Alvaro Herrera
<alvherre@2ndquadrant.com> wrote:
> I don't think that's going to be anything but unwelcome noise. What
> would they do if they became aware of the issue? They could switch
> providers, but that only works for so long. As soon as Gmail switches
> to p=reject, we've lost. We got away with doing it for Yahoo because
> there's not a lot of people using that -- not on these lists anyway.
On further thought I think Gmail going p=reject is the wrong thing to
worry about. The thing we need to check is how major mail providers
like Gmail and Yahoo handle SPF failures *today*. There are plenty of
domains we probably don't want to miss emails from that *already* have
p=reject. For example if a Google employee mails us from @google.com
[*] today that domain has p=reject so will everyone reading the list
on Gmail or Yahoo miss the email? I bet other major companies have
p=reject on their corporate domains as well.
Google doesn't actually reject, but it increases the likelyhood of it hitting spam significantly. However, they put a fairly low value on the SPF records. In my experience, it seems they put a much higher value on DKIM (failed or not).
Of course, Google also only actually *supports* email if both sender and receiver is on gmail. Anything else is "we hope it works". (Yes, I have official responses from google paid support saying they only support scenarios where both sender and receiver is on gmail)
On Fri, Nov 27, 2015 at 6:43 AM, Stefan Kaltenbrunner <stefan@kaltenbrunner.cc> wrote: > > well not changing the subject seems like something we could do without > fuss - not changing the body would likely mean we would (again) get a > number of people asking "how do I unsubscribe", but maybe we will have > to live with that. I've never thought the footer actually helped with that. We still get plenty of such emails anyways. On a hopeful note if the emails pass DKIM I wonder if our List-Unsubscribe link will start working. It should be making an "Unsubscribe" button appear in Gmail but doesn't currently seem to work. It's not clear if it will though since Gmail seems to think it's associated with the sender -- i.e. that all emails are spam. -- greg
On Fri, Nov 27, 2015 at 10:57 AM, Greg Stark <stark@mit.edu> wrote:
-- On Fri, Nov 27, 2015 at 6:43 AM, Stefan Kaltenbrunner
<stefan@kaltenbrunner.cc> wrote:
>
> well not changing the subject seems like something we could do without
> fuss - not changing the body would likely mean we would (again) get a
> number of people asking "how do I unsubscribe", but maybe we will have
> to live with that.
I've never thought the footer actually helped with that. We still get
plenty of such emails anyways.
We used to get more of them. Whether it's because of this or just because people are more used to it is a different question of course.
On a hopeful note if the emails pass DKIM I wonder if our
List-Unsubscribe link will start working. It should be making an
"Unsubscribe" button appear in Gmail but doesn't currently seem to
work. It's not clear if it will though since Gmail seems to think it's
associated with the sender -- i.e. that all emails are spam.
Do you have any examples of lists where it *does* work? LIke, could it be because our list-unsubscribe links are mailto: links and not http(s) links for example?
On Fri, Nov 27, 2015 at 10:02 AM, Magnus Hagander <magnus@hagander.net> wrote: > Do you have any examples of lists where it *does* work? LIke, could it be > because our list-unsubscribe links are mailto: links and not http(s) links > for example? From what I read they prefer mailto links. But apparently they only display them from high reputation sources -- a comment that only makes sense if you consider the list-unsubscribe as the source of the email which only really makes sense in the context of marketing emails. So I don't know if they'll work for actual lists. -- greg