Thread: mailing list redirect for bug numbers?

Hi,

How hard would it be to have a redirect similar to
https://www.postgresql.org/message-id/<id>

that accepted bug numbers instead of message ids?  I don't know the
precise database schema of the archives, but I assume it could be done
with a prefix query that filters the sender to @postgresql.org, the list
to pgsql-bugs, and the prefix to "BUG #<bugno>" or such.  Or perhaps
there's a database table with bugs -> messageid mappings somewhere? Or
could be created using a query like the above?

It'd be neat to link to bugs from commit messages in a clearer format
(i.e. to the bug number, rather than it being one of potentially
multiple message ids), and it also makes manual lookup nicer.

Greetings,

Andres Freund

Hi Andres,

On 1/14/19 5:18 PM, Andres Freund wrote:
> Hi,
>
> How hard would it be to have a redirect similar to
> https://www.postgresql.org/message-id/<id>
>
> that accepted bug numbers instead of message ids?  I don't know the
> precise database schema of the archives, but I assume it could be done
> with a prefix query that filters the sender to @postgresql.org, the list
> to pgsql-bugs, and the prefix to "BUG #<bugno>" or such.

The bug ID numbers are generated from pgweb using:

    SELECT nextval('bug_id_seq')

And then prefixed to the email thread as per the above.

More on this in a second.

> Or perhaps
> there's a database table with bugs -> messageid mappings somewhere? Or
> could be created using a query like the above?

IMV that would be an excellent suggestion. My guess is in order to make
that work, we would create the mapping when the initial bug report makes
it into the archives.

> It'd be neat to link to bugs from commit messages in a clearer format
> (i.e. to the bug number, rather than it being one of potentially
> multiple message ids), and it also makes manual lookup nicer.

Agreed, that sounds like a nicer UX.

The only big catch I see is that if someone emails -bugs directly, no
number is assigned, so we would have to leave that be.

I don't know if we would want to use "/message-id/" as the parent URL,
just in case someone sent a message with an ID of just digits (for
whatever reason). Dare I suggest something like "/bugs/<id>/?"

Assuming buy-in, what would need to be done is:

- Adjust the message import script to parse inbound messages with above
message beginning to -bugs. Determine if it is the first message to the
thread / bug ID is already registered. If it does not exist, record the
bug ID, message ID combo in a new table

- Write a one time script to map old bug id to first message id in the
thread.

- Update the urls.py in pgarchives to handle said route and fail
gracefully if bug ID does not exist

- Note in pgweb where the email is generated that any changes to email
subject could break things.

And that should be that.

Thoughts?

Jonathan

Hi,

On 2019-01-15 11:57:21 +0100, Magnus Hagander wrote:
> On Tue, Jan 15, 2019 at 11:43 AM Jonathan S. Katz <jkatz@postgresql.org>
> wrote:
> 
> > Hi Andres,
> >
> > On 1/14/19 5:18 PM, Andres Freund wrote:
> > > Hi,
> > >
> > > How hard would it be to have a redirect similar to
> > > https://www.postgresql.org/message-id/<id>
> > >
> > > that accepted bug numbers instead of message ids?  I don't know the
> > > precise database schema of the archives, but I assume it could be done
> > > with a prefix query that filters the sender to @postgresql.org, the list
> > > to pgsql-bugs, and the prefix to "BUG #<bugno>" or such.
> >
> > The bug ID numbers are generated from pgweb using:
> >
> >         SELECT nextval('bug_id_seq')
> >
> > And then prefixed to the email thread as per the above.
> >
> 
> Worth noticing is that it's also prefixed to the message-id, but that's a
> fairly new thing.

So we can't rely on that for reconstructing the historic mapping...


> > > It'd be neat to link to bugs from commit messages in a clearer format
> > > (i.e. to the bug number, rather than it being one of potentially
> > > multiple message ids), and it also makes manual lookup nicer.
> >
> > Agreed, that sounds like a nicer UX.
> >
> > The only big catch I see is that if someone emails -bugs directly, no
> > number is assigned, so we would have to leave that be.
> 
> Yeah, those would be entirely out of scope.

Agreed. I previously wished there were an email based interface to
submitting a bug, but that'd just be that, a new interface for
submitting a bug.

> I don't know if we would want to use "/message-id/" as the parent URL,
> > just in case someone sent a message with an ID of just digits (for
> > whatever reason). Dare I suggest something like "/bugs/<id>/?
> >
> 
> Or would we perhaps want tp use the postgr.es urls-shorterner with just a
> /b/ (like we have /m/ for messageids)?

Both? postgr.es/b/ redirects to the /bugs/ address, similar to /m/
redirecting to /message-id/?


> > - Adjust the message import script to parse inbound messages with above
> > message beginning to -bugs. Determine if it is the first message to the
> > thread / bug ID is already registered. If it does not exist, record the
> > bug ID, message ID combo in a new table
> >
> 
> That would suddenly put a very hard coded assumption into the archives code
> about this format, which I think is a bad idea in general since the
> archives code is not specific to the pgsql list usage *at all* today....

Couldn't the bug form instead just insert the message id it got from the
email it sent into the mapping database?  That ought to be pretty easy,
and would make the redirect dirt-cheap.  And reconstructing the historic
mapping shouldn't be more than one or two slow queries over the archive.

Greetings,

Andres Freund

Greetings,

* Andres Freund (andres@anarazel.de) wrote:
> On 2019-01-15 11:57:21 +0100, Magnus Hagander wrote:
> > On Tue, Jan 15, 2019 at 11:43 AM Jonathan S. Katz <jkatz@postgresql.org>
> > wrote:
> > > On 1/14/19 5:18 PM, Andres Freund wrote:
> > > > How hard would it be to have a redirect similar to
> > > > https://www.postgresql.org/message-id/<id>
> > > >
> > > > that accepted bug numbers instead of message ids?  I don't know the
> > > > precise database schema of the archives, but I assume it could be done
> > > > with a prefix query that filters the sender to @postgresql.org, the list
> > > > to pgsql-bugs, and the prefix to "BUG #<bugno>" or such.
> > >
> > > The bug ID numbers are generated from pgweb using:
> > >
> > >         SELECT nextval('bug_id_seq')
> > >
> > > And then prefixed to the email thread as per the above.
> >
> > Worth noticing is that it's also prefixed to the message-id, but that's a
> > fairly new thing.
>
> So we can't rely on that for reconstructing the historic mapping...

No, but the subject could be used, of course.

> > > > It'd be neat to link to bugs from commit messages in a clearer format
> > > > (i.e. to the bug number, rather than it being one of potentially
> > > > multiple message ids), and it also makes manual lookup nicer.
> > >
> > > Agreed, that sounds like a nicer UX.
> > >
> > > The only big catch I see is that if someone emails -bugs directly, no
> > > number is assigned, so we would have to leave that be.
> >
> > Yeah, those would be entirely out of scope.
>
> Agreed. I previously wished there were an email based interface to
> submitting a bug, but that'd just be that, a new interface for
> submitting a bug.

This got me wondering...  why?

This discussion is all about building a mapping between a Bug# and a
message-ID.  If we do that, why not fill it in completely?  As in, have
a Bug# for every thread on -bugs, either the one assigned by the webpage
or one which we assign when we see the email come in and then backfill
that?

We'd then need a way to display the complete mapping somewhere, but I
wouldn't think that would be terribly difficult to do..

> > I don't know if we would want to use "/message-id/" as the parent URL,
> > > just in case someone sent a message with an ID of just digits (for
> > > whatever reason). Dare I suggest something like "/bugs/<id>/?
> >
> > Or would we perhaps want tp use the postgr.es urls-shorterner with just a
> > /b/ (like we have /m/ for messageids)?
>
> Both? postgr.es/b/ redirects to the /bugs/ address, similar to /m/
> redirecting to /message-id/?

Seems like a reasonable idea to me.

> > > - Adjust the message import script to parse inbound messages with above
> > > message beginning to -bugs. Determine if it is the first message to the
> > > thread / bug ID is already registered. If it does not exist, record the
> > > bug ID, message ID combo in a new table
> >
> > That would suddenly put a very hard coded assumption into the archives code
> > about this format, which I think is a bad idea in general since the
> > archives code is not specific to the pgsql list usage *at all* today....
>
> Couldn't the bug form instead just insert the message id it got from the
> email it sent into the mapping database?  That ought to be pretty easy,
> and would make the redirect dirt-cheap.  And reconstructing the historic
> mapping shouldn't be more than one or two slow queries over the archive.

That seems like it'd work for messages using the form, if we're ok with
having the mapping live on wrigleys, which would probably be fine.

For the messages sent to -bugs that don't use the form, we could either
have a cronjob that runs regularly to assign bug #s, or we could hack up
the import code as contemplated above, or have emails to -bugs also go
to another system which just assigns the ID and then creates the
mapping.  Of course, there may be other ways, these are just some that
come to mind.

Thanks!

Stephen

Hi,

On 2019-01-16 13:38:55 -0500, Stephen Frost wrote:
> * Andres Freund (andres@anarazel.de) wrote:
> > On 2019-01-15 11:57:21 +0100, Magnus Hagander wrote:
> > > On Tue, Jan 15, 2019 at 11:43 AM Jonathan S. Katz <jkatz@postgresql.org>
> > > > The only big catch I see is that if someone emails -bugs directly, no
> > > > number is assigned, so we would have to leave that be.
> > > 
> > > Yeah, those would be entirely out of scope.
> > 
> > Agreed. I previously wished there were an email based interface to
> > submitting a bug, but that'd just be that, a new interface for
> > submitting a bug.
> 
> This got me wondering...  why?

Why I want it, or why I think it's out of scope?


> This discussion is all about building a mapping between a Bug# and a
> message-ID.  If we do that, why not fill it in completely?  As in, have
> a Bug# for every thread on -bugs, either the one assigned by the webpage
> or one which we assign when we see the email come in and then backfill
> that?

Well, one problem is that we can't offer the same subject based UI, at
least not trivially. We'd have to rewrite emails that aren't replies
to an existing thread to come from a different sender to have a
different subject and not fall afoul of dkim et al.

It seems pretty reasonable to do that, but not a small task, so I just
don't think it should be bundled with the, I think, smaller task of
just building the mapping. Doesn't seem like there'd be a lot of
wasted effort by doing these separately.


Greetings,

Andres Freund

Greetings,

* Andres Freund (andres@anarazel.de) wrote:
> On 2019-01-16 13:38:55 -0500, Stephen Frost wrote:
> > * Andres Freund (andres@anarazel.de) wrote:
> > > On 2019-01-15 11:57:21 +0100, Magnus Hagander wrote:
> > > > On Tue, Jan 15, 2019 at 11:43 AM Jonathan S. Katz <jkatz@postgresql.org>
> > > > > The only big catch I see is that if someone emails -bugs directly, no
> > > > > number is assigned, so we would have to leave that be.
> > > >
> > > > Yeah, those would be entirely out of scope.
> > >
> > > Agreed. I previously wished there were an email based interface to
> > > submitting a bug, but that'd just be that, a new interface for
> > > submitting a bug.
> >
> > This got me wondering...  why?
>
> Why I want it, or why I think it's out of scope?

"Why does it need to be a new interface?" is what I was going for there.

> > This discussion is all about building a mapping between a Bug# and a
> > message-ID.  If we do that, why not fill it in completely?  As in, have
> > a Bug# for every thread on -bugs, either the one assigned by the webpage
> > or one which we assign when we see the email come in and then backfill
> > that?
>
> Well, one problem is that we can't offer the same subject based UI, at
> least not trivially. We'd have to rewrite emails that aren't replies
> to an existing thread to come from a different sender to have a
> different subject and not fall afoul of dkim et al.

No, we can't rewrite the subject, and I wasn't suggesting that we do so.

> It seems pretty reasonable to do that, but not a small task, so I just
> don't think it should be bundled with the, I think, smaller task of
> just building the mapping. Doesn't seem like there'd be a lot of
> wasted effort by doing these separately.

I'm not against doing them independently, but I don't think the level of
effort required to simply fill in a bug number for any message-id we
find that doesn't match the subject-based search we'll be doing for the
bug # would be very much.  The bigger question that I'm trying to ask
here is for input on how useful it'd end up being or if people don't
think they'd actually use it.

As I recall, we actively discussed doing something similar for -hackers,
to shorten up the URLs going into commit messages but there was some
concern over having a mapping from IDs to message-IDs.  I'd have to go
dig up the thread to remember what the issue was there.

Thanks!

Stephen

On 2019-Jan-16, Stephen Frost wrote:

> As I recall, we actively discussed doing something similar for -hackers,
> to shorten up the URLs going into commit messages but there was some
> concern over having a mapping from IDs to message-IDs.  I'd have to go
> dig up the thread to remember what the issue was there.

The issue is that people working offline want to be able to figure out
emails in their local mailboxes looking just at the contents of a commit
message.  This is trivial if the URLs contain the message-id, and
impossible if they don't.

With bug numbers, the situation is the same: if, while offline, you have
a commit message carrying a bug number, and an offline mailbox where
pgsql-bugs threads are tagged with the same bug numbers, it's easy to
look up the thread based only on the contents of the commit message.  If
you have to contact a web interface to figure out what the thread is,
that workflow fails.

Based on this, my opinion is that a redirection system that handles
  https://postgr.es/bug/8470
by redirecting to message
  https://postgr.es/m/E1VOpEt-0003nc-4J@wrigleys.postgresql.org
works fine, because I can look up the thread by looking for the bug
number in the subject.  If we do anything more complicated than that
(say a database filled with message-ids based on bug IDs that don't
appear in the message subject), that workflow fails when I'm offline.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2019-01-16 18:54:40 -0300, Alvaro Herrera wrote:
> On 2019-Jan-16, Stephen Frost wrote:
> 
> > As I recall, we actively discussed doing something similar for -hackers,
> > to shorten up the URLs going into commit messages but there was some
> > concern over having a mapping from IDs to message-IDs.  I'd have to go
> > dig up the thread to remember what the issue was there.
> 
> The issue is that people working offline want to be able to figure out
> emails in their local mailboxes looking just at the contents of a commit
> message.  This is trivial if the URLs contain the message-id, and
> impossible if they don't.
> 
> With bug numbers, the situation is the same: if, while offline, you have
> a commit message carrying a bug number, and an offline mailbox where
> pgsql-bugs threads are tagged with the same bug numbers, it's easy to
> look up the thread based only on the contents of the commit message.  If
> you have to contact a web interface to figure out what the thread is,
> that workflow fails.
> 
> Based on this, my opinion is that a redirection system that handles
>   https://postgr.es/bug/8470
> by redirecting to message
>   https://postgr.es/m/E1VOpEt-0003nc-4J@wrigleys.postgresql.org
> works fine, because I can look up the thread by looking for the bug
> number in the subject.  If we do anything more complicated than that
> (say a database filled with message-ids based on bug IDs that don't
> appear in the message subject), that workflow fails when I'm offline.

Well said.

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> With bug numbers, the situation is the same: if, while offline, you have
> a commit message carrying a bug number, and an offline mailbox where
> pgsql-bugs threads are tagged with the same bug numbers, it's easy to
> look up the thread based only on the contents of the commit message.  If
> you have to contact a web interface to figure out what the thread is,
> that workflow fails.

Is it possible to add custom email headers in the pglister system,
something like maybe X-PostgreSQL-Bug, so that the bug ID number is
clearly assigned to emails?

Such a system might also be backwards compatible when backfilling bug
numbers to threads that don't have them yet. Local archives will need to
be synced again of course, but then it's easy to grep for the
X-PostgreSQL-Bug and find the email thread again, right?

Regards,
-- 
dim

Greetings,

* Magnus Hagander (magnus@hagander.net) wrote:
> On Thu, Jan 17, 2019 at 12:09 PM Dimitri Fontaine <dim@tapoueh.org> wrote:
>
> > Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> > > With bug numbers, the situation is the same: if, while offline, you have
> > > a commit message carrying a bug number, and an offline mailbox where
> > > pgsql-bugs threads are tagged with the same bug numbers, it's easy to
> > > look up the thread based only on the contents of the commit message.  If
> > > you have to contact a web interface to figure out what the thread is,
> > > that workflow fails.
> >
> > Is it possible to add custom email headers in the pglister system,
> > something like maybe X-PostgreSQL-Bug, so that the bug ID number is
> > clearly assigned to emails?

This was exactly what I was thinking too, to avoid the issue with the
Subject field.

> > Such a system might also be backwards compatible when backfilling bug
> > numbers to threads that don't have them yet. Local archives will need to
> > be synced again of course, but then it's easy to grep for the
> > X-PostgreSQL-Bug and find the email thread again, right?
>
> Doing that in pglister seems like a terrible idea. But if we want to, we
> could do it in the actual bug generation form, sure. That would be trivial.

Doing it in the bug generation form would only be half a solution
though.  Beyond the concern about pglister being too 'PG' specific,
what's the issue with having it able to add such headers..?

> But we can't do that backdated on existing mails. In the archives they're
> immutable. So they'd be for new emails only. So I'm not sure it would
> actually help very much?

We could certainly provide the mapping for old emails even if we don't
want to actually change the existing emails (although I'm not entirely
convinced it'd be such a bad idea to include the bug numbers somehow..),
and, really, we're talking about commits going forward, so is the issue
that old emails don't have it actually a problem?  New emails would and
the commit log moving forward is much more likely to reference new bugs
than old..

Thanks!

Stephen

Greetings,

* Magnus Hagander (magnus@hagander.net) wrote:
> On Thu, Jan 17, 2019 at 4:42 PM Stephen Frost <sfrost@snowman.net> wrote:
> > > Doing that in pglister seems like a terrible idea. But if we want to, we
> > > could do it in the actual bug generation form, sure. That would be
> > trivial.
> >
> > Doing it in the bug generation form would only be half a solution
> > though.  Beyond the concern about pglister being too 'PG' specific,
> > what's the issue with having it able to add such headers..?
>
> Sure, but I fail to see the *gain* with having it. If the contents of the
> header is based on what's already in the email, it doesn't add any new
> information. The bug number is *already* in the message, why copy it?

Uhhh, no, the point here was to assign bug numbers for emails to -bugs
which *don't* go through the bugs form and therefore didn't have the bug
number info in the message.

Doing it for the ones that *did* go through the form might be nice
because it'd add consistency as to where to find the bug number and that
could possibly even be done across replies that might have changed the
Subject line and removed the bug and such.

> > But we can't do that backdated on existing mails. In the archives they're
> > > immutable. So they'd be for new emails only. So I'm not sure it would
> > > actually help very much?
> >
> > We could certainly provide the mapping for old emails even if we don't
> > want to actually change the existing emails (although I'm not entirely
> > convinced it'd be such a bad idea to include the bug numbers somehow..),
> > and, really, we're talking about commits going forward, so is the issue
> > that old emails don't have it actually a problem?  New emails would and
> > the commit log moving forward is much more likely to reference new bugs
> > than old..
>
> Right. I'm not saying we shouldn't provide the mapping for old ones -- we
> definitely should. In fact I've gotten pretty far on the road of
> backfilling that with some tricky regepx (and yes, we have things like
> duplicate bugs with the same bug id and things in the archives -- the kind
> of stuff that happens when you don't actually store things in, say, a
> database).
>
> But that's unrelated to providing an additional custom header to an email
> that already contains that information.

For emails from the bugs form, having the bug number in a header
(instead of just the subject) seems like it could be independently
useful, but the discussion here was about providing a way for bug
numbers to be assigned based on just an inbound email- one that didn't
use the form.

Thanks!

Stephen

Magnus Hagander <magnus@hagander.net> writes:
> On Thu, Jan 17, 2019 at 7:28 PM Stephen Frost <sfrost@snowman.net> wrote:
>> Uhhh, no, the point here was to assign bug numbers for emails to -bugs
>> which *don't* go through the bugs form and therefore didn't have the bug
>> number info in the message.

> Without having the ability to properly merge bugs and to structured cross
> referencing and such, I think that's a really bad idea. That's going to
> cause more problems than it's fixing.

Agreed.  If we have bug numbers assigned to messages that aren't bugs,
or are replies to bugs, it's just going to be a mess.

            regards, tom lane

Greetings,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Magnus Hagander <magnus@hagander.net> writes:
> > On Thu, Jan 17, 2019 at 7:28 PM Stephen Frost <sfrost@snowman.net> wrote:
> >> Uhhh, no, the point here was to assign bug numbers for emails to -bugs
> >> which *don't* go through the bugs form and therefore didn't have the bug
> >> number info in the message.
>
> > Without having the ability to properly merge bugs and to structured cross
> > referencing and such, I think that's a really bad idea. That's going to
> > cause more problems than it's fixing.
>
> Agreed.  If we have bug numbers assigned to messages that aren't bugs,

We *already* have bug numbers regularly assigned to messages that aren't
bugs because they're non-bugs that come through the bugs form, and I
can't say that I see it being a serious issue, and there really isn't
anything we're going to be able to do to deal with it anyway, no matter
what we do.

> or are replies to bugs, it's just going to be a mess.

In my suggestion, replies to bugs that are sent by a sensible MUA would
have the bug # of the bug being replied to- in exactly the same way that
our archives have a mapping from message IDs to thread IDs internally
and that's how we get the threading in the archives.

My suggestion would just mean that a thread like this:

https://www.postgresql.org/message-id/flat/2047094.V130LYfLq4%40station53.ousa.org

would get a bug ID assigned to it, and that bug ID would show up in a
'X-Pg-BugId' field and you'd be able to have a link to the thread like
this:

https://postgr.es/b/123456

Another idea along this same vein would be to provide a way for the
thread IDs that we already assign to be available as a link and then
include that thread ID as an extra header.

Now, Magnus brought up a reasonable concern that making pgLister have to
figure out the threading would add a bunch of stuff into it that
currently lives in the archives instead (which is independent), and
that's a pretty good point, so, here's a different idea- we could just
assign an ID to every message that comes through pgLister from some
sequence (or maybe a sequence per mailing list..) and then provide a
mapping from that ID to the message in the archives.

In other words, we'd add a header like:

   X-PG-MessageId: 123453

And then be able to use links like:

https://postgr.es/p/pgsql-hackers/123453

This would work across lists, not require much from pglister, and be
able to be searched in an offline manner through local archives.

So, this seems to help with "we want a shorter URL to put into the
commit log that we can also search through offline email archives" but
it doesn't solve the "we want a bug #" (or, at least, it doesn't solve
it any more than the current system does).

I'm left wondering what the bug #'s use really is though.  We use it in
some places to refer to a particular report that came in, but we
regularly use message-IDs and URLs too because we have lots of things
that come in via -general or -hackers or directly to -bugs without going
through the bugs form.

Just to wrap this up, what I'm trying to get at is that I'd rather we
try to solve for the specific issue that came up rather than building a
solution on something that's already only a partial answer to begin
with, in that we often want to link from the commits to discussions on
-hackers or to emails to -bugs that didn't have a bug # and those aren't
addressed with this particular approach.

Thanks!

Stephen

Stephen Frost <sfrost@snowman.net> writes:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> Agreed.  If we have bug numbers assigned to messages that aren't bugs,
>> or are replies to bugs, it's just going to be a mess.

> In my suggestion, replies to bugs that are sent by a sensible MUA would
> have the bug # of the bug being replied to-

Uh, how?  Assuming that "your suggestion" refers to the 'X-Pg-BugId'
idea, I think the chances of that being included in replies are nil.

> In other words, we'd add a header like:
>    X-PG-MessageId: 123453
> And then be able to use links like:
> https://postgr.es/p/pgsql-hackers/123453

Seems like this is reinventing message-ids, and not very well either,
since copies received via a direct cc: rather than via the list would
lack the field.  (Hence, you're mistaken to claim this would be
locally searchable.)

> Just to wrap this up, what I'm trying to get at is that I'd rather we
> try to solve for the specific issue that came up rather than building a
> solution on something that's already only a partial answer to begin
> with, in that we often want to link from the commits to discussions on
> -hackers or to emails to -bugs that didn't have a bug # and those aren't
> addressed with this particular approach.

The existing solution is "use the message-id", and that seems to work
well enough.  Yes, gmail's message-ids are annoyingly long, but that
seems like only a cosmetic objection.  I'm not seeing anything here that
really looks like it'd be an improvement.

            regards, tom lane

Greetings,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > * Tom Lane (tgl@sss.pgh.pa.us) wrote:
> >> Agreed.  If we have bug numbers assigned to messages that aren't bugs,
> >> or are replies to bugs, it's just going to be a mess.
>
> > In my suggestion, replies to bugs that are sent by a sensible MUA would
> > have the bug # of the bug being replied to-
>
> Uh, how?  Assuming that "your suggestion" refers to the 'X-Pg-BugId'
> idea, I think the chances of that being included in replies are nil.

As I said, it would be done in the same way that our archives already
figure out threading and pglister could then add it.

If we want something that would work with replies for direct CC, then
we'd have to (ab)use a field that we know MUAs will copy to the next
email on a reply and there's unfortunately few of those.  At least
some bug trackers have dedicated email addresses for individual bugs,
probably for that reason, among others.

> > In other words, we'd add a header like:
> >    X-PG-MessageId: 123453
> > And then be able to use links like:
> > https://postgr.es/p/pgsql-hackers/123453
>
> Seems like this is reinventing message-ids, and not very well either,

What's being asked for certainly seemed like something that's very much
like a message-ID to me, but shorter so that it fits more conveniently
into commit messages and other places.  As such, it's not exactly
coincidence that this suggestion seems pretty similar to a message-id.

> since copies received via a direct cc: rather than via the list would
> lack the field.  (Hence, you're mistaken to claim this would be
> locally searchable.)

Ok, it's only locally searchable for people who are subscribed to our
mailing lists and keep local copies of the emails they get from our list
server, but isn't that going to cover the vast majority of the people
who are going to be looking in the commit log and trying to find a
matching thread in their local archive..?

> > Just to wrap this up, what I'm trying to get at is that I'd rather we
> > try to solve for the specific issue that came up rather than building a
> > solution on something that's already only a partial answer to begin
> > with, in that we often want to link from the commits to discussions on
> > -hackers or to emails to -bugs that didn't have a bug # and those aren't
> > addressed with this particular approach.
>
> The existing solution is "use the message-id", and that seems to work
> well enough.  Yes, gmail's message-ids are annoyingly long, but that
> seems like only a cosmetic objection.  I'm not seeing anything here that
> really looks like it'd be an improvement.

I'm not particularly against using message IDs, just to be clear, but
what started this was a complaint regarding them.

Going back to Andres' original ask on this subject, his justification
was:

On 1/14/19 5:18 PM, Andres Freund wrote:
> It'd be neat to link to bugs from commit messages in a clearer format
> (i.e. to the bug number, rather than it being one of potentially
> multiple message ids), and it also makes manual lookup nicer.

I took 'clearer format' to be implying 'shorter'.  The 'manual lookup'
comment would seem to imply, to me at least, that it'd be nicer for
cases where the commit message is "Fixed bug #12345", so there's two
sides to that- one is what goes in the commit message and the other is
how to look up what's in the commit message.

If we're just going to put 'Bug #X' in the commit message for cases
where there's a bug number, then we need to provide a better way to look
up what that bug is, and that's fine- but it doesn't help cases when
bugs get sent directly to the bugs list, which is unfortunate, and it
doesn't help at all for "Discussion" type links that go to threads on
-hackers.

I'm not sure that I get the concern around 'multiple message IDs' as any
of them really would work to get one to the thread and then either the
archives or your MUA could probably figure out the thread, though my
tendency would be to just link to the first message ID in the thread.

If we just agree to always use the message-ID of the first message in
the thread when we're closing out a bug, and people are happy with using
message-ID, then I'm not really sure what the bug # is doing for us (and
I see that that question from my prior email wasn't answered).

Thanks!

Stephen

Stephen Frost <sfrost@snowman.net> writes:
> If we just agree to always use the message-ID of the first message in
> the thread when we're closing out a bug, and people are happy with using
> message-ID, then I'm not really sure what the bug # is doing for us (and
> I see that that question from my prior email wasn't answered).

Yeah, I'm coming to the conclusion that there's nothing worth changing
here.  In an ideal world we'd have a bug number for every bug, but we'd
have to give up more than that's worth.

            regards, tom lane

On 2019-Jan-18, Stephen Frost wrote:

> > since copies received via a direct cc: rather than via the list would
> > lack the field.  (Hence, you're mistaken to claim this would be
> > locally searchable.)
> 
> Ok, it's only locally searchable for people who are subscribed to our
> mailing lists and keep local copies of the emails they get from our list
> server, but isn't that going to cover the vast majority of the people
> who are going to be looking in the commit log and trying to find a
> matching thread in their local archive..?

It doesn't work in that case either if you're CCed and you get the
direct copy before the list copy; that one won't have the header.
In my case I just deduplicate with formail/reformail and it simply
throws the second one away, and I suspect others do likewise.  I don't
know what does Gmail do.

> If we just agree to always use the message-ID of the first message in
> the thread when we're closing out a bug, and people are happy with using
> message-ID, then I'm not really sure what the bug # is doing for us (and
> I see that that question from my prior email wasn't answered).

Hmm, sometimes I refer to an intermediate message in a thread, not
necessarily the first, when something was discovered in the middle of
some discussion.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Greetings,

* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> Stephen Frost <sfrost@snowman.net> writes:
> > If we just agree to always use the message-ID of the first message in
> > the thread when we're closing out a bug, and people are happy with using
> > message-ID, then I'm not really sure what the bug # is doing for us (and
> > I see that that question from my prior email wasn't answered).
>
> Yeah, I'm coming to the conclusion that there's nothing worth changing
> here.  In an ideal world we'd have a bug number for every bug, but we'd
> have to give up more than that's worth.

I didn't come to the conclusion that we just should leave everything
as-is.  I was hoping to either get to a point where either we come up
with a sensible way to provide a bug number or other identifier for
things we typically want to link to from commits, or decide that the bug
number isn't really all that useful and get rid of it.

If we're going to keep it and just only use it some of the time and punt
on the rest then I guess the direction that Andres and Magnus were
heading in is alright and maybe better than nothing, but it does mean
we'll continue to recommend the web form to people even though it's
kinda awkward and annoying.

Thanks!

Stephen

Greetings,

* Magnus Hagander (magnus@hagander.net) wrote:
> On Fri, Jan 18, 2019 at 4:40 PM Stephen Frost <sfrost@snowman.net> wrote:
> > If we want something that would work with replies for direct CC, then
> > we'd have to (ab)use a field that we know MUAs will copy to the next
> > email on a reply and there's unfortunately few of those.  At least
> > some bug trackers have dedicated email addresses for individual bugs,
> > probably for that reason, among others.
>
> Technically we already have this for threads started in the form these
> days, as you can trivially get the bug number out of the message-id, and
> the message-id is one of the fields that *is* typically preserved (in the
> References header).

Hmmm, we certainly can't change the message-ID in-transit, but I wonder
about the References field and/or In-Reply-To...

> It won't cover threads that are responded to by creating a whole new thread
> and referencing the bug in the free text part of it, but there's really no
> way to handle that form inside the email...

Yeah, which also means it's difficult for someone who isn't on the list
and not on the original email to add to a bug...  That might be solved
if/when we implement the "email me this message" feature that we've
discussed a few times.

Thanks!

Stephen

Greetings,

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> On 2019-Jan-18, Stephen Frost wrote:
>
> > > since copies received via a direct cc: rather than via the list would
> > > lack the field.  (Hence, you're mistaken to claim this would be
> > > locally searchable.)
> >
> > Ok, it's only locally searchable for people who are subscribed to our
> > mailing lists and keep local copies of the emails they get from our list
> > server, but isn't that going to cover the vast majority of the people
> > who are going to be looking in the commit log and trying to find a
> > matching thread in their local archive..?
>
> It doesn't work in that case either if you're CCed and you get the
> direct copy before the list copy; that one won't have the header.
> In my case I just deduplicate with formail/reformail and it simply
> throws the second one away, and I suspect others do likewise.  I don't
> know what does Gmail do.

I used to do that too but then you may also miss headers from pglister
which direct where the email gets saved, at least for me, so I've gone
away from using a local de-dup system like that based on message ID
(independently, I have also seen idiot mail senders duplicate message
IDs for different messages, which has also made my shy of having
something that just throws away email based on that...).

> > If we just agree to always use the message-ID of the first message in
> > the thread when we're closing out a bug, and people are happy with using
> > message-ID, then I'm not really sure what the bug # is doing for us (and
> > I see that that question from my prior email wasn't answered).
>
> Hmm, sometimes I refer to an intermediate message in a thread, not
> necessarily the first, when something was discovered in the middle of
> some discussion.

That certainly is a perfectly acceptable option, imv, though I suppose
it does complicate things a bit since then you do have the multiple
message IDs question, but if what you really want to link to is
something down in a thread, even if it's a thread with a bug ID, I would
say that the issue is still there even if we add a way to link to bugs
by ID.

I suppose, in the end, committers will still need to be thinking about
what they put in the commit messages regarding links, but then, that's
also not necessairly a bad thing.

Thanks!

Stephen

On 2019-Jan-19, Stephen Frost wrote:

> I didn't come to the conclusion that we just should leave everything
> as-is.  I was hoping to either get to a point where either we come up
> with a sensible way to provide a bug number or other identifier for
> things we typically want to link to from commits, or decide that the bug
> number isn't really all that useful and get rid of it.

Well, we already have bug numbers in bugs submitted via the website and
they seem to work pretty well.

What if we add an email interface that creates bugs with IDs, but
nothing more than that?  I'm thinking a special address such as
pgsql-create-bug@postgresql.org that creates the bug ID and resends to
pgsql-bugs@postgresql.org CCing the bug reporter, after changing Subject
to include the bug ID and the From/Sender to an address under our
control (to avoid DKIM problems with the reporter's domain).  It would
work normally using pgsql-bugs from that point on.

If we have our system preserve References/In-Reply-To headers, I think
it would even work to add a CC the special address in the middle of a
regular thread in a mailing list (any mailing list, not just pgsql-bugs)
to spawn a new bug.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Greetings,

* Magnus Hagander (magnus@hagander.net) wrote:
> On Thu, Feb 7, 2019 at 1:26 AM Alvaro Herrera <alvherre@2ndquadrant.com>
> wrote:
> > On 2019-Jan-19, Stephen Frost wrote:
> > > I didn't come to the conclusion that we just should leave everything
> > > as-is.  I was hoping to either get to a point where either we come up
> > > with a sensible way to provide a bug number or other identifier for
> > > things we typically want to link to from commits, or decide that the bug
> > > number isn't really all that useful and get rid of it.
> >
> > Well, we already have bug numbers in bugs submitted via the website and
> > they seem to work pretty well.

This really depends on what the expectations are.  There's certainly a
lot of people who would say that our bug handling system is rather..
lacking.  I agree that we are doing a pretty good job just assigning a
bug number to bugs submitted via the web form, but that's a really low
bar and doesn't do things we want.

> > What if we add an email interface that creates bugs with IDs, but
> > nothing more than that?  I'm thinking a special address such as
> > pgsql-create-bug@postgresql.org that creates the bug ID and resends to
> > pgsql-bugs@postgresql.org CCing the bug reporter, after changing Subject
> > to include the bug ID and the From/Sender to an address under our
> > control (to avoid DKIM problems with the reporter's domain).  It would
> > work normally using pgsql-bugs from that point on.

Yes, this could possibly work, and would be an independent application
which wouldn't require hacking up pglister, which addresses one of the
concerns raised.

> > If we have our system preserve References/In-Reply-To headers, I think
> > it would even work to add a CC the special address in the middle of a
> > regular thread in a mailing list (any mailing list, not just pgsql-bugs)
> > to spawn a new bug.
>
> We could do something like that yes, and we've discussed doing it before.

Right.

> It would work as long as we have some sort of "moderation queue" on it,
> otherwise we'll be assigning and reposting a lot of spam.

One thing I really don't want to do is make the moderators of some list
like this the ones who have to make the decision about if a report is
really a bug or not.  For one thing, it's at least sometimes useful to
have the question asked and answered for the archives, whatever it is,
even if it isn't a bug.

> But let's not go jump at things left and right ATM. We already have another
> thread discussing a possible bugtracker (where a prototype has at some
> point been built and is shown). And we discussed yet another way (based off
> the cf app) at the fosdem devmeeting. So while we should certainly keep
> discussing things, let's not build something that does part of it just to
> end up with a mish mash of different things later.

Agreed on this, 100%.  This is largely why I chimed in on this thread to
begin with because it felt like we were headed down the 'mish mash'
route.

Thanks!

Stephen

Greetings,

* Magnus Hagander (magnus@hagander.net) wrote:
> On Thu, Feb 7, 2019 at 12:45 PM Stephen Frost <sfrost@snowman.net> wrote:
> > One thing I really don't want to do is make the moderators of some list
> > like this the ones who have to make the decision about if a report is
> > really a bug or not.  For one thing, it's at least sometimes useful to
> > have the question asked and answered for the archives, whatever it is,
> > even if it isn't a bug.
>
> Right, but that's a problem we *already* have.
>
> We will have to keep assigning those bug ids. The idea is we need to block
> pure spam. Just like moderators of the bug form and of the docs form have
> to do that *today*. Except it's likely going ot be more if it is a public
> email address.

Sure, I don't have any issue with having a new system go through
moderation in the same way that the webform submissions go through
moderation today, to just filter out pure spam.

Not sure why you think it'll be more spam than the amount we get to our
other lists, but it doesn't seem like that matters all that much either
way, it'll be whatever it ends up being.

> The reason being that once we resend it from *our* address, it will
> otherwise bypass all spam filters, and *our* addresses will start getting
> penalized for the spam. Given that for this to work we have to change the
> sender/from to be noreply@postgresql.org (or similar), like we do with docs
> and bugs today.

Right, we'd definitely need to have moderation in place to avoid having
our servers sending out spam and getting penalized for it (I don't know
that this is really all that different from the existing situation with
our lists, since it's our servers and most email providers care a lot
more about the server sending the spam than about what the From: address
in the spam is...).

Anyway, I think we agree here that we'd want this to be moderated, to
avoid having our systems send out spam, either way, so the rest is
likely moot.

Thanks!

Stephen