Thread: New archives for testing

After way too much time, in mainly due to long delays between doing
different parts, I think we're finally ready to do some public testing
of the new mailinglist archive site/code. The biggest technical change
is that with this code, the data is now stored in (I know, this is
shocking for a project like ours!) a database, which means we can do a
lot more with the presentation than we could before. The main
differences from before are:

* No more breaking threads at month boundaries. A thread can go on for
any amount of time, containing any amount of messages. Our current
record is 344 messages in a thread. Threading is not perfect, of
course, so there are bound to be some threads - particularly old ones
- that are broken into two. This becomes more visible now that we
don't arbitrarily break *all* threads, but it's fundamentally a
problem with the underlying data not containing all the information we
need.

* A thread is no longer bound to a list. A list is just a "tag" on a
thread, meaning a single thread can be in multiple lists. As soon as a
message is CCed to multiple lists, the whole thread will show up on
them, meaning it's now possible to follow a discussion when it's
moved.

* It's now possible to navigate the whole thread from inside a message
view, using a dropdown listing the whole thing. It's still possible to
navigate with to the next and previous in thread at the bottom of the
message, just like before.

* There's also a "flat view" that shows an entire message thread on
one page, for those who prefer that kind of view. (It works pretty
well in most cases, but becomes a huge page for example on the 344
message thread)

* There is no longer a per-month, per-list sequence number for each
message, that causes havoc if accidentally reset (as has happened a
couple of times in the mhonarc history, when the mbox files have been
edited - intenitionally or not). Instead, the messageid url is the
primary URL for all pages, and it's permanent.

* For SEO reasons mainly, the new archives live under the main website
URL space, starting point is http://www.postgresql.org/list/

* Raw messages and mbox files are now protected with http basic
authentication and a fixed password. In the old archives, the software
worked hard to obfuscate email addresses in the headers and such of
the visible email, but they were all fully visible if you clicked
"raw" or downloaded the mbox, making the antispam measures kind of
pointless. The username and password for the protection is listed in
the password prompt, so it shouldn't be a problem for any manual
navigation, but hopefully enough to keep bots out.

* Contents are no longer updated by a cronjob running every 15
minutes. Instead, email is injected directly into the archives as soon
as it leaves majordomo on the list server, and becomes available
within second(s). It still needs to pass antispam, enter majordomo and
possibly be moderated, so it doesn't mean a second after someone hit
"send" in their MUA, but it should be significantly faster than
before.


Now, having said that, I'd like to see some more people testing it
than the few people I've forced to do it so far. The way to test it is
to go to http://www.postgresql.org/list/ and pick your list. You can
also go to http://www.postgresql.org/message-id/<message-id> to view a
message in a thread - that should also work fine if you just replace
"archives" with "www" in the URL of an existing message in the
archives *if* you were using the message-id based url.

A few things that are known not to work at this point:
* Searching by message-id will not work. This is because the new
search code hasn't been deployed, since it would replace the old one
and can't be launched side by side.

* This also means that searching in general will return search hits on
the old archives site, not the new one, even when searching from the
new one.

* There is no redirect from the old archives. There is code available
so that once we consider it production, all old
archives.postgresql.org URLs will redirect to the new site - directly
to the proper message. But this is not deployed yet, as this is a
partial deploy.



So. Please go test it. And give some feedback.

While there is a lot that can be done to improve the experience, and
we can discuss that endlessly, I would like to ask people for now to
focus on things that *don't work*, or things that are *worse than
before*. Once we've tried to deal with those as well as we can, we can
switch over. Then we can improve things further. But let's not get
bogged down trying to add every new feature now - let's just get it
far enough to make it better than before as a first step.


--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On Fri, Dec 28, 2012 at 03:32:21PM +0100, Magnus Hagander wrote:
> So. Please go test it. And give some feedback.
> 
> While there is a lot that can be done to improve the experience, and
> we can discuss that endlessly, I would like to ask people for now to
> focus on things that *don't work*, or things that are *worse than
> before*. Once we've tried to deal with those as well as we can, we can
> switch over. Then we can improve things further. But let's not get
> bogged down trying to add every new feature now - let's just get it
> far enough to make it better than before as a first step.

I am confused about how to read a thread.  If I an on this email
message:
http://www.postgresql.org/message-id/CAHGQGwFkhcwJbMgqmGQkUDoHOLxtC--9XpZXQ_t8669YFpRP-w@mail.gmail.com

how do I read the next email in the thread.  Is it the two "Responses"? 
The old system would just have a "next in thread" option and could just
keeping hitting "next in thread" to see all the thread emails.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + It's impossible for everything to be true. +

On Sat, Dec 29, 2012 at 12:22 AM, Bruce Momjian <bruce@momjian.us> wrote:
> On Fri, Dec 28, 2012 at 03:32:21PM +0100, Magnus Hagander wrote:
>> So. Please go test it. And give some feedback.
>>
>> While there is a lot that can be done to improve the experience, and
>> we can discuss that endlessly, I would like to ask people for now to
>> focus on things that *don't work*, or things that are *worse than
>> before*. Once we've tried to deal with those as well as we can, we can
>> switch over. Then we can improve things further. But let's not get
>> bogged down trying to add every new feature now - let's just get it
>> far enough to make it better than before as a first step.
>
> I am confused about how to read a thread.  If I an on this email
> message:
>
>         http://www.postgresql.org/message-id/CAHGQGwFkhcwJbMgqmGQkUDoHOLxtC--9XpZXQ_t8669YFpRP-w@mail.gmail.com
>
> how do I read the next email in the thread.  Is it the two "Responses"?

Yes, "responses" corresponds to what was previously "follow ups".

You can also navigate the thread using the thread dropdown in the
message header.

> The old system would just have a "next in thread" option and could just
> keeping hitting "next in thread" to see all the thread emails.

If you keep hitting next in thread, I don't think you get that though
- if the thread splits out into two, don't you just end up walking
down one branch?

If you want to navigate the whole thread at once, using the "flat
view" is probably more efficient - it gives you a lot fewer clicks.
Try clicking the "flat" link in the message header

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On Sat, Dec 29, 2012 at 10:53:49AM +0100, Magnus Hagander wrote:
> You can also navigate the thread using the thread dropdown in the
> message header.
> 
> > The old system would just have a "next in thread" option and could just
> > keeping hitting "next in thread" to see all the thread emails.
> 
> If you keep hitting next in thread, I don't think you get that though
> - if the thread splits out into two, don't you just end up walking
> down one branch?
> 
> If you want to navigate the whole thread at once, using the "flat
> view" is probably more efficient - it gives you a lot fewer clicks.
> Try clicking the "flat" link in the message header

Ah, I like the flat view.  Good.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + It's impossible for everything to be true. +

Magnus Hagander <magnus@hagander.net> writes:
> So. Please go test it. And give some feedback.

I like it better than the old one. The thread navigation feels strange
as a drop down menu. The attachement downloading works better than
before, as it keeps the file name.

Congrats, and +1 for going live with the current version!

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

On Fri, Dec 28, 2012 at 2:32 PM, Magnus Hagander <magnus@hagander.net> wrote:
>
> * Raw messages and mbox files are now protected with http basic
> authentication and a fixed password. In the old archives, the software
> worked hard to obfuscate email addresses in the headers and such of
> the visible email, but they were all fully visible if you clicked
> "raw" or downloaded the mbox, making the antispam measures kind of
> pointless. The username and password for the protection is listed in
> the password prompt, so it shouldn't be a problem for any manual
> navigation, but hopefully enough to keep bots out.

The prompt isn't shown on all browsers, so we should stick it on the
website somewhere too.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Sun, Dec 30, 2012 at 7:28 PM, Dave Page <dpage@pgadmin.org> wrote:
> On Fri, Dec 28, 2012 at 2:32 PM, Magnus Hagander <magnus@hagander.net> wrote:
>>
>> * Raw messages and mbox files are now protected with http basic
>> authentication and a fixed password. In the old archives, the software
>> worked hard to obfuscate email addresses in the headers and such of
>> the visible email, but they were all fully visible if you clicked
>> "raw" or downloaded the mbox, making the antispam measures kind of
>> pointless. The username and password for the protection is listed in
>> the password prompt, so it shouldn't be a problem for any manual
>> navigation, but hopefully enough to keep bots out.
>
> The prompt isn't shown on all browsers, so we should stick it on the
> website somewhere too.

Ugh, that's annoying.

Do you think we can get away without putting it next to every single
link? Because I'm not sure how we can do that without making it look
like crap. But if we don't, are people likely to ever read it?

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus@hagander.net> writes:
> On Sun, Dec 30, 2012 at 7:28 PM, Dave Page <dpage@pgadmin.org> wrote:
>> The prompt isn't shown on all browsers, so we should stick it on the
>> website somewhere too.

> Ugh, that's annoying.

> Do you think we can get away without putting it next to every single
> link? Because I'm not sure how we can do that without making it look
> like crap. But if we don't, are people likely to ever read it?

How big a problem is this?  When I checked, both Safari and Firefox
showed the prompt.  If there are just a few little-used browsers that
fail to show it, I'm not convinced that we have to clutter the pages
for everybody to cater to them.  I can think of more than a few other
sites where that prompt is pretty damn essential for usability, so
I would argue that a browser that doesn't show it is broken anyhow.

A bigger issue is that with both Safari and Firefox, the first login
attempt failed and I had to try again.  Now maybe I fat-fingered the
password both times, but I suspect there is something wrong there.
        regards, tom lane

Magnus Hagander <magnus@hagander.net> writes:
> On Sun, Dec 30, 2012 at 7:28 PM, Dave Page <dpage@pgadmin.org> wrote:
>> The prompt isn't shown on all browsers, so we should stick it on the
>> website somewhere too.

> Ugh, that's annoying.

> Do you think we can get away without putting it next to every single
> link? Because I'm not sure how we can do that without making it look
> like crap. But if we don't, are people likely to ever read it?

How big a problem is this?  When I checked, both Safari and Firefox
showed the prompt.  If there are just a few little-used browsers that
fail to show it, I'm not convinced that we have to clutter the pages
for everybody to cater to them.  I can think of more than a few other
sites where that prompt is pretty damn essential for usability, so
I would argue that a browser that doesn't show it is broken anyhow.

On Sun, Dec 30, 2012 at 9:53 PM, Dave Page <dpage@pgadmin.org> wrote:
>
> On Sunday, December 30, 2012, Tom Lane wrote:
>>
>> Magnus Hagander <magnus@hagander.net> writes:
>> > On Sun, Dec 30, 2012 at 7:28 PM, Dave Page <dpage@pgadmin.org> wrote:
>> >> The prompt isn't shown on all browsers, so we should stick it on the
>> >> website somewhere too.
>>
>> > Ugh, that's annoying.
>>
>> > Do you think we can get away without putting it next to every single
>> > link? Because I'm not sure how we can do that without making it look
>> > like crap. But if we don't, are people likely to ever read it?
>>
>> How big a problem is this?  When I checked, both Safari and Firefox
>> showed the prompt.  If there are just a few little-used browsers that
>> fail to show it, I'm not convinced that we have to clutter the pages
>> for everybody to cater to them.  I can think of more than a few other
>> sites where that prompt is pretty damn essential for usability, so
>> I would argue that a browser that doesn't show it is broken anyhow.
>
>
> I don't think it was originally intended as a prompt (it's the security
> realm actually), but most browsers showed it anyway and it's been (ab)used
> that way for years. FYI, the browser I saw not displaying it was Safari on
> iOS, so most definitely not 'little used'.

No, but not showing it makes it a pretty useless browser since it's
supposed to tell the user which password to use when different
sections on a site has different passwords.

That said, it doesn't matter how stupid or useless it is, if it's
reality :) We just have to deal with it. I'm not too worried about
iphone users - i doubt either the raw or the mbox view is very
interesting to them. Same for mbox users on iPad - however, I can
certainly see iPad users who want to get the raw view.

Right now, mobile is about 1.2% of our visitors to the archives.
Safari on ios 0.5%. iPhone is just over 0.3% and iPad just over 0.3%.

So it's a very small portion of our visitors. While the total numbers
are likely going up, I'm not sure those who actually want the raw
and/or mbox files are going to go up.

FWIW, it works fine in Chrome (46%), Firefox (36%) and Safari-desktop
(3%). Unverified at this point are IE (11%) and Opera(2%), the rest
are really far down the list.

So the question is how much effort we want to put into it. If we make
the 401 page itself contain the text, does that show up in safari
after authentication has failed, or does it show some custom page?

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On Sun, Dec 30, 2012 at 10:32 PM, Magnus Hagander <magnus@hagander.net> wrote:
> On Sun, Dec 30, 2012 at 9:53 PM, Dave Page <dpage@pgadmin.org> wrote:
>>
>> On Sunday, December 30, 2012, Tom Lane wrote:
>>>
>>> Magnus Hagander <magnus@hagander.net> writes:
>>> > On Sun, Dec 30, 2012 at 7:28 PM, Dave Page <dpage@pgadmin.org> wrote:
>>> >> The prompt isn't shown on all browsers, so we should stick it on the
>>> >> website somewhere too.
>>>
>>> > Ugh, that's annoying.
>>>
>>> > Do you think we can get away without putting it next to every single
>>> > link? Because I'm not sure how we can do that without making it look
>>> > like crap. But if we don't, are people likely to ever read it?
>>>
>>> How big a problem is this?  When I checked, both Safari and Firefox
>>> showed the prompt.  If there are just a few little-used browsers that
>>> fail to show it, I'm not convinced that we have to clutter the pages
>>> for everybody to cater to them.  I can think of more than a few other
>>> sites where that prompt is pretty damn essential for usability, so
>>> I would argue that a browser that doesn't show it is broken anyhow.
>>
>>
>> I don't think it was originally intended as a prompt (it's the security
>> realm actually), but most browsers showed it anyway and it's been (ab)used
>> that way for years. FYI, the browser I saw not displaying it was Safari on
>> iOS, so most definitely not 'little used'.
>
> No, but not showing it makes it a pretty useless browser since it's
> supposed to tell the user which password to use when different
> sections on a site has different passwords.
>
> That said, it doesn't matter how stupid or useless it is, if it's
> reality :) We just have to deal with it. I'm not too worried about
> iphone users - i doubt either the raw or the mbox view is very
> interesting to them. Same for mbox users on iPad - however, I can
> certainly see iPad users who want to get the raw view.
>
> Right now, mobile is about 1.2% of our visitors to the archives.
> Safari on ios 0.5%. iPhone is just over 0.3% and iPad just over 0.3%.
>
> So it's a very small portion of our visitors. While the total numbers
> are likely going up, I'm not sure those who actually want the raw
> and/or mbox files are going to go up.
>
> FWIW, it works fine in Chrome (46%), Firefox (36%) and Safari-desktop
> (3%). Unverified at this point are IE (11%) and Opera(2%), the rest
> are really far down the list.
>
> So the question is how much effort we want to put into it. If we make
> the 401 page itself contain the text, does that show up in safari
> after authentication has failed, or does it show some custom page?

ads just confirmed it works on IE as well.

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On 30 December 2012 21:36, Magnus Hagander <magnus@hagander.net> wrote:
> ads just confirmed it works on IE as well.

Have you tried using a service to do this for you?


https://browsershots.org/http://www.postgresql.org/message-id/CAHGQGwFkhcwJbMgqmGQkUDoHOLxtC--9XpZXQ_t8669YFpRP-w@mail.gmail.com

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

On Sun, Dec 30, 2012 at 10:43 PM, Peter Geoghegan <peter@2ndquadrant.com> wrote:
> On 30 December 2012 21:36, Magnus Hagander <magnus@hagander.net> wrote:
>> ads just confirmed it works on IE as well.
>
> Have you tried using a service to do this for you?
>
>
https://browsershots.org/http://www.postgresql.org/message-id/CAHGQGwFkhcwJbMgqmGQkUDoHOLxtC--9XpZXQ_t8669YFpRP-w@mail.gmail.com

The point is what happens when you go to the raw url. It won't me
submit any new ones, but I somehow have a feeling it won't work with
the password protected URL...

And also, it doesn't include the mobile browsers anyway, which seem to
be the only place this one has issues.

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus@hagander.net> writes:
> On Sun, Dec 30, 2012 at 9:53 PM, Dave Page <dpage@pgadmin.org> wrote:
>> I don't think it was originally intended as a prompt (it's the security
>> realm actually), but most browsers showed it anyway and it's been (ab)used
>> that way for years. FYI, the browser I saw not displaying it was Safari on
>> iOS, so most definitely not 'little used'.

> No, but not showing it makes it a pretty useless browser since it's
> supposed to tell the user which password to use when different
> sections on a site has different passwords.
> ...
> So the question is how much effort we want to put into it. If we make
> the 401 page itself contain the text, does that show up in safari
> after authentication has failed, or does it show some custom page?

At least on iOS 6, Safari doesn't seem to show any 401 page at all.
When you hit the "raw" link, you get an "Authentication required"
popup with just space for username and password.  If you put in
a wrong value, the popup re-appears.  There's not much you can
do except hit "Cancel".  Not very helpful at all I'd say.  (Now
admittedly, on a phone-size screen it's not clear that there's
room for much of a prompt, but still...)

Having just done the experiment, though, I'd have to say that the
usability of the archives is pretty darn low regardless of this.
Too many very small links too close together --- there's basically
no way to hit what you want accurately without zooming way in first.
(And that was on an iPad; don't even want to think about a phone.)
I can't see anybody really caring about either the mbox or raw links
in that context.

But on the third hand ... could we rig it to accept any old name and
password?  The mere occurrence of a challenge ought to be enough to
discourage most bots.
        regards, tom lane

On Mon, Dec 31, 2012 at 11:47 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Magnus Hagander <magnus@hagander.net> writes:
>> On Sun, Dec 30, 2012 at 9:53 PM, Dave Page <dpage@pgadmin.org> wrote:
>>> I don't think it was originally intended as a prompt (it's the security
>>> realm actually), but most browsers showed it anyway and it's been (ab)used
>>> that way for years. FYI, the browser I saw not displaying it was Safari on
>>> iOS, so most definitely not 'little used'.
>
>> No, but not showing it makes it a pretty useless browser since it's
>> supposed to tell the user which password to use when different
>> sections on a site has different passwords.
>> ...
>> So the question is how much effort we want to put into it. If we make
>> the 401 page itself contain the text, does that show up in safari
>> after authentication has failed, or does it show some custom page?
>
> At least on iOS 6, Safari doesn't seem to show any 401 page at all.
> When you hit the "raw" link, you get an "Authentication required"
> popup with just space for username and password.  If you put in
> a wrong value, the popup re-appears.  There's not much you can
> do except hit "Cancel".  Not very helpful at all I'd say.  (Now
> admittedly, on a phone-size screen it's not clear that there's
> room for much of a prompt, but still...)

Well, the page usually shows up once you hit cancel. It's not very
user friendly, but that page is at least in theory customizable. But I
think a lot of browsers don't show it.

There is plenty of room on the phone screen to do a prompt. At least
android has no problem at all with it. But that doesn't really matter
if a platform that's half of our mobile visitors can't handle it -
because we can't change that. Unless we want to take the same approach
as we do with some of the windows code, which is say "it's good
enough, if people want the better functionality they should pick a
more suitable platform". (which for the access of raw or mbox isn't
entirely unreasonable, really..)


> Having just done the experiment, though, I'd have to say that the
> usability of the archives is pretty darn low regardless of this.
> Too many very small links too close together --- there's basically
> no way to hit what you want accurately without zooming way in first.

I guess I'm spoiled by a browser that auto-zooms just the links when
you accidentally click next to another one, making that a non-issue.
But probably more useful, we could do with a mobile adapted version
*period*. The whole site, where the archives inherits the style, works
fairly badly on small screens (and really big ones, it only really
works well for medium sized ones).

But is it actually any worse than the old archives? Because they work
pretty bad in mobile as well, don't they? Personally, I find them even
harder since the text of the emails tends to be smaller in comparison
to the header... And if it's not a regression against the new ones, I
think it needs to go on the TODO list rather than being a blocker..


> (And that was on an iPad; don't even want to think about a phone.)
> I can't see anybody really caring about either the mbox or raw links
> in that context.
>
> But on the third hand ... could we rig it to accept any old name and
> password?  The mere occurrence of a challenge ought to be enough to
> discourage most bots.

Not easily. We could for the raw links, because that authentication
prompt comes from our app. But the mboxes are served directly by the
webserver, which has a fixed password list. So we'd have to write our
own auth module to do that, which is a piece of work I don't think we
want to take on.


--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus@hagander.net> writes:
> On Mon, Dec 31, 2012 at 11:47 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> At least on iOS 6, Safari doesn't seem to show any 401 page at all.
>> When you hit the "raw" link, you get an "Authentication required"
>> popup with just space for username and password.  If you put in
>> a wrong value, the popup re-appears.  There's not much you can
>> do except hit "Cancel".  Not very helpful at all I'd say.  (Now
>> admittedly, on a phone-size screen it's not clear that there's
>> room for much of a prompt, but still...)

> Well, the page usually shows up once you hit cancel.

Not in Safari...

> There is plenty of room on the phone screen to do a prompt. At least
> android has no problem at all with it. But that doesn't really matter
> if a platform that's half of our mobile visitors can't handle it -
> because we can't change that. Unless we want to take the same approach
> as we do with some of the windows code, which is say "it's good
> enough, if people want the better functionality they should pick a
> more suitable platform". (which for the access of raw or mbox isn't
> entirely unreasonable, really..)

I don't have a problem with that approach.  As you say, a
mobile-optimized version of the site would be the only real fix for some
of these issues, and it doesn't seem to me that we have the manpower or
interest for that.  (Yet, anyway.)

> But is it actually any worse than the old archives?

Only in this one regard of having an unexpected password prompt show up.
        regards, tom lane

Magnus Hagander wrote:

> * It's now possible to navigate the whole thread from inside a message
> view, using a dropdown listing the whole thing. It's still possible to
> navigate with to the next and previous in thread at the bottom of the
> message, just like before.

After exchanging a couple of comments with Dimitri over IM on this
topic, I wonder if there's a better way to represent the sorrounding
thread contect, rather than just "responses" and "in response to" links.
The dropdown is good to have the whole thing in a single view, but when
Dimitri pointed me to this archive sample:
https://lists.gnu.org/archive/html/emacs-devel/2012-12/msg00692.html
I wonder if we can do something similar.  So we would list:

1. the parent of the current message (a clickable link)
2. all children of the current message's parent, including the current
message itself.  All but the current one would be clickable links.
Current one would be in a different color, and perhaps have some other
"you are here" marker.
3. all children of the current message

This way, we're not drawing a full graph of the thread (which would be
too large), but a graph complete enough to see the immediate context.

(It might be useful to annotate the pruned ancestors and sub-branches
with some numbers to let the user know there are hidden messages there.)

--
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Alvaro Herrera wrote:
> Magnus Hagander wrote:
>
> > * It's now possible to navigate the whole thread from inside a message
> > view, using a dropdown listing the whole thing. It's still possible to
> > navigate with to the next and previous in thread at the bottom of the
> > message, just like before.
>
> After exchanging a couple of comments with Dimitri over IM on this
> topic, I wonder if there's a better way to represent the sorrounding
> thread contect, rather than just "responses" and "in response to" links.
> The dropdown is good to have the whole thing in a single view, but when
> Dimitri pointed me to this archive sample:
> https://lists.gnu.org/archive/html/emacs-devel/2012-12/msg00692.html
> I wonder if we can do something similar.

FWIW I don't think we should hold migration for this; I'm just pointing
out a possible way to improve this *after* we migrate.

I also think we should *not* hold migration due to the auth issues Tom
pointed out for mbox and raw access.  I mean, what's the use in having
the mbox in a phone or tablet?  The raw message might be useful, if you
have a good enough text editor on the phone; I don't really know if
Emacs has been ported to iPad?

--
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Jan 3, 2013 at 5:18 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>
> I don't really know if
> Emacs has been ported to iPad?

No, why would anyone want that? Vim is available of course :-)

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>> After exchanging a couple of comments with Dimitri over IM on this
>> topic, I wonder if there's a better way to represent the sorrounding
>> thread contect, rather than just "responses" and "in response to" links.
>> The dropdown is good to have the whole thing in a single view, but when
>> Dimitri pointed me to this archive sample:
>> https://lists.gnu.org/archive/html/emacs-devel/2012-12/msg00692.html
>> I wonder if we can do something similar.
>
> FWIW I don't think we should hold migration for this; I'm just pointing
> out a possible way to improve this *after* we migrate.

+1 to both.

> I also think we should *not* hold migration due to the auth issues Tom
> pointed out for mbox and raw access.  I mean, what's the use in having
> the mbox in a phone or tablet?  The raw message might be useful, if you
> have a good enough text editor on the phone; I don't really know if
> Emacs has been ported to iPad?

There is of course Emacs Gestures for touch screens. Still, +1.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> I also think we should *not* hold migration due to the auth issues Tom
> pointed out for mbox and raw access.  I mean, what's the use in having
> the mbox in a phone or tablet?  The raw message might be useful, if you
> have a good enough text editor on the phone; I don't really know if
> Emacs has been ported to iPad?

FWIW, it wasn't me that pointed those out.  I'm fine with not fixing
them before we migrate (in fact, not sure we need to fix 'em ever).
        regards, tom lane

Magnus Hagander wrote:

> Now, having said that, I'd like to see some more people testing it
> than the few people I've forced to do it so far. The way to test it is
> to go to http://www.postgresql.org/list/ and pick your list. You can
> also go to http://www.postgresql.org/message-id/<message-id> to view a
> message in a thread - that should also work fine if you just replace
> "archives" with "www" in the URL of an existing message in the
> archives *if* you were using the message-id based url.

Actually, hold the presses.  See this message:
http://www.postgresql.org/message-id/20110907091634.GL24583@sonic.net
It says "the patch is attached", but there is no attachment.
The old archives do contain the patch.

--
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Fri, Jan 4, 2013 at 3:19 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
> Magnus Hagander wrote:
>
>> Now, having said that, I'd like to see some more people testing it
>> than the few people I've forced to do it so far. The way to test it is
>> to go to http://www.postgresql.org/list/ and pick your list. You can
>> also go to http://www.postgresql.org/message-id/<message-id> to view a
>> message in a thread - that should also work fine if you just replace
>> "archives" with "www" in the URL of an existing message in the
>> archives *if* you were using the message-id based url.
>
> Actually, hold the presses.  See this message:
> http://www.postgresql.org/message-id/20110907091634.GL24583@sonic.net
> It says "the patch is attached", but there is no attachment.
> The old archives do contain the patch.

Bug found and fixed. The problem was that we missed attachments that
were text/plain, did not have a name, but did have a filename. I first
thought it was another case of the From line in mbox, but if you
looked at the raw output you could see that we actually had the whole
thing in there. So in passing, I also wrote a small tool that lets us
reparse messages that are already in the db, whenever we change/fix
the parsing rules.

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

Another minor gripe about the new archives: compare
http://archives.postgresql.org/pgsql-hackers/2013-01/msg00218.php
http://www.postgresql.org/message-id/17517.1357496283@sss.pgh.pa.us

Although we're not exactly big on pretty pictures on these lists,
I think it's a usability disimprovement that the new archive fails
to include the image in-line.
        regards, tom lane

On Sun, Jan 6, 2013 at 8:12 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Another minor gripe about the new archives: compare
> http://archives.postgresql.org/pgsql-hackers/2013-01/msg00218.php
> http://www.postgresql.org/message-id/17517.1357496283@sss.pgh.pa.us
>
> Although we're not exactly big on pretty pictures on these lists,
> I think it's a usability disimprovement that the new archive fails
> to include the image in-line.

Depends on the image, I think. For this specific image, I agree it is.
But if the image is larger (which is not unusual for example for
screenshots) it breaks the layout pretty badly.

We could inline all images, with that risk. Or maybe we could actually
parse the image to figure out it's size, and show it only when it's
smaller than a certain size. That might be a useful compromise...

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On 6 January 2013 20:41, Magnus Hagander <magnus@hagander.net> wrote:
> We could inline all images, with that risk. Or maybe we could actually
> parse the image to figure out it's size, and show it only when it's
> smaller than a certain size. That might be a useful compromise...

How likely is it in practice that an image is any other type of image?
Ideally, you'd be able to do something with the image metadata to
inline an image if its resolution is below a certain threshold. If
that doesn't work out, I'd just inline png, gif and svg images, and
leave it at that.

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

On Sun, Jan 6, 2013 at 9:51 PM, Peter Geoghegan <peter@2ndquadrant.com> wrote:
> On 6 January 2013 20:41, Magnus Hagander <magnus@hagander.net> wrote:
>> We could inline all images, with that risk. Or maybe we could actually
>> parse the image to figure out it's size, and show it only when it's
>> smaller than a certain size. That might be a useful compromise...
>
> How likely is it in practice that an image is any other type of image?
> Ideally, you'd be able to do something with the image metadata to
> inline an image if its resolution is below a certain threshold. If
> that doesn't work out, I'd just inline png, gif and svg images, and
> leave it at that.

Well, that's what I'm considering, but inlining screenshots at high
resolutions is going to look Really Bad... But yes, the idea would be
to parse out the resolution from the metadata somehow.


--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On Sun, Jan 6, 2013 at 8:53 PM, Magnus Hagander <magnus@hagander.net> wrote:
> Well, that's what I'm considering, but inlining screenshots at high
> resolutions is going to look Really Bad... But yes, the idea would be
> to parse out the resolution from the metadata somehow.

I would be more concerned with the file size. You don't want someone
to be able to make an archive page send a hapless viewer a 100M image
file. Luckily file size doesn't require parsing the meta data.

Can you not deal with the image size by just sticking a css attribute
for max-height and max-width on the inlined image?

-- 
greg

On Sun, Jan 6, 2013 at 9:59 PM, Greg Stark <stark@mit.edu> wrote:
> On Sun, Jan 6, 2013 at 8:53 PM, Magnus Hagander <magnus@hagander.net> wrote:
>> Well, that's what I'm considering, but inlining screenshots at high
>> resolutions is going to look Really Bad... But yes, the idea would be
>> to parse out the resolution from the metadata somehow.
>
> I would be more concerned with the file size. You don't want someone
> to be able to make an archive page send a hapless viewer a 100M image
> file. Luckily file size doesn't require parsing the meta data.
>
> Can you not deal with the image size by just sticking a css attribute
> for max-height and max-width on the inlined image?

That's certainly another option - limit it by size that way, and then
limiting it by filesize for whether we include it or not. We can even
give it scrollbars, I guess. That does make even more sense.

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus@hagander.net> writes:
> On Sun, Jan 6, 2013 at 9:59 PM, Greg Stark <stark@mit.edu> wrote:
>> I would be more concerned with the file size. You don't want someone
>> to be able to make an archive page send a hapless viewer a 100M image
>> file. Luckily file size doesn't require parsing the meta data.
>> 
>> Can you not deal with the image size by just sticking a css attribute
>> for max-height and max-width on the inlined image?

> That's certainly another option - limit it by size that way, and then
> limiting it by filesize for whether we include it or not. We can even
> give it scrollbars, I guess. That does make even more sense.

A file-size-based limit on what we'll inline seems perfectly sensible
from here.  As for scrollbars, if it's big enough to need those, we
probably don't want it inline ...
        regards, tom lane

On Sun, Jan 6, 2013 at 10:25 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Magnus Hagander <magnus@hagander.net> writes:
>> On Sun, Jan 6, 2013 at 9:59 PM, Greg Stark <stark@mit.edu> wrote:
>>> I would be more concerned with the file size. You don't want someone
>>> to be able to make an archive page send a hapless viewer a 100M image
>>> file. Luckily file size doesn't require parsing the meta data.
>>>
>>> Can you not deal with the image size by just sticking a css attribute
>>> for max-height and max-width on the inlined image?
>
>> That's certainly another option - limit it by size that way, and then
>> limiting it by filesize for whether we include it or not. We can even
>> give it scrollbars, I guess. That does make even more sense.
>
> A file-size-based limit on what we'll inline seems perfectly sensible
> from here.  As for scrollbars, if it's big enough to need those, we
> probably don't want it inline ...

Yeah, I was thinking scrollbars rather than cut. Turns out the
max-width option actually rescales the image, so we're fine.

I've applied a change that makes it now inline all images <75kb, and
rescaling them down to 600 pixels wide if they are bigger.

The example URL you used have been reloaded from the new code, so it
shows what it looks like. Any other pages that had images on them that
were already viewed are currently cached the old way, but will
automatically update once they expire from the cache (max 4 hours).

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus@hagander.net> writes:
> I've applied a change that makes it now inline all images <75kb, and
> rescaling them down to 600 pixels wide if they are bigger.

Better, but in
http://www.postgresql.org/message-id/8803.1357514243@sss.pgh.pa.us
it seems to have forgotten about the second attachment ...
        regards, tom lane

On Mon, Jan 7, 2013 at 12:21 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Magnus Hagander <magnus@hagander.net> writes:
>> I've applied a change that makes it now inline all images <75kb, and
>> rescaling them down to 600 pixels wide if they are bigger.
>
> Better, but in
> http://www.postgresql.org/message-id/8803.1357514243@sss.pgh.pa.us
> it seems to have forgotten about the second attachment ...

Turns out that one was because the second attachment didn't have a
name, and as such it was confusing it with the majordomo-added footer.
Basically, any text/plain parts that didn't have a filename were
ignored. Code updated so it now includes any except the first one
(which is the mail itself) as an attachment.

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On Fri, Dec 28, 2012 at 03:32:21PM +0100, Magnus Hagander wrote:
> * There's also a "flat view" that shows an entire message thread on
> one page, for those who prefer that kind of view. (It works pretty
> well in most cases, but becomes a huge page for example on the 344
> message thread)

I will use this view all the time.  Thanks.

> * Raw messages and mbox files are now protected with http basic
> authentication and a fixed password.

For at least pgsql-hackers and pgsql-bugs, the mbox files appear to have
frozen on 2012-12-09.  The 201212 files have no messages past that date, and
the 201301 files do not exist.

On Sun, Jan 13, 2013 at 6:00 PM, Noah Misch <noah@leadboat.com> wrote:
> On Fri, Dec 28, 2012 at 03:32:21PM +0100, Magnus Hagander wrote:
>> * There's also a "flat view" that shows an entire message thread on
>> one page, for those who prefer that kind of view. (It works pretty
>> well in most cases, but becomes a huge page for example on the 344
>> message thread)
>
> I will use this view all the time.  Thanks.
>
>> * Raw messages and mbox files are now protected with http basic
>> authentication and a fixed password.
>
> For at least pgsql-hackers and pgsql-bugs, the mbox files appear to have
> frozen on 2012-12-09.  The 201212 files have no messages past that date, and
> the 201301 files do not exist.

Ah, it appears I forgot to enable a cronjob. Should be done now.


--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/

On Sat, Jan 05, 2013 at 03:05:58PM +0100, Magnus Hagander wrote:
> On Fri, Jan 4, 2013 at 3:19 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
> > Magnus Hagander wrote:
> >
> >> Now, having said that, I'd like to see some more people testing it
> >> than the few people I've forced to do it so far. The way to test it is
> >> to go to http://www.postgresql.org/list/ and pick your list. You can
> >> also go to http://www.postgresql.org/message-id/<message-id> to view a
> >> message in a thread - that should also work fine if you just replace
> >> "archives" with "www" in the URL of an existing message in the
> >> archives *if* you were using the message-id based url.
> >
> > Actually, hold the presses.  See this message:
> > http://www.postgresql.org/message-id/20110907091634.GL24583@sonic.net
> > It says "the patch is attached", but there is no attachment.
> > The old archives do contain the patch.
> 
> Bug found and fixed. The problem was that we missed attachments that
> were text/plain, did not have a name, but did have a filename. I first
> thought it was another case of the From line in mbox, but if you
> looked at the raw output you could see that we actually had the whole
> thing in there. So in passing, I also wrote a small tool that lets us
> reparse messages that are already in the db, whenever we change/fix
> the parsing rules.

The following message exhibits similar symptoms; the attachment appears in the
raw and mbox versions only:

http://www.postgresql.org/message-id/20130103031358.GB11705@tornado.leadboat.com

It, too, is text/plain with a "filename" and no "name".  Some attachments I
sent on 2013-01-08, using the same email toolchain, show up fine.

On Thu, Jan 17, 2013 at 4:05 AM, Noah Misch <noah@leadboat.com> wrote:
> On Sat, Jan 05, 2013 at 03:05:58PM +0100, Magnus Hagander wrote:
>> On Fri, Jan 4, 2013 at 3:19 PM, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>> > Magnus Hagander wrote:
>> >
>> >> Now, having said that, I'd like to see some more people testing it
>> >> than the few people I've forced to do it so far. The way to test it is
>> >> to go to http://www.postgresql.org/list/ and pick your list. You can
>> >> also go to http://www.postgresql.org/message-id/<message-id> to view a
>> >> message in a thread - that should also work fine if you just replace
>> >> "archives" with "www" in the URL of an existing message in the
>> >> archives *if* you were using the message-id based url.
>> >
>> > Actually, hold the presses.  See this message:
>> > http://www.postgresql.org/message-id/20110907091634.GL24583@sonic.net
>> > It says "the patch is attached", but there is no attachment.
>> > The old archives do contain the patch.
>>
>> Bug found and fixed. The problem was that we missed attachments that
>> were text/plain, did not have a name, but did have a filename. I first
>> thought it was another case of the From line in mbox, but if you
>> looked at the raw output you could see that we actually had the whole
>> thing in there. So in passing, I also wrote a small tool that lets us
>> reparse messages that are already in the db, whenever we change/fix
>> the parsing rules.
>
> The following message exhibits similar symptoms; the attachment appears in the
> raw and mbox versions only:
>
> http://www.postgresql.org/message-id/20130103031358.GB11705@tornado.leadboat.com
>
> It, too, is text/plain with a "filename" and no "name".  Some attachments I
> sent on 2013-01-08, using the same email toolchain, show up fine.

Yeah, Alvaro fixed that one - I just wanted to note it here so you
knew it was fixed :)

We haven't figured out how to auto-reparse all those messages yet, so
right now we're fixing them on a case by case basis when they're
found. So keep reporting them!

--Magnus HaganderMe: http://www.hagander.net/Work: http://www.redpill-linpro.com/