Re: bytea encode performance issues - Mailing list pgsql-general

From Daniel Verite
Subject Re: bytea encode performance issues
Date
Msg-id 636207a4-7b4a-45ff-8c59-d53c3b3e7855@mm
Whole thread Raw
In response to Re: bytea encode performance issues  (Steve Atkins <steve@blighty.com>)
List pgsql-general
    Steve Atkins wrote:

> So, yeah, you're right. Generally, email is too complex to deal with

> in the database as anything other than an opaque bytea blob, along
> with some metadata

Only because that's the choice made by dbmail. As an IMAP server, it
doesn't _have_ to do more. The downside is that the database is not as
useful as it could be.

I happen to have developed my own OSS project on exactly this idea: to
have a database of mail with contents in normalized form and
ready-to-be-queried. An picture of the schema can be seen here:
 http://www.manitou-mail.org/articles/db-diagram.html
the architecture being this:
 http://www.manitou-mail.org/schemas/schema1.png

There's nothing particularly remarkable about the schema, except that
there is no trace left of the initial encapsulation of the data inside
an RFC822 message and its associated rules about structure and
encoding.

The next step has been to write a MUA that talks directly in SQL to the
database, and the resulting speed and efficiency is much better than
with traditional IMAP-based MUAs.
As an example related to search, I have this 10Gb database containing
600k mails, and hundreds of results for a full-text search typically
come back to the MUA in a couple of seconds, Gmail-like, on a low-grade
server to which I'm remotely connected through an SSH tunnel. SQL is so
much better without an IMAP layer on top of it...
Now, my dedicated MUA isn't as feature-rich as other popular mailers,
and it can't be used offline despite being a desktop app, and has other
deficiencies, but other mailer/server combinations come with their own
sets of problems and inadequacies, too :)

 Regards,
--
 Daniel
 PostgreSQL-powered mail user agent and storage:
http://www.manitou-mail.org

pgsql-general by date:

Previous
From: "Chandra ASGI Tech"
Date:
Subject: Re: How to get many data at once?
Next
From: "Hiroshi Saito"
Date:
Subject: Re: psqlodbc on Vista Ultimate 64