Thread: Need help with quote escaping in exim for postgresql
Hi, I am the maintainer of Debian's packages for exim4, a powerful and versatile Mail Transfer Agent developed in Cambridge and in wide use throughout the Free Software Community (http://www.exim.org/). One of our daemon flavours has PostgreSQL support. Our security guys have found a flaw in exim regarding quote escaping for PostgreSQL. The bug is filed in Debian's BTS as http://bugs.debian.org/369351 and was transferred to exim's Bugzilla installation as http://www.exim.org/bugzilla/show_bug.cgi?id=107. Personally, I do not have any PostgreSQL experience (and do not have time and expertise to accumulate any), and the PostgreSQL support code in exim was contributed some time ago and Philip Hazel, exim's author, doesn't know too much about PostgreSQL as well. From what I understand, the correct way would be to use PQescapeStringConn, but that function needs an established connection, and exim performs string escape "early", way before the actual connection is established. I'd appreciate if anybody familiar with PostgreSQL programming could take a look at the two bug reports and probably exim's program code and suggest a possible solution, preferably in the bugzilla issue log referenced above. I'll monitor this thread for possible solutions and help, though. Any help would be greatly appreciated. Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Mannheim, Germany | lose things." Winona Ryder | Fon: *49 621 72739834 Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835
On Fri, Jul 07, 2006 at 03:48:00PM +0200, Marc Haber wrote: > Hi, > > I am the maintainer of Debian's packages for exim4, a powerful and > versatile Mail Transfer Agent developed in Cambridge and in wide use > throughout the Free Software Community (http://www.exim.org/). > > One of our daemon flavours has PostgreSQL support. Our security guys > have found a flaw in exim regarding quote escaping for PostgreSQL. The > bug is filed in Debian's BTS as http://bugs.debian.org/369351 and was > transferred to exim's Bugzilla installation as > http://www.exim.org/bugzilla/show_bug.cgi?id=107. Whether or not the quick fix works for you depends entirly on the encoding used by the client to talk to the database. If the connection is encoded using UTF-8 or any of the Latin series, then it will be fine. The only time it does not work is if the encoding is an encoding where the quote or backslash character can appear as the second character of a multibyte char. This doesn't happen with UTF-8 or any latin encoding. http://www.postgresql.org/docs/techdocs.50 This bit may be useful also (especially the second point): There are a number of mitigating factors that may keep particular applications from being subject to these security risks: * If application always sends untrusted strings as out-of-line parameters, instead of embedding them into SQL commands, it is not vulnerable. * If client_encoding is a single-byte encoding (e.g., one of the LATINx family), there is no vulnerability. * If application cannot pass invalidly encoded data to the server, there is no vulnerability (this probably includes all Java applications, for example, because of Java's handling of Unicode strings). The easiest may be to simply always set the client encoding to something like UTF-8 and work the escaping rules so they work with that. Hope this helps, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to litigate.
Attachment
On Fri, Jul 07, 2006 at 03:48:00PM +0200, Marc Haber wrote: > From what I understand, the correct way would be to use > PQescapeStringConn, but that function needs an established connection, > and exim performs string escape "early", way before the actual > connection is established. I just downloaded the code and looked. The code never looks or checks the encoding of the database. This is bad from a security point of view because that means you have no idea how your queries are going to be interpreted. I'd suggest adding a PQsetClientEncoding(conn, "Latin1") right after you establish a connection. I'm not sure if Exim has any kind of declaration about what encoding strings have internally. You could use UTF-8 but then postgres would complain if you pass any strings that arn't valid UTF-8. They may or may not be desirable. SQL_ASCII may also be an option (assign no special meaning to characters at all), but I'm less sure of that. Can email address contain multibyte characters? I didn't think so... What about the configuration file? Hope this helps, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to litigate.
Attachment
On Fri, Jul 07, 2006 at 04:53:14PM +0200, Martijn van Oosterhout wrote: > On Fri, Jul 07, 2006 at 03:48:00PM +0200, Marc Haber wrote: > > I am the maintainer of Debian's packages for exim4, a powerful and > > versatile Mail Transfer Agent developed in Cambridge and in wide use > > throughout the Free Software Community (http://www.exim.org/). > > > > One of our daemon flavours has PostgreSQL support. Our security guys > > have found a flaw in exim regarding quote escaping for PostgreSQL. The > > bug is filed in Debian's BTS as http://bugs.debian.org/369351 and was > > transferred to exim's Bugzilla installation as > > http://www.exim.org/bugzilla/show_bug.cgi?id=107. > > Whether or not the quick fix works for you depends entirly on the > encoding used by the client to talk to the database. I am afraid that we don't have any information about the encoding used since exim allows arbitrary SQL commands to be given in the run-time configuration file. > * If application always sends untrusted strings as out-of-line > parameters, instead of embedding them into SQL commands, it is not > vulnerable. exim's configuration holds SQL string literals like |server_condition = "${if and { \ | {!eq{$2}{}} \ | {!eq{$3}{}} \ | {crypteq{$3}{${lookup mysql{SELECT password FROM customers WHERE ( domain_name = \ | '${domain:$2}' \ | AND local_part = '${local_part:$2}') OR user_id='$2'}{$value}fail}} }} {yes}{no}}" (which is code helping exim to do SMTP AUTH against a password database stored in mysql. Using PostgreSQL is very similiar, so you'll get the idea - taken from http://www.exim.org/eximwiki/AuthenticatedSmtpUsingMysql). > * If client_encoding is a single-byte encoding (e.g., one of the > LATINx family), there is no vulnerability. From what I can see, exim configuration does not allow an encoding to be set fot the communication with pgsql, and the code does not match on "client_encoding". Am I right to assume that without special handling, anything passed to it from the configuration is passed on verbatim to the pgsql server? > * If application cannot pass invalidly encoded data to the server, > there is no vulnerability (this probably includes all Java > applications, for example, because of Java's handling of Unicode > strings). I am afraid that it is quite easy to (mis)configure exim so that it is possible to pass invalidly encoded data. > The easiest may be to simply always set the client encoding to > something like UTF-8 and work the escaping rules so they work with > that. If that's possible, it might. We'd need to know how our input data is encoded and recode to UTF-8, right? Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Mannheim, Germany | lose things." Winona Ryder | Fon: *49 621 72739834 Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835
Hi, On Fri, Jul 07, 2006 at 05:15:11PM +0200, Martijn van Oosterhout wrote: > On Fri, Jul 07, 2006 at 03:48:00PM +0200, Marc Haber wrote: > > From what I understand, the correct way would be to use > > PQescapeStringConn, but that function needs an established connection, > > and exim performs string escape "early", way before the actual > > connection is established. > > I just downloaded the code and looked. Thank you very much. > The code never looks or checks the encoding of the database. This is > bad from a security point of view because that means you have no idea > how your queries are going to be interpreted. That's the way exim has always done it. > I'd suggest adding a PQsetClientEncoding(conn, "Latin1") right after > you establish a connection. I'm not sure if Exim has any kind of > declaration about what encoding strings have internally. No, it does not. > You could use UTF-8 but then postgres would complain if you pass any > strings that arn't valid UTF-8. They may or may not be desirable. Possible not desireable. > SQL_ASCII may also be an option (assign no special meaning to > characters at all), but I'm less sure of that. Can email address > contain multibyte characters? I didn't think so... E-Mail addreses themselves can't, but the "comment" field of an address can. > What about the configuration file? It probably can as well. Please note that exim is so flexible that it is possible to implement mail spool storage in an SQL database. In this case, we'd write data which originated in an untrusted source to the database, not knowing about encoding at all. I'm going to point Philip to this thread. Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Mannheim, Germany | lose things." Winona Ryder | Fon: *49 621 72739834 Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835
* Martijn van Oosterhout: > * If application always sends untrusted strings as out-of-line > parameters, instead of embedding them into SQL commands, it is not > vulnerable. This paragraph should explictly mention PQexecParams (which everybody should use anyway). It seems that Exim's architecture prevents the use of PQexecParams, though.
On Sun, Jul 09, 2006 at 06:16:48PM +0200, Marc Haber wrote: > > I'd suggest adding a PQsetClientEncoding(conn, "Latin1") right after > > you establish a connection. I'm not sure if Exim has any kind of > > declaration about what encoding strings have internally. > > No, it does not. That's your fundamental problem. You cannot safely quote strings if you do not know the encoding of the string you're escaping. That's fundamentally what this bug is about. > > SQL_ASCII may also be an option (assign no special meaning to > > characters at all), but I'm less sure of that. Can email address > > contain multibyte characters? I didn't think so... > > E-Mail addreses themselves can't, but the "comment" field of an > address can. But no validation is done on these strings, ofcourse. > > What about the configuration file? > > It probably can as well. Ditto. I'd suggest forcing the client_encoding to SQL_ASCII. That way multibyte characters will simply get dumped into the strings in the backend. But at least there won't be any issues with the server misinterpreting your string. It may be worthwhile allowing the user to set the encoding, but that just opens the issue up again, because then you actually have to validate the strings are the encoding you think they are. > > Please note that exim is so flexible that it is possible to implement > mail spool storage in an SQL database. In this case, we'd write data > which originated in an untrusted source to the database, not knowing > about encoding at all. That will work just fine, as long as the database encoding is SQL_ASCII, in which case there is no problem... What we're talking about here is invalidly coded UTF-8 and things like SJIS. Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to litigate.
Attachment
Marc Haber wrote: > Please note that exim is so flexible that it is possible to implement > mail spool storage in an SQL database. In this case, we'd write data > which originated in an untrusted source to the database, not knowing > about encoding at all. If you are going to store things in multiple encodings and you don't know (or don't want to waste time figuring out) which encoding each is on, probably you should be using SQL_ASCII. This means Postgres itself will never try to mess with bytes (it will never recode stuff). The downside is that the sort order may be incorrect in some queries, but I'm not sure if this is a great problem for a mail spool. Otherwise you do have to declare the encoding you are going to enter data in (using the client_encoding config option). Not doing it causes all sort of weird problems. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
> > SQL_ASCII may also be an option (assign no special meaning to > > characters at all), but I'm less sure of that. Can email address > > contain multibyte characters? I didn't think so... > > E-Mail addreses themselves can't, but the "comment" field of an > address can. The comment field itself, in RFC2822 addresses is ascii. However there is a hack, by which this ascii string may be interpretted as representing a string in another encoding. I don't remember the number of the RFC describing how this works, but it shouldn't be hard to find.
On Jul 11, 2006, at 6:29 PM, Bruno Wolff III wrote: >>> SQL_ASCII may also be an option (assign no special meaning to >>> characters at all), but I'm less sure of that. Can email address >>> contain multibyte characters? I didn't think so... >> >> E-Mail addreses themselves can't, but the "comment" field of an >> address can. > > The comment field itself, in RFC2822 addresses is ascii. However > there is > a hack, by which this ascii string may be interpretted as > representing a > string in another encoding. I don't remember the number of the RFC > describing > how this works, but it shouldn't be hard to find. RFC 1522. Whether you'd want to open that can of worms by decoding headers in that format to some other character set for email is a good question (especially as a lot of spam has headers that end up containing illegal characters if you do that). If you were to do that you'd probably have to deal with i18n domain name encoding too, which is even more hideous. Fortunately all this stuff is MUA-side, not MTA-side, so exim should ignore it. SQL_ASCII all the way. Cheers, Steve
[ Coming late to the thread... ]
Steve Atkins <steve@blighty.com> writes:
> Fortunately all this stuff is MUA-side, not MTA-side, so exim
> should ignore it. SQL_ASCII all the way.
I concur. The recent encoding fixes are for the situation where the
database server believes a multibyte encoding is in use, but the client
code is ignorant of that encoding and either (a) sends invalidly encoded
data or (b) does escaping that mangles multibyte characters.
If your client-side code is encoding agnostic, then using SQL_ASCII
(which is also effectively encoding agnostic) for both client_encoding
and server_encoding will work nicely.
A possibly safer choice is to use LATIN1 (or another single-byte
encoding) instead; this will avoid problems if someone connects to the
database with a client_encoding other than SQL_ASCII and expects data
to be delivered to him in that encoding.
I would *not* recommend using UTF8 if you want to store arbitrary data
without worrying about encoding issues.
regards, tom lane
On Fri, Jul 07, 2006 at 03:48:00PM +0200, Marc Haber wrote: > I am the maintainer of Debian's packages for exim4, a powerful and > versatile Mail Transfer Agent developed in Cambridge and in wide use > throughout the Free Software Community (http://www.exim.org/). > > One of our daemon flavours has PostgreSQL support. Our security guys > have found a flaw in exim regarding quote escaping for PostgreSQL. The > bug is filed in Debian's BTS as http://bugs.debian.org/369351 and was > transferred to exim's Bugzilla installation as > http://www.exim.org/bugzilla/show_bug.cgi?id=107. I would like to thank anybody who commented in this thread. Special thanks go to Florian Weimer, who has done the task of communicating with Philip Hazel, the exim author, and working out a fix which has since been released as part of exim 4.63. Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Mannheim, Germany | lose things." Winona Ryder | Fon: *49 621 72739834 Nordisch by Nature | How to make an American Quilt | Fax: *49 621 72739835