Thread: prevent encoding conversion recursive error
As this thread reports (some sever messages are in Chinese):
http://archives.postgresql.org/pgsql-bugs/2005-07/msg00247.php
a SQL grammar error could crash the error stack.
My explaination is that "select;" incurs a parse error and this error
message is
supposed to be translated into your encoding, but unfortunately not every
UTF8 character is necessarily be encoded as GB18030, which will cause an
infinite recursive elogs just like this:
1:elog(parse_error) // contain unencodable characters
2: elog(report_not_translatable) // contain unencodable characters
again
3: elog(report_report_not_translatable)
4: elog(report_report_report_not_translatable)
5: ...
and corrupt the elog stack.
To fix this, we just change errmsg() to errmsg_internal() to avoid
tranlation which could stop the recursion at step 2.
Regards,
Qingqing
---
Index: backend/utils/mb/conv.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/utils/mb/conv.c,v
retrieving revision 1.53
diff -c -r1.53 conv.c
*** backend/utils/mb/conv.c 15 Jun 2005 00:15:08 -0000 1.53
--- backend/utils/mb/conv.c 4 Aug 2005 08:33:57 -0000
***************
*** 380,386 ****
{
ereport(WARNING,
(errcode(ERRCODE_UNTRANSLATABLE_CHARACTER),
! errmsg("ignoring unconvertible UTF8
character 0x%04x",
iutf)));
continue;
}
--- 380,386 ----
{
ereport(WARNING,
(errcode(ERRCODE_UNTRANSLATABLE_CHARACTER),
! errmsg_internal("ignoring unconvertible
UTF8 character 0x%04x",
iutf)));
continue;
}
***************
*** 449,455 ****
{
ereport(WARNING,
(errcode(ERRCODE_UNTRANSLATABLE_CHARACTER),
! errmsg("ignoring unconvertible %s
character 0x%04x",
(&pg_enc2name_tbl[encoding])->name, iiso)));
continue;
}
--- 449,455 ----
{
ereport(WARNING,
(errcode(ERRCODE_UNTRANSLATABLE_CHARACTER),
! errmsg_internal("ignoring
unconvertible %s character 0x%04x",
(&pg_enc2name_tbl[encoding])->name, iiso)));
continue;
}
"Qingqing Zhou" <zhouqq@cs.toronto.edu> writes: > As this thread reports (some sever messages are in Chinese): > http://archives.postgresql.org/pgsql-bugs/2005-07/msg00247.php > a SQL grammar error could crash the error stack. Hmm, I thought we had fixed that years ago. > To fix this, we just change errmsg() to errmsg_internal() to avoid > tranlation which could stop the recursion at step 2. This is a really ugly solution ... and I don't think it solves the general problem anyway, since this isn't the only possible error message. I don't seem to have gotten the original problem report, and the archive page is pretty useless because all the non-ASCII characters have gotten changed to "?". Could you post a self-contained example case? Might be best to wrap it as a compressed attachment so it doesn't get munged in transmission. regards, tom lane
"Tom Lane" <tgl@sss.pgh.pa.us> writes > > This is a really ugly solution ... and I don't think it solves the > general problem anyway, since this isn't the only possible error message. > Yeah, it is not a very clean solution. Do you mean the general problem is "prevent recursive error reporting because of the error in transalting error message"? I put the image of the reporting email here: http://www.cs.toronto.edu/~zhouqq/encode.jpg Regards, Qingqing
"Qingqing Zhou" <zhouqq@cs.toronto.edu> writes: > Yeah, it is not a very clean solution. Do you mean the general problem is > "prevent recursive error reporting because of the error in transalting error > message"? > I put the image of the reporting email here: > http://www.cs.toronto.edu/~zhouqq/encode.jpg Actually, I believe the general problem is that the gettext software is doing the wrong internal character-set conversion for translated message texts. I can get this same crash on a Linux machine if I have server encoding = utf8 and client encoding = gb18030 and I set lc_messages = zh_TW ... but if I instead make lc_messages = zh_CN, no problem. The backend zh_TW.po file contains msgid "ignoring unconvertible UTF-8 character 0x%04x" msgstr "忽略無法轉換的UTF-8字元0x%04x" and if I read the header correctly, this is claimed to be in UTF8 encoding. So it ought to be delivered as-is when in a UTF8 database. But tracing through the failure with gdb, I see that what is actually delivered back from gettext() is (gdb) p str $1 = 0x82e8a74 "����?��??��UTF-8��Ԫ0xd4da" (gdb) x/32cx str 0x82e8a74: 0xba 0xf6 0xc2 0xd4 0x3f 0xb7 0xa8 0x3f 0x82e8a7c: 0x3f 0xb5 0xc4 0x55 0x54 0x46 0x2d 0x38 0x82e8a84: 0xd7 0xd6 0xd4 0xaa 0x30 0x78 0x64 0x34 0x82e8a8c: 0x64 0x61 0x00 0x7e 0x7f 0x7f 0x7f 0x7f (gdb) so some sort of conversion has taken place. I had initially initialized the database with initdb --locale=zh_CN, which was interpreted by Postgres as requesting EUC_CN encoding. I suspect the above is the EUC_CN equivalent of the message text from the .po file, and that the real problem is that gettext() has not been told the correct character set to convert messages to. ISTM we've seen this issue before and Peter had an idea how to fix it, but I forget the details. Peter? regards, tom lane
I wrote: > ...real problem is that gettext() has not been told the correct character > set to convert messages to. > ISTM we've seen this issue before and Peter had an idea how to fix it, > but I forget the details. Peter? A little bit of digging in the list archives located http://archives.postgresql.org/pgsql-hackers/2003-11/msg01299.php in which Peter opines : - lc_collate and lc_ctype need to be held fixed in the entire cluster. : : - Gettext relies on iconv character set conversion, which relies on : lc_ctype, which leads to a complete screw-up in the server because of : the previous item. which seems to fit with my observation: the message texts are being converted to the cluster's original encoding rather than the encoding that's active in the current database. This does not look real easy to fix. Who's up for reimplementing gettext and a few other pieces from scratch? There is a separate line of thought here, which is that we are unlikely ever to get this completely perfect, and so it'd be good if errors during error processing didn't lead to recursion and PANIC. I don't have an idea how to solve that one either. regards, tom lane
I wrote: > This does not look real easy to fix. Who's up for reimplementing > gettext and a few other pieces from scratch? However, I did find http://gnu.miscellaneousmirror.org/software/libc/manual/html_node/Charset-conversion-in-gettext.html#Charset-conversion-in-gettext which leads to the question "why aren't we using bind_textdomain_codeset() to tell gettext what character set it should produce"? regards, tom lane
Am Dienstag, 9. August 2005 04:51 schrieb Tom Lane: > which leads to the question "why aren't we using > bind_textdomain_codeset() to tell gettext what character set it should > produce"? That would probably require us to solve the question on how to translate PostgreSQL encoding names to OS encoding names.
Peter Eisentraut <peter_e@gmx.net> writes:
> Am Dienstag, 9. August 2005 04:51 schrieb Tom Lane:
>> which leads to the question "why aren't we using
>> bind_textdomain_codeset() to tell gettext what character set it should
>> produce"?
> That would probably require us to solve the question on how to translate
> PostgreSQL encoding names to OS encoding names.
Yeah, but don't we already have some code for that (or, actually, the
reverse direction) in initdb? It's probably not perfect, but it'd be
a lot better than crashing.
regards, tom lane
Am Sonntag, 14. August 2005 23:48 schrieb Tom Lane: > Yeah, but don't we already have some code for that (or, actually, the > reverse direction) in initdb? It's probably not perfect, but it'd be > a lot better than crashing. The reverse direction is a lot simpler because we know the set of possible output values. I'm not sure how to do the mapping in the direction of the OS.
Peter Eisentraut wrote: > Am Sonntag, 14. August 2005 23:48 schrieb Tom Lane: > > Yeah, but don't we already have some code for that (or, actually, the > > reverse direction) in initdb? It's probably not perfect, but it'd be > > a lot better than crashing. > > The reverse direction is a lot simpler because we know the set of possible > output values. I'm not sure how to do the mapping in the direction of the > OS. Is there a TODO here? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Is there a TODO here?
Yeah:
* Fix problems with wrong runtime encoding conversion for NLS message files
One thing that occurred to me is that we might be able to simplify the
problem by adopting a project standard that all NLS message files shall
be in UTF8, period. Then we only have one encoding name to figure out
rather than N. Maybe this doesn't help much ...
regards, tom lane
Am Sonntag, 21. August 2005 01:48 schrieb Tom Lane: > One thing that occurred to me is that we might be able to simplify the > problem by adopting a project standard that all NLS message files shall > be in UTF8, period. Then we only have one encoding name to figure out > rather than N. Maybe this doesn't help much ... I suppose this would then break NLS on Windows, no? -- Peter Eisentraut http://developer.postgresql.org/~petere/
On Thu, Sep 01, 2005 at 05:48:55PM +0200, Peter Eisentraut wrote: > Am Sonntag, 21. August 2005 01:48 schrieb Tom Lane: > > One thing that occurred to me is that we might be able to simplify the > > problem by adopting a project standard that all NLS message files shall > > be in UTF8, period. Then we only have one encoding name to figure out > > rather than N. Maybe this doesn't help much ... > > I suppose this would then break NLS on Windows, no? We now have a patch to handle UTF-8 on Windows, via recoding to UTF-16 and back, so I guess not (not sure though). It would manage to annoy me as a translator, but nothing too serious really. -- Alvaro Herrera -- Valdivia, Chile Architect, www.EnterpriseDB.com "Escucha y olvidar�s; ve y recordar�s; haz y entender�s" (Confucio)