Thread: BUG #4186: set lc_messages does not work

BUG #4186: set lc_messages does not work

From
"Thomas H"
Date:
The following bug has been logged online:

Bug reference:      4186
Logged by:          Thomas H
Email address:      me@alternize.com
PostgreSQL version: 8.3.1
Operating system:   Windows 2003
Description:        set lc_messages does not work
Details:

the patch discussed here [1] that supposedly made the win32 msvc builds use
lc_locale properly has flaws.

while indeed it does force pgsql to use the native win32 locale for error
messages (before 8.3.1, system messages were always english), it broke the
functionality to actually have the lc_messages locale set manually through
conf or in a user session.

the following output is from a win2003 system with German_Switzerland.1252
locale. the queries are identically to the one used as examples by the patch
author.
i have added a "select x;" to trigger a system error message to show that
its actually misbehaving - all output is always in (swiss) german despite
the set locale.

- thomas

[1]
http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php

--------------------------------

Dies ist psql 8.3.1, das interaktive PostgreSQL-Terminal.

Geben Sie ein:  \copyright für Urheberrechtsinformationen
                \h für Hilfe über SQL-Anweisungen
                \? für Hilfe über interne Anweisungen
                \g oder Semikolon, um eine Anfrage auszuführen
                \q um zu beenden

Warnung: Konsolencodeseite (850) unterscheidet sich von der Windows-
         Codeseite (1252). 8-Bit-Zeichen funktionieren möglicherweise
nicht
         richtig. Einzelheiten finden Sie auf der psql-Handbuchseite unter
         »Notes for Windows users«.

endor=# set lc_messages to 'sv_SE';
SET
endor=# select to_char((current_date + s.a),'TMDay TMMonth YYYY') as dates
from
 generate_series(0,6) as s(a);
        dates
---------------------
 Donnerstag Mai 2008
 Freitag Mai 2008
 Samstag Mai 2008
 Sonntag Mai 2008
 Montag Mai 2008
 Dienstag Mai 2008
 Mittwoch Mai 2008
(7 Zeilen)

endor=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;
                ^
endor=#
endor=# set lc_messages to 'de_DE';
SET
endor=# select to_char((current_date + s.a),'TMDay TMMonth YYYY') as dates
from
 generate_series(0,6) as s(a);
        dates
---------------------
 Donnerstag Mai 2008
 Freitag Mai 2008
 Samstag Mai 2008
 Sonntag Mai 2008
 Montag Mai 2008
 Dienstag Mai 2008
 Mittwoch Mai 2008
(7 Zeilen)

endor=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;
                ^
endor=#
endor=# set lc_messages to 'English_United_States';
SET
endor=# select to_char((current_date + s.a),'TMDay TMMonth YYYY') as dates
from
 generate_series(0,6) as s(a);
        dates
---------------------
 Donnerstag Mai 2008
 Freitag Mai 2008
 Samstag Mai 2008
 Sonntag Mai 2008
 Montag Mai 2008
 Dienstag Mai 2008
 Mittwoch Mai 2008
(7 Zeilen)

endor=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;
                ^
endor=#
endor=# set lc_messages to 'fr';
SET
endor=# select to_char((current_date + s.a),'TMDay TMMonth YYYY') as dates
from
 generate_series(0,6) as s(a);
        dates
---------------------
 Donnerstag Mai 2008
 Freitag Mai 2008
 Samstag Mai 2008
 Sonntag Mai 2008
 Montag Mai 2008
 Dienstag Mai 2008
 Mittwoch Mai 2008
(7 Zeilen)

endor=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;

Re: BUG #4186: set lc_messages does not work

From
Euler Taveira de Oliveira
Date:
Thomas H wrote:

> the patch discussed here [1] that supposedly made the win32 msvc builds use
> lc_locale properly has flaws.
>
I think you misunderstood the feature [1] added recently. This new
feature doesn't rely on lc_messages to localize the day and month names;
it uses lc_time. Look at the archives for reference.

[1] http://archives.postgresql.org/pgsql-patches/2008-04/msg00451.php


--
   Euler Taveira de Oliveira
   http://www.timbira.com/

Re: BUG #4186: set lc_messages does not work

From
"Thomas H."
Date:
euler taveira de olivieira wrote:

>> the patch discussed here [1] that supposedly made the win32 msvc
>> builds use
>> lc_locale properly has flaws.
>>
> I think you misunderstood the feature [1] added recently. This new

actually no. the problem is as i intended to point out with the system
generated error messages. their language is supposedly controlled by
LC_MESSAGES (as was  the time/date output, according to the first
february patch submission).

please observe the (previously already submitted) test queries. i've
removed the date/time testqueries to no further distract from the
problem. the bogus query "select x;" always results in a german error
messages no matter what LC_MESSAGES is set:


endor=# set lc_messages to 'sv_SE';
SET
endor=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;
                 ^
endor=#
endor=# set lc_messages to 'de_DE';
SET
endor=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;
                 ^
endor=#
endor=# set lc_messages to 'English_United_States';
SET
endor=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;
                 ^
endor=#
endor=# set lc_messages to 'fr';
SET
endor=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;


setting LC_MESSAGES in postgres.conf doesn't change anything either. and
the comment there says explicitly that LC_MESSAGES is used for system
errors:

lc_messages = 'English_United_States'            # locale for system error message
                    # strings

nevertheless, everything is outputted in german now, errors as well as
logs. which seems to hint at 8.3.1 ignoring the LC_MESSAGES and always
using the os' locale.

regards,
thomas

Re: BUG #4186: set lc_messages does not work

From
Euler Taveira de Oliveira
Date:
Thomas H. wrote:

> please observe the (previously already submitted) test queries. i've
> removed the date/time testqueries to no further distract from the
> problem. the bogus query "select x;" always results in a german error
> messages no matter what LC_MESSAGES is set:
>
OK, that's another problem. AFAIK, that's a known problem because
Windows doesn't have LC_MESSAGES. The above comment (pg_locale.c)
suggests that there is no verification for the locale that is been set.
A possible solution is to use IsValidLocaleName() [1] or
LocaleNameToLCID() [2] but it seems that they're only available for
Vista. :( Maybe we could emulate one of these functions with a mapping
table [3]. [searching ...] It seems there are problems with LCIDs; they
don't describe the locales acurately. pgwin hackers?
Could you try to use one of the locale names described in [4]?

/*
  * LC_MESSAGES category does not exist everywhere, but accept it anyway
  *
  * On Windows, we can't even check the value, so the non-doit case is a
  * no-op
  */

[1] http://msdn.microsoft.com/en-us/library/ms776379(VS.85).aspx
[2] http://msdn.microsoft.com/en-us/library/ms776388(VS.85).aspx
[3] http://msdn.microsoft.com/en-us/library/ms776260.aspx


--
   Euler Taveira de Oliveira
   http://www.timbira.com/

Re: BUG #4186: set lc_messages does not work

From
"Thomas H."
Date:
Euler Taveira de Oliveira wrote:

>> please observe the (previously already submitted) test queries. i've
>> removed the date/time testqueries to no further distract from the
>> problem. the bogus query "select x;" always results in a german error
>> messages no matter what LC_MESSAGES is set:
>>
> OK, that's another problem. AFAIK, that's a known problem because
> Windows doesn't have LC_MESSAGES. The above comment (pg_locale.c)
> suggests that there is no verification for the locale that is been set.
> A possible solution is to use IsValidLocaleName() [1] or
> LocaleNameToLCID() [2] but it seems that they're only available for
> Vista. :( Maybe we could emulate one of these functions with a mapping
> table [3]. [searching ...] It seems there are problems with LCIDs; they
> don't describe the locales acurately. pgwin hackers?

how does LC_MESSAGES differ from for example LC_TIME? in LC_TIME, the
checking of the specified locale seems to work:

endor=# set LC_MESSAGES = 'en-US';
SET
endor=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;
                 ^
endor=# set LC_TIME = 'en-US';
FEHLER:  ungültiger Wert für Parameter »lc_time«: »en-US«
endor=# set LC_TIME = 'en';
FEHLER:  ungültiger Wert für Parameter »lc_time«: »en«
endor=# set LC_TIME = 'English';
SET

maybe one could as a workaround just use the lc_time locale checks for
lc_messages on windows systems? or at least match against the internal
pgsql supported translations. i don't mind having to specify "en"
instead of "English" if that gets me english error messages ;)

> Could you try to use one of the locale names described in [4]?

i take it you meant link [3]. i've tried 'en-US' and others, same
problem, errors in german (excerpt above).

> [1] http://msdn.microsoft.com/en-us/library/ms776379(VS.85).aspx
> [2] http://msdn.microsoft.com/en-us/library/ms776388(VS.85).aspx
> [3] http://msdn.microsoft.com/en-us/library/ms776260.aspx
>

regards,
thomas

Re: BUG #4186: set lc_messages does not work

From
Tom Lane
Date:
"Thomas H" <me@alternize.com> writes:
> Bug reference:      4186
> Logged by:          Thomas H
> Email address:      me@alternize.com
> PostgreSQL version: 8.3.1
> Operating system:   Windows 2003
> Description:        set lc_messages does not work
> Details:

> the patch discussed here [1] that supposedly made the win32 msvc builds use
> lc_locale properly has flaws.

I think a large part of the confusion that's been evidenced in this
thread is because you are submitting a "bug report" about a patch that is
not in fact in 8.3.1 (and is unlikely ever to appear in 8.3.x at all).
It's unclear what you are really testing: stock 8.3.1?  8.3.1 plus that
patch?  CVS HEAD plus the patch?

            regards, tom lane

Re: BUG #4186: set lc_messages does not work

From
"Thomas H."
Date:
From: Tom Lane <tgl@sss.pgh.pa.us>

>> the patch discussed here [1] that supposedly made the win32 msvc builds use
>> lc_locale properly has flaws.
>
> I think a large part of the confusion that's been evidenced in this
> thread is because you are submitting a "bug report" about a patch that is
> not in fact in 8.3.1 (and is unlikely ever to appear in 8.3.x at all).
> It's unclear what you are really testing: stock 8.3.1?  8.3.1 plus that
> patch?  CVS HEAD plus the patch?

it's the 8.3.1 win32 binary coming from postgresql.org i'm using.

i was digging in the mailing list archives for some answers and found
the conversation about the patch which i believed to be included in
8.3.1. sorry for the mess :|

nevertheless the problem/bug exists: changing LC_MESSAGES has no effect
on the windows boxes, while it works on the non-win32 systems. all i
really would like is to get english system messages back on our
non-english win32 servers - like they were pre 8.3.1.

the main reason for this being: the german error messages have non-ascii
special characters in the error text ('»' and '«') which mess with some
of our applications (whereas english messages use '"').


regards,
thomas

Re: BUG #4186: set lc_messages does not work

From
Tom Lane
Date:
"Thomas H." <me@alternize.com> writes:
> nevertheless the problem/bug exists: changing LC_MESSAGES has no effect
> on the windows boxes, while it works on the non-win32 systems. all i
> really would like is to get english system messages back on our
> non-english win32 servers - like they were pre 8.3.1.

So far as I can tell, the backend's handling of LC_MESSAGES hasn't
changed at all between 8.1.2 and 8.3.1, so if it used to work for you
then there's been some other relevant change.  Any idea what?

            regards, tom lane

Re: BUG #4186: set lc_messages does not work

From
"Thomas H."
Date:
From: Tom Lane <tgl@sss.pgh.pa.us>
> "Thomas H." <me@alternize.com> writes:
>> nevertheless the problem/bug exists: changing LC_MESSAGES has no effect
>> on the windows boxes, while it works on the non-win32 systems. all i
>> really would like is to get english system messages back on our
>> non-english win32 servers - like they were pre 8.3.1.
>
> So far as I can tell, the backend's handling of LC_MESSAGES hasn't
> changed at all between 8.1.2 and 8.3.1, so if it used to work for you
> then there's been some other relevant change.  Any idea what?

well... i'm not saying it worked in earlier versions. but prior to 8.3.1
(i've tested 8.3.0 and 8.2.3), the error messages were in english no
matter what the config file or the os locale says.

now, with 8.3.1, error & system messages are always in the os' locale,
and thus the bugreport. there is currently no way to set the pg system
messages' language, as the LC_MESSAGES setting seems to be defunct on
win32 systems.

as a small "proof", i installed the win32 8.3.0 from postgresql.org as a
fresh install and changed the postgresl.conf' lc_messages value to
'English_UnitedStates'. here's the psql output:

8.3.0:
-----------------------------------
postgres=# show lc_messages;
     lc_messages
--------------------
  English_UnitedStates
(1 row)

postgres=# select x;
ERROR:  column "x" does not exist
LINE 1: select x;
                ^
postgres=# set lc_messages='French';
SET
postgres=# select x;
ERROR:  column "x" does not exist
LINE 1: select x;
                ^
postgres=#
-----------------------------------

--> all system messages are in english.


after *upgrading* to 8.3.1 (again, using the official binaries), the
output looks like this:


8.3.1:
-----------------------------------
postgres=# show lc_messages;
      lc_messages
----------------------
  English_UnitedStates
(1 Zeile)

postgres=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;
                 ^
postgres=# set lc_messages='French';
SET
postgres=# select x;
FEHLER:  Spalte »x« existiert nicht
ZEILE 1: select x;
                 ^
postgres=# show lc_messages;
  lc_messages
-------------
  French
(1 Zeile)

postgres=#

-----------------------------------

--> all system messages are in german (the os' locale)

so clearly between 8.3.0 and 8.3.1, something must have changed. but the
only patch that concerned win32 msvc/locale is the one you said wasn't
even included...

what i noticed: if i delete the folder share/locale/de/ the system
messages are back to english - but that can't be THE solution, can it? :)

regards,
thomas

Re: BUG #4186: set lc_messages does not work

From
"Thomas H."
Date:
From: Thomas H. <me@alternize.com>
> what i noticed: if i delete the folder share/locale/de/ the system
> messages are back to english - but that can't be THE solution, can it? :)

well, it actually was the solution, at least to the weird part of the
problem:

there are two versions of win32 binaries available on postgres.org.
there's the installer msi version, and there is the installerless zip
version.

the installer version does NOT install the folder "share/locale/"! so
when using the msi installer, postgres has no translations at all - thus
the fallback to english.

the zipped version contains the "share/locale/" folder. for installing
8.3.0, i've been using the msi installer version. for upgrading to
8.3.1, i've been unpacking the files from zip version into the
postgresql directory...

i've just verified that the 8.3.1 msi version provided on postgres.org
also does NOT contain the "locale" folder & files. should i report this
as a separate bug/problem?

so at least that explains the "changed" behaviour. nevertheless,
LC_MESSAGES seems to be defunct - with the "locale" folder present, pg
always picks the os' language and ignores the lc_message value.

- thomas

Re: BUG #4186: set lc_messages does not work

From
"Dave Page"
Date:
On Thu, May 29, 2008 at 2:05 AM, Thomas H. <me@alternize.com> wrote:
> From: Thomas H. <me@alternize.com>
>>
> i've just verified that the 8.3.1 msi version provided on postgres.org also
> does NOT contain the "locale" folder & files. should i report this as a
> separate bug/problem?

How exactly did you do that? My installation certainly has it. If
memory serves, it's not installed by default (you have to select it in
the feature list), but it's there alright.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

Re: BUG #4186: set lc_messages does not work

From
"Thomas H."
Date:
From: Dave Page <dpage@pgadmin.org>
> On Thu, May 29, 2008 at 2:05 AM, Thomas H. <me@alternize.com> wrote:
>> From: Thomas H. <me@alternize.com>
>> i've just verified that the 8.3.1 msi version provided on postgres.org also
>> does NOT contain the "locale" folder & files. should i report this as a
>> separate bug/problem?
>
> How exactly did you do that? My installation certainly has it. If
> memory serves, it's not installed by default (you have to select it in
> the feature list), but it's there alright.
>

hmm by just clicking through the standard settings. i've seen now that
the "national language support" is set to do not install by default. so
it is a feature, not a bug, sorry.

i was under the obviously wrong impression a zip-file upgrade would be
the same as an msi-upgrade.

- thomas

Re: BUG #4186: set lc_messages does not work

From
"Dave Page"
Date:
On Thu, May 29, 2008 at 12:23 PM, Thomas H. <me@alternize.com> wrote:

> i was under the obviously wrong impression a zip-file upgrade would be the
> same as an msi-upgrade.

Eeep - don't do that!! That is a *very* good way to annoy the Windows
Installer and trick it into thinking things need to be repaired.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

Re: BUG #4186: set lc_messages does not work

From
Magnus Hagander
Date:
Thomas H. wrote:
> From: Thomas H. <me@alternize.com>
> > what i noticed: if i delete the folder share/locale/de/ the system
> > messages are back to english - but that can't be THE solution, can
> > it? :)
>
> well, it actually was the solution, at least to the weird part of the
> problem:
>
> there are two versions of win32 binaries available on postgres.org.
> there's the installer msi version, and there is the installerless zip
> version.
>
> the installer version does NOT install the folder "share/locale/"! so
> when using the msi installer, postgres has no translations at all -
> thus the fallback to english.

Are you sure you didn't just skip enabling National Language Support?
It's not included by default, but should be available.


> so at least that explains the "changed" behaviour. nevertheless,
> LC_MESSAGES seems to be defunct - with the "locale" folder present,
> pg always picks the os' language and ignores the lc_message value.

This looks like I can reproduce though, at least on cvs head. Did this
work for you in previous versions?

//Magnus

Re: BUG #4186: set lc_messages does not work

From
Tom Lane
Date:
Magnus Hagander <magnus@hagander.net> writes:
> Thomas H. wrote:
>> so at least that explains the "changed" behaviour. nevertheless,
>> LC_MESSAGES seems to be defunct - with the "locale" folder present,
>> pg always picks the os' language and ignores the lc_message value.

> This looks like I can reproduce though, at least on cvs head. Did this
> work for you in previous versions?

Maybe we were using a different build of gettext in the previous
releases, one that didn't look at the same info as the current code?

Anyway the patch mentioned at the start of the thread
http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
purports to fix this.  It doesn't seem to have gotten reviewed
though.

            regards, tom lane

Re: BUG #4186: set lc_messages does not work

From
Bruce Momjian
Date:
Tom Lane wrote:
> Magnus Hagander <magnus@hagander.net> writes:
> > Thomas H. wrote:
> >> so at least that explains the "changed" behaviour. nevertheless,
> >> LC_MESSAGES seems to be defunct - with the "locale" folder present,
> >> pg always picks the os' language and ignores the lc_message value.
>
> > This looks like I can reproduce though, at least on cvs head. Did this
> > work for you in previous versions?
>
> Maybe we were using a different build of gettext in the previous
> releases, one that didn't look at the same info as the current code?
>
> Anyway the patch mentioned at the start of the thread
> http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
> purports to fix this.  It doesn't seem to have gotten reviewed
> though.

Agreed.  Magnus, someone, can we get feedback on the patch at this URL?

    http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

Re: BUG #4186: set lc_messages does not work

From
Magnus Hagander
Date:
Bruce Momjian wrote:
> Tom Lane wrote:
>> Magnus Hagander <magnus@hagander.net> writes:
>>> Thomas H. wrote:
>>>> so at least that explains the "changed" behaviour. nevertheless,
>>>> LC_MESSAGES seems to be defunct - with the "locale" folder present,
>>>> pg always picks the os' language and ignores the lc_message value.
>>> This looks like I can reproduce though, at least on cvs head. Did this
>>> work for you in previous versions?
>> Maybe we were using a different build of gettext in the previous
>> releases, one that didn't look at the same info as the current code?
>>
>> Anyway the patch mentioned at the start of the thread
>> http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
>> purports to fix this.  It doesn't seem to have gotten reviewed
>> though.
>
> Agreed.  Magnus, someone, can we get feedback on the patch at this URL?
>
>     http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php

IIRC, there was further work to be done on the patch before it was to be
applied, and we held off the review until then.

Gevik - can you comment on this? Where are we, what needs ot be done still?

//Magnus

Re: BUG #4186: set lc_messages does not work

From
"Hiroshi Saito"
Date:
Hi Magnus.

I have tried with Inoue-san, investigation of this problem, and adjustment.
http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/LC_TIME_PATCH/pg8.3.3-to_char_gettext_format.png
Native-strftime was proposed by Tom-san. It corrects(LC_TIME) from 8.3.3.(LC_MESSAGES)
http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/LC_TIME_PATCH/
http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/LC_TIME_PATCH/pg_locale_patch
It has not resulted in the still final conclusion.
However, Possibly this may be the optimal one.
http://winpg.jp/~saito/pg_work/LC_MESSAGE_CHECK/LC_TIME_PATCH/wcsftimeck.c
I am creating the proposal of patch with Inoue-san.

Regards,
Hiroshi Saito

----- Original Message -----
From: "Magnus Hagander" <magnus@hagander.net>


> Bruce Momjian wrote:
>> Tom Lane wrote:
>>> Magnus Hagander <magnus@hagander.net> writes:
>>>> Thomas H. wrote:
>>>>> so at least that explains the "changed" behaviour. nevertheless,
>>>>> LC_MESSAGES seems to be defunct - with the "locale" folder present,
>>>>> pg always picks the os' language and ignores the lc_message value.
>>>> This looks like I can reproduce though, at least on cvs head. Did this
>>>> work for you in previous versions?
>>> Maybe we were using a different build of gettext in the previous
>>> releases, one that didn't look at the same info as the current code?
>>>
>>> Anyway the patch mentioned at the start of the thread
>>> http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
>>> purports to fix this.  It doesn't seem to have gotten reviewed
>>> though.
>>
>> Agreed.  Magnus, someone, can we get feedback on the patch at this URL?
>>
>> http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
>
> IIRC, there was further work to be done on the patch before it was to be
> applied, and we held off the review until then.
>
> Gevik - can you comment on this? Where are we, what needs ot be done still?
>
> //Magnus
>
>
> --
> Sent via pgsql-bugs mailing list (pgsql-bugs@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-bugs

Re: BUG #4186: set lc_messages does not work

From
"Gevik Babakhani"
Date:
My apologies on this late reply.
The way LC_MESSAGES is handled on windows is much less efficient and faulty.
While ago I started with a patch to fix some of the issues I encountered on
windows and LC_MESSAGES. But I stopped working on that patch because this
problem needed to be fixed on many other places. In Windows, handling
LC_MESSAGES will not work the same way as *nix systems, forcing us to make
ugly workarounds. (as I actually was doing with my patch)

To my opinion, unless we think of a coherent solution for handling
LC_MESSAGES/locale for both *nix and win32 platforms, fixing lc_messages and
locale issues would break more than fixing it.

BTW: The gettext library we are using on win32 is a very old one.

Regards,
Gevik.


> -----Original Message-----
> From: Magnus Hagander [mailto:magnus@hagander.net]
> Sent: Tuesday, August 05, 2008 4:54 PM
> To: Bruce Momjian
> Cc: Tom Lane; Thomas H.; pgsql-bugs@postgresql.org; Gevik Babakhani
> Subject: Re: [BUGS] BUG #4186: set lc_messages does not work
>
> Bruce Momjian wrote:
> > Tom Lane wrote:
> >> Magnus Hagander <magnus@hagander.net> writes:
> >>> Thomas H. wrote:
> >>>> so at least that explains the "changed" behaviour. nevertheless,
> >>>> LC_MESSAGES seems to be defunct - with the "locale"
> folder present,
> >>>> pg always picks the os' language and ignores the
> lc_message value.
> >>> This looks like I can reproduce though, at least on cvs head. Did
> >>> this work for you in previous versions?
> >> Maybe we were using a different build of gettext in the previous
> >> releases, one that didn't look at the same info as the
> current code?
> >>
> >> Anyway the patch mentioned at the start of the thread
> >> http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
> >> purports to fix this.  It doesn't seem to have gotten reviewed
> >> though.
> >
> > Agreed.  Magnus, someone, can we get feedback on the patch
> at this URL?
> >
> >
> http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
>
> IIRC, there was further work to be done on the patch before
> it was to be applied, and we held off the review until then.
>
> Gevik - can you comment on this? Where are we, what needs ot
> be done still?
>
> //Magnus
>

Re: BUG #4186: set lc_messages does not work

From
Bruce Momjian
Date:
Tom Lane wrote:
> Magnus Hagander <magnus@hagander.net> writes:
> > Thomas H. wrote:
> >> so at least that explains the "changed" behaviour. nevertheless,
> >> LC_MESSAGES seems to be defunct - with the "locale" folder present,
> >> pg always picks the os' language and ignores the lc_message value.
>
> > This looks like I can reproduce though, at least on cvs head. Did this
> > work for you in previous versions?
>
> Maybe we were using a different build of gettext in the previous
> releases, one that didn't look at the same info as the current code?
>
> Anyway the patch mentioned at the start of the thread
> http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
> purports to fix this.  It doesn't seem to have gotten reviewed
> though.

Where are we on this?

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

Re: BUG #4186: set lc_messages does not work

From
Bruce Momjian
Date:
Gevik Babakhani wrote:
> My apologies on this late reply.
> The way LC_MESSAGES is handled on windows is much less efficient and faulty.
> While ago I started with a patch to fix some of the issues I encountered on
> windows and LC_MESSAGES. But I stopped working on that patch because this
> problem needed to be fixed on many other places. In Windows, handling
> LC_MESSAGES will not work the same way as *nix systems, forcing us to make
> ugly workarounds. (as I actually was doing with my patch)
>
> To my opinion, unless we think of a coherent solution for handling
> LC_MESSAGES/locale for both *nix and win32 platforms, fixing lc_messages and
> locale issues would break more than fixing it.
>
> BTW: The gettext library we are using on win32 is a very old one.

This is the most recent posting on the topic, I think.  Status?

---------------------------------------------------------------------------


>
> Regards,
> Gevik.
>
>
> > -----Original Message-----
> > From: Magnus Hagander [mailto:magnus@hagander.net]
> > Sent: Tuesday, August 05, 2008 4:54 PM
> > To: Bruce Momjian
> > Cc: Tom Lane; Thomas H.; pgsql-bugs@postgresql.org; Gevik Babakhani
> > Subject: Re: [BUGS] BUG #4186: set lc_messages does not work
> >
> > Bruce Momjian wrote:
> > > Tom Lane wrote:
> > >> Magnus Hagander <magnus@hagander.net> writes:
> > >>> Thomas H. wrote:
> > >>>> so at least that explains the "changed" behaviour. nevertheless,
> > >>>> LC_MESSAGES seems to be defunct - with the "locale"
> > folder present,
> > >>>> pg always picks the os' language and ignores the
> > lc_message value.
> > >>> This looks like I can reproduce though, at least on cvs head. Did
> > >>> this work for you in previous versions?
> > >> Maybe we were using a different build of gettext in the previous
> > >> releases, one that didn't look at the same info as the
> > current code?
> > >>
> > >> Anyway the patch mentioned at the start of the thread
> > >> http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
> > >> purports to fix this.  It doesn't seem to have gotten reviewed
> > >> though.
> > >
> > > Agreed.  Magnus, someone, can we get feedback on the patch
> > at this URL?
> > >
> > >
> > http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
> >
> > IIRC, there was further work to be done on the patch before
> > it was to be applied, and we held off the review until then.
> >
> > Gevik - can you comment on this? Where are we, what needs ot
> > be done still?
> >
> > //Magnus
> >

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

Re: BUG #4186: set lc_messages does not work

From
Hiroshi Inoue
Date:
Bruce Momjian wrote:
> Tom Lane wrote:
>> Magnus Hagander <magnus@hagander.net> writes:
>>> Thomas H. wrote:
>>>> so at least that explains the "changed" behaviour. nevertheless,
>>>> LC_MESSAGES seems to be defunct - with the "locale" folder present,
>>>> pg always picks the os' language and ignores the lc_message value.
>>> This looks like I can reproduce though, at least on cvs head. Did this
>>> work for you in previous versions?
>> Maybe we were using a different build of gettext in the previous
>> releases, one that didn't look at the same info as the current code?
>>
>> Anyway the patch mentioned at the start of the thread
>> http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
>> purports to fix this.  It doesn't seem to have gotten reviewed
>> though.
>
> Where are we on this?

Saito-san and I have been working on this item.
The attached patch is for MSVC version and does the following
when built under vc8 or later version MSVC.

1. Accepts Windows or ISO style locale name.
2. _putenv the ISO style locale name as LC_MESSAGES environemnt
    variable so as to be referenced by the gettext library.
    Note that we have to call _putenv() in msvcrt.dll not in
    each MSVC version's runtime library msvcrxx.dll.
3. Calls SetThreadLocale() if necessary.
    The currently used libintl3.dll(0.14.4.1592) seems to need
    this handling. ISTM it is badly built and ignores the setting
    of step 2.

regards,
Hiroshi Inoue





Index: pg_locale.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/utils/adt/pg_locale.c,v
retrieving revision 1.42
diff -c -c -r1.42 pg_locale.c
*** pg_locale.c    23 Sep 2008 09:20:36 -0000    1.42
--- pg_locale.c    7 Dec 2008 15:53:58 -0000
***************
*** 101,112 ****
--- 101,303 ----
   * LC_XXX variables have been set correctly.  (Thank you Perl for making this
   * kluge necessary.)
   */
+
+ #if defined(WIN32) && defined(_MSC_VER)
+ typedef int (_cdecl *PUTENVPROC)(const char *);
+ #if defined(LC_MESSAGES) && (_MSC_VER >= 1400)
+ #define    _TYPE_LOCALE_T_AVAILABLE
+ #include <shlwapi.h>
+ typedef const char * (_cdecl *NLLOCALENAMEPOSIXPROC)(int, const char *);
+
+ /*
+  *    Never use DLLGETVERSIONPROC typedef'd in Shlwapi.h.
+  *    It's problematic and would cause a crash.
+  */
+ typedef HRESULT (_cdecl *DLLGETVERSIONFUNC)(DLLVERSIONINFO *);
+
+ static char get_lang[64] = "";
+ static char get_country[64] = "";
+ static LCID glcid = (LCID) -1;
+
+ static BOOL CALLBACK lclist(LPTSTR lcname)
+ {
+     static char tmp_country[128] = "";
+     DWORD    lcid;
+     char    llang[32], lcountry[32];
+
+     sscanf_s(lcname, "%x", &lcid);
+     GetLocaleInfoA(lcid, LOCALE_SISO639LANGNAME, llang, sizeof(llang));
+     if (0 != _stricmp(llang, get_lang))
+         return TRUE;
+     GetLocaleInfoA(lcid, LOCALE_SISO3166CTRYNAME, lcountry, sizeof(lcountry));
+     if ('\0' == get_country[0])
+     {
+         if (SUBLANG_DEFAULT == SUBLANGID(LANGIDFROMLCID(lcid)))
+         {
+             glcid = lcid;
+             return FALSE;
+         }
+         return TRUE;
+     }
+     if (0 == _stricmp(lcountry, get_country))
+     {
+         glcid = lcid;
+         return FALSE;
+     }
+     return TRUE;
+ }
+ #endif /* defined(LC_MESSAGES) && (_MSC_VER >= 1400) */
+
+ /*
+  *    This function can accept Windows or ISO style locale name.
+  *    Calls SetThreadLocale() if neccesary.
+  *    Sets the ISO style LC_MESSAGES environment variable using
+  *    _putenv() in msvcrt.dll which may be referenced by gettext.
+  *    Returns ISO style LC_MESSAGES locale name.
+  */
+ static char *adjust_LC_MESSAGES(const char *locale, char *envbuf, char *new_locale)
+ {
+ #ifdef    _TYPE_LOCALE_T_AVAILABLE
+     char    *rtn_locale = locale;
+     bool    isClocale = false;
+     LCID    lcid = (LCID) -1;
+     int        usecategory = LC_CTYPE;
+     _locale_t    loct = NULL;
+
+     if (0 == stricmp("c", locale) ||
+         0 == stricmp("posix", locale))
+         isClocale = true;
+     if (isClocale)
+         loct = _create_locale(usecategory, "C");
+     else
+         loct = _create_locale(usecategory, locale);
+
+     if (NULL != loct)
+     {
+         lcid = loct->locinfo->lc_handle[usecategory];
+         if (0 == lcid)
+             lcid = MAKELCID(MAKELANGID(LANG_ENGLISH, SUBLANG_ENGLISH_US), SORT_DEFAULT);
+         _free_locale(loct);
+     }
+     else
+     {
+         char del[16];
+         int scount = sscanf_s(locale, "%[^_-.]%[_-.]%[^.]", get_lang, sizeof(get_lang), del, sizeof(del),
get_country,sizeof(get_country)); 
+         switch (scount)
+         {
+             case 1:
+             case 2:
+                 get_country[0] = '\0';
+                 break;
+             case 3:
+                 if ('.' == del[0])
+                     get_country[0] = '\0';
+                 break;
+         }
+         glcid = (LCID) -1;
+         EnumSystemLocales(lclist, LCID_SUPPORTED);
+         lcid = glcid;
+     }
+     if ((LCID) -1 == lcid)
+         return NULL;
+     else
+     {
+         /*
+          *    Though GNU gettext FAQ says setting the environtment LC_MESSAGES etc variables
+          *    determines the locale used by gettext, it doesn't seem true for 0.14.4.1952
+          *    binary version of libintl3.dll. To make sure we do a test and set not only the
+          *    LC_MESSAGES variable but also the current thread's locale if the test shows that
+          *    the DLL doesn't follow the specification(FAQ).
+          */
+         static    bool    initCheck = true, setThread = false, setEnv = true;
+         HMODULE hModule;
+         DLLGETVERSIONFUNC DllGetVersion;
+
+         if (initCheck &&
+             NULL != (hModule = LoadLibrary("libintl3.dll")) &&
+             NULL != (DllGetVersion = (DLLGETVERSIONFUNC) GetProcAddress(hModule, "DllGetVersion")))
+         {
+             DLLVERSIONINFO2 dvi;
+             HRESULT    hr;
+             NLLOCALENAMEPOSIXPROC locale_name_posix;
+
+             dvi.info1.cbSize = sizeof(dvi);
+             hr = DllGetVersion((DLLVERSIONINFO *) &dvi);
+             if (SUCCEEDED(hr) &&
+                 MAKEDLLVERULL(0, 14, 4, 1952) == dvi.ullVersion &&
+                 NULL != (locale_name_posix = (NLLOCALENAMEPOSIXPROC) GetProcAddress(hModule,
"_nl_locale_name_posix")))
+             {
+
+                 /*
+                  *        Here we test the DLL's behavior.
+                  *
+                  *    Does the function _nl_locale_name_posix() which is
+                  *    internally called by gettext() call setlocale()?
+                   */
+                 if (NULL != locale_name_posix(LC_CTYPE, "ignore_this_value") &&
+                             NULL == locale_name_posix(LC_MESSAGES, "ignore_this_value"))
+                 {
+                     setThread = true;
+                     elog(WARNING, "linking a mal-configured libintl3.dll?");
+                 }
+             }
+         }
+         initCheck = false;
+         if (NULL != hModule)
+             FreeLibrary(hModule);
+         if (setThread)
+         {
+             SetThreadLocale(lcid);
+         }
+         if (setEnv && !isClocale)
+         {
+             char    isolang[64], isocrty[64];
+
+             GetLocaleInfoA(lcid, LOCALE_SISO639LANGNAME, isolang, sizeof(isolang));
+             GetLocaleInfoA(lcid, LOCALE_SISO3166CTRYNAME, isocrty, sizeof(isocrty));
+             snprintf(new_locale, LC_ENV_BUFSIZE - 1, "%s_%s", isolang, isocrty);
+             rtn_locale = new_locale;
+         }
+     }
+ #endif /* _TYPE_LOCALE_T_AVAILABLE */
+ #if (_MSC_VER >= 1300)
+     {
+         /*
+          *    Each MSVC version has its own _putenv() in its runtime library msvcrXX.dll.
+          *    We call _putenv() in msvcrt.dll so as to be referenced by GnuWin32 library.
+          */
+         HMODULE    hmodule;
+         static PUTENVPROC    putenvFunc = NULL;
+
+         if (NULL == putenvFunc)
+         {
+             if (hmodule = GetModuleHandle("msvcrt"), NULL == hmodule)
+                 return NULL;
+             putenvFunc = (PUTENVPROC)GetProcAddress(hmodule, "_putenv");
+         }
+         if (NULL == putenvFunc)
+             return NULL;
+         snprintf(envbuf, LC_ENV_BUFSIZE - 1, "%s=%s", "LC_MESSAGES", rtn_locale);
+         if (putenvFunc(envbuf))
+             return NULL;
+         /* Refresh msgid pool maintained by gettext? */
+         textdomain("postgres");
+     }
+ #endif /* _MSC_VER >= 1300 */
+
+     return rtn_locale;
+ }
+ #endif /* defined(WIN32) && defined(_MSC_VER) */
+
  char *
  pg_perm_setlocale(int category, const char *locale)
  {
      char       *result;
      const char *envvar;
      char       *envbuf;
+ #ifdef    LC_MESSAGES
+     char    newlocale[128];
+ #endif /* LC_MESSAGES */

  #ifndef WIN32
      result = setlocale(category, locale);
***************
*** 147,152 ****
--- 338,347 ----
          case LC_MESSAGES:
              envvar = "LC_MESSAGES";
              envbuf = lc_messages_envbuf;
+ #if defined(WIN32) && defined(_MSC_VER)
+             if (result = adjust_LC_MESSAGES(result, envbuf, newlocale), NULL == result)
+                 return NULL;
+ #endif /* WIN32 */
              break;
  #endif
          case LC_MONETARY:
***************
*** 165,170 ****
--- 360,366 ----
              elog(FATAL, "unrecognized LC category: %d", category);
              envvar = NULL;        /* keep compiler quiet */
              envbuf = NULL;
+             return NULL;
              break;
      }

***************
*** 182,189 ****
      if (!SetEnvironmentVariable(envvar, result))
          return NULL;
      if (_putenv(envbuf))
!         return NULL;
! #endif

      return result;
  }
--- 378,385 ----
      if (!SetEnvironmentVariable(envvar, result))
          return NULL;
      if (_putenv(envbuf))
!         result = NULL;
! #endif /* WIN32 */

      return result;
  }

Re: BUG #4186: set lc_messages does not work

From
"Gevik Babakhani"
Date:
AFAIK we did not discuss this further. LC_MESSAGES and changing locale
functionality for functions like to_date on windows remains broken.

To summarize the problem:

1) On *nix systems, we use lc_messages and lc_locale both for functions
(like to to_date and to_char) and other system messages where locale plays
any role. Lc_messages and lc_locale are read from the environment variables.
Unfortunately on Windows the meaning of these variables is not the same as
on *nix systems, resulting locale assignment behave unexpectedly (broken).

2) On both systems we are using gettext library which is responsible to read
and translate to the correct locale. Gettext on *nix and Windows (and MAC)
also has different implementations for the same functionality. Not to
mention we are using a very old version of gettext on Windows.


3) As I was working on the patch to fix this I encountered places where the
usage of LC_MESSAGES and LC_LOCALE was not clear, especially for functions
where the functionality of Oracle was mimicked.

4) The entire problem came to light when we started to compile PG with MSVC
(8.3+). On 8.2 which was compiled using MinGW (I guess we do not support it
anymore) the behavior of lc_messages and lc_locale was also broken but not
as visible as with MSVC.


5) To give you an example: On *nix, the locale system acts when the value of
lc_messages or lc_locale (environment variables) changes. The expected
values are like: "en_US". On Windows on the other hand, en_US has no
meaning. It becomes something like "English (United States)", which is
ignored by gettext. Leaving us with no alternative than translating the
"English (United States)" to "en_US" and force it to gettext which does not
work all the time because gettext discards the forced value when it reloads
itself.


6) As I mentioned before, I stopped developing a patch for this because this
problem is somewhat deeper than it appears. Any attempt to fix this problem
will also result changing code for both locale aware functions like to_date
and to_char as for when to use lc_locale and lc_messages in the codebase.

Any thoughts?

Regards,
Gevik Babakhani
------------------------------------------------
PostgreSQL NL       http://www.postgresql.nl
TrueSoftware BV     http://www.truesoftware.nl
------------------------------------------------








> -----Original Message-----
> From: Bruce Momjian [mailto:bruce@momjian.us]
> Sent: Sunday, December 07, 2008 4:26 AM
> To: Gevik Babakhani
> Cc: 'Magnus Hagander'; 'Tom Lane'; 'Thomas H.';
> pgsql-bugs@postgresql.org
> Subject: Re: [BUGS] BUG #4186: set lc_messages does not work
>
> Gevik Babakhani wrote:
> > My apologies on this late reply.
> > The way LC_MESSAGES is handled on windows is much less
> efficient and faulty.
> > While ago I started with a patch to fix some of the issues I
> > encountered on windows and LC_MESSAGES. But I stopped
> working on that
> > patch because this problem needed to be fixed on many other
> places. In
> > Windows, handling LC_MESSAGES will not work the same way as *nix
> > systems, forcing us to make ugly workarounds. (as I
> actually was doing
> > with my patch)
> >
> > To my opinion, unless we think of a coherent solution for handling
> > LC_MESSAGES/locale for both *nix and win32 platforms, fixing
> > lc_messages and locale issues would break more than fixing it.
> >
> > BTW: The gettext library we are using on win32 is a very old one.
>
> This is the most recent posting on the topic, I think.  Status?
>
> --------------------------------------------------------------
> -------------
>
>
> >
> > Regards,
> > Gevik.
> >
> >
> > > -----Original Message-----
> > > From: Magnus Hagander [mailto:magnus@hagander.net]
> > > Sent: Tuesday, August 05, 2008 4:54 PM
> > > To: Bruce Momjian
> > > Cc: Tom Lane; Thomas H.; pgsql-bugs@postgresql.org; Gevik
> Babakhani
> > > Subject: Re: [BUGS] BUG #4186: set lc_messages does not work
> > >
> > > Bruce Momjian wrote:
> > > > Tom Lane wrote:
> > > >> Magnus Hagander <magnus@hagander.net> writes:
> > > >>> Thomas H. wrote:
> > > >>>> so at least that explains the "changed" behaviour.
> > > >>>> nevertheless, LC_MESSAGES seems to be defunct - with
> the "locale"
> > > folder present,
> > > >>>> pg always picks the os' language and ignores the
> > > lc_message value.
> > > >>> This looks like I can reproduce though, at least on cvs head.
> > > >>> Did this work for you in previous versions?
> > > >> Maybe we were using a different build of gettext in
> the previous
> > > >> releases, one that didn't look at the same info as the
> > > current code?
> > > >>
> > > >> Anyway the patch mentioned at the start of the thread
> > > >>
> http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
> > > >> purports to fix this.  It doesn't seem to have gotten reviewed
> > > >> though.
> > > >
> > > > Agreed.  Magnus, someone, can we get feedback on the patch
> > > at this URL?
> > > >
> > > >
> > > http://archives.postgresql.org/pgsql-patches/2008-02/msg00038.php
> > >
> > > IIRC, there was further work to be done on the patch
> before it was
> > > to be applied, and we held off the review until then.
> > >
> > > Gevik - can you comment on this? Where are we, what needs
> ot be done
> > > still?
> > >
> > > //Magnus
> > >
>
> --
>   Bruce Momjian  <bruce@momjian.us>        http://momjian.us
>   EnterpriseDB                             http://enterprisedb.com
>
>   + If your life is a hard drive, Christ can be your backup. +
>