Thread: BUG #15772: Some messages in log files are in ANSI encoding while server encoding is UTF8

The following bug has been logged on the website:

Bug reference:      15772
Logged by:          Eugene Podshivalov
Email address:      yaugenka@gmail.com
PostgreSQL version: 11.2
Operating system:   Windows 10
Description:

My postgresql.conf has the following locale settings
----
#client_encoding = sql_ascii        # actually, defaults to database encoding

# These settings are initialized by initdb, but they can be changed.
lc_messages = 'Russian_Russia.1251'            # locale for system error message
strings
lc_monetary = 'Russian_Russia.1251'            # locale for monetary formatting
lc_numeric = 'Russian_Russia.1251'            # locale for number formatting
lc_time = 'Russian_Russia.1251'                # locale for time formatting
----
Server encoding is "UTF8".
Messages in the log file are usually in UTF8, but some messages are logged
in ANSI encoding.
Here are some example cases (in the Russian language)  when ANSI is used
instead of UTF8
--
СООБЩЕНИЕ:  контрольные точки происходят слишком часто (через 19 сек.)
ПОДСКАЗКА:  Возможно, стоит увеличить параметр "max_wal_size".
--
СООБЩЕНИЕ:  получен запрос на быстрое выключение
СООБЩЕНИЕ:  прерывание всех активных транзакций
--
СООБЩЕНИЕ:  система БД была выключена:
СООБЩЕНИЕ:  система БД готова принимать подключения

On Thu, Apr 18, 2019 at 01:53:18PM +0000, PG Bug reporting form wrote:
> The following bug has been logged on the website:
> 
> Bug reference:      15772
> Logged by:          Eugene Podshivalov
> Email address:      yaugenka@gmail.com
> PostgreSQL version: 11.2
> Operating system:   Windows 10
> Description:        
> 
> My postgresql.conf has the following locale settings
> ----
> #client_encoding = sql_ascii        # actually, defaults to database encoding
> 
> # These settings are initialized by initdb, but they can be changed.
> lc_messages = 'Russian_Russia.1251'            # locale for system error message
> strings
> lc_monetary = 'Russian_Russia.1251'            # locale for monetary formatting
> lc_numeric = 'Russian_Russia.1251'            # locale for number formatting
> lc_time = 'Russian_Russia.1251'                # locale for time formatting
> ----
> Server encoding is "UTF8".
> Messages in the log file are usually in UTF8, but some messages are logged
> in ANSI encoding.
> Here are some example cases (in the Russian language)  when ANSI is used
> instead of UTF8
> --
> СООБЩЕНИЕ:  контрольные точки происходят слишком часто (через 19 сек.)
> ПОДСКАЗКА:  Возможно, стоит увеличить параметр "max_wal_size".
> --
> СООБЩЕНИЕ:  получен запрос на быстрое выключение
> СООБЩЕНИЕ:  прерывание всех активных транзакций
> --
> СООБЩЕНИЕ:  система БД была выключена:
> СООБЩЕНИЕ:  система БД готова принимать подключения

I am kind of confused since all the messages look like Russian to me,
except for the mention of "max_wal_size".  When you say ANSI, do you
mean ISO-8859-5 - Cyrillic, or ASCII?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On Thu, Apr 18, 2019 at 05:40:59PM +0300, Eugene Podshivalov wrote:
> Bruce,
> Here is a screenshot of how looks like when I open the log file in notepad++
> and switch encoding from UTF8 to ANSI.
> image.png

Uh, I see what you mean.  Can you give us a message that is OK and one
that is messed up, but the English versions of those?   I still don't
know what ANSI is?  What does the output look like in UTF8 mode?

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +

On 2019-Apr-18, Eugene Podshivalov wrote:

> Bruce,
> Here is a screenshot of how looks like when I open the log file in
> notepad++ and switch encoding from UTF8 to ANSI.
> [image: image.png]

I suppose you have databases with the single-byte encoding amidst your
UTF8 ones.  AFAIK the log file registers the log entries in the same
encoding that the database uses.  Different databases can use different
encodings.

That's pretty broken, but it's how it is.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> I suppose you have databases with the single-byte encoding amidst your
> UTF8 ones.  AFAIK the log file registers the log entries in the same
> encoding that the database uses.  Different databases can use different
> encodings.

> That's pretty broken, but it's how it is.

Yeah, and it's not easy to improve on.  If we tried to convert all
log messages to the same encoding, which one would that be?
(Please, no nonsense about UTF8 being a universal solution.
The Japanese don't think so, for instance.)

Also, what do you do if you get an encoding conversion failure?

That's even before you get into implementation-dependent problems,
like what to do early in process startup before the encoding
conversion machinery is operational.

A more realistic idea might be to have separate log files for
different encodings, though that has a bunch of management issues
to solve as well.

            regards, tom lane