Thread: BUG #13427: postgres.exe fails to start on Korean Windows Server 2008: cannot perform encoding conversion outsid

The following bug has been logged on the website:

Bug reference:      13427
Logged by:          Dmitri Bourlatchkov
Email address:      Dmitri.Bourlatchkov@software.dell.com
PostgreSQL version: 9.4.3
Operating system:   Windows Server 2008 (Korean)
Description:

0. Log in as a local Administrator into a Korean version of the Windows
Server 2008
1. Install PostgreSQL 9.4.3 using the Windows .exe installer from
EnterpriseDB
2. Open cmd.exe shell
3. cd to potgresql/bin directory
4. Run postgresql.exe (no args)
5. Observe the following error message:

----
FATAL:  XX000: cannot perform encoding conversion outside a transaction
LOCATION:  pg_do_encoding_conversion, src\backend\utils\mb\mbutils.c:360
----

A similar message shows up when using pg_ctl.exe to start PostgreSQL.

Trouble-shooting attempted:
1. Run set LC_CTYPE=C
2. Run postgresql.exe (no args)
3. The following expected error shows up:

----
Execution of PostgreSQL by a user with administrative permissions is not
permitted.
The server must be started under an unprivileged user ID to prevent
possible system security compromises.  See the documentation for
more information on how to properly start the server.
----

Note: PostgreSQL _is_ able to run as a Windows Service without any
environment modifications.

Note: If the LC_CTYPE=C env. variable is set, PosgreSQL can also be started
successfully via pg_ctl (under the Administrator user).

Thanks,

  Dmitri.

On Wed, Jun 10, 2015 at 02:32:35PM +0000, Dmitri.Bourlatchkov@software.dell.com wrote:
> PostgreSQL version: 9.4.3
> Operating system:   Windows Server 2008 (Korean)

> 0. Log in as a local Administrator into a Korean version of the Windows
> Server 2008
> 1. Install PostgreSQL 9.4.3 using the Windows .exe installer from
> EnterpriseDB
> 2. Open cmd.exe shell
> 3. cd to potgresql/bin directory
> 4. Run postgresql.exe (no args)
> 5. Observe the following error message:
>
> ----
> FATAL:  XX000: cannot perform encoding conversion outside a transaction
> LOCATION:  pg_do_encoding_conversion, src\backend\utils\mb\mbutils.c:360
> ----
>
> A similar message shows up when using pg_ctl.exe to start PostgreSQL.

Thanks for this report.  One can reproduce this without Korean-specific OS
configuration by setting LC_CTYPE=Korean in the environment and then, as you
say, running postgres without arguments.  Stack trace:

    postgres.exe!errfinish(int dummy, ...) Line 412
    postgres.exe!elog_finish(int elevel, const char * fmt, ...) Line 1376
    postgres.exe!pg_do_encoding_conversion(unsigned char * src, int len, int src_encoding, int dest_encoding) Line 362
    postgres.exe!pgwin32_message_to_UTF16(const char * str, int len, int * utf16len) Line 1081
    postgres.exe!write_console(const char * line, int len) Line 2111
    postgres.exe!write_stderr(const char * fmt, ...) Line 3592
    postgres.exe!SelectConfigFiles(const char * userDoption, const char * progname) Line 4297
    postgres.exe!PostmasterMain(int argc, char * * argv) Line 802

Commit 49c817e is the proximate cause.  On my Windows Server 2008 system,
Korean is the only locale affected.  It's affected because pg_enc2name_tbl
lacks a UHC <-> CP949 mapping.  I'll double check, but adding that mapping
should fix this particular test case.

That leaves other ways of reaching this error.  I think e.g. an LC_CTYPE=C,
ENCODING=LATIN9 database will still take this path.  Any EmitErrorReport() can
run pgwin32_message_to_UTF16(), and those have plenty of ways to run outside a
transaction.  I am inclined to restore the 9.3 and earlier semantics by having
pgwin32_message_to_UTF16() proceed, when outside a transaction, as though the
string is already UTF8.  A robust fix would be to cache enough information
about the message encoding to convert outside a transaction, exactly the way
we protect database<->client encoding conversions.  That feels more like a
master-only change, though it's a borderline case.

> Trouble-shooting attempted:
> 1. Run set LC_CTYPE=C
> 2. Run postgresql.exe (no args)
> 3. The following expected error shows up:
>
> ----
> Execution of PostgreSQL by a user with administrative permissions is not
> permitted.
> The server must be started under an unprivileged user ID to prevent
> possible system security compromises.  See the documentation for
> more information on how to properly start the server.
> ----
>
> Note: PostgreSQL _is_ able to run as a Windows Service without any
> environment modifications.
>
> Note: If the LC_CTYPE=C env. variable is set, PosgreSQL can also be started
> successfully via pg_ctl (under the Administrator user).

That looks normal.  pg_ctl has special code in CreateRestrictedProcess() to
drop administrative privileges.  To run raw postgres without pg_ctl, use an
unprivileged account or drop privileges similarly in your own wrapper.

On Tue, Aug 04, 2015 at 11:32:11PM -0400, Noah Misch wrote:
> On Wed, Jun 10, 2015 at 02:32:35PM +0000, Dmitri.Bourlatchkov@software.dell.com wrote:
> > 4. Run postgresql.exe (no args)
> > 5. Observe the following error message:
> >
> > ----
> > FATAL:  XX000: cannot perform encoding conversion outside a transaction
> > LOCATION:  pg_do_encoding_conversion, src\backend\utils\mb\mbutils.c:360
> > ----

> Commit 49c817e is the proximate cause.  On my Windows Server 2008 system,
> Korean is the only locale affected.  It's affected because pg_enc2name_tbl
> lacks a UHC <-> CP949 mapping.  I'll double check, but adding that mapping
> should fix this particular test case.
>
> That leaves other ways of reaching this error.  I think e.g. an LC_CTYPE=C,
> ENCODING=LATIN9 database will still take this path.

LATIN9 does not, but LATIN8 does.  Plain 'make check' fails early in Korean
locales, and 'make check EXTRA_REGRESS_OPTS="--encoding=LATIN8 --no-locale"'
fails in any OS locale.

> Any EmitErrorReport() can
> run pgwin32_message_to_UTF16(), and those have plenty of ways to run outside a
> transaction.  I am inclined to restore the 9.3 and earlier semantics by having
> pgwin32_message_to_UTF16() proceed, when outside a transaction, as though the
> string is already UTF8.

I am attaching the two patches I plan to use.  Either one by itself fixes the
symptom you observed.

> A robust fix would be to cache enough information
> about the message encoding to convert outside a transaction, exactly the way
> we protect database<->client encoding conversions.  That feels more like a
> master-only change, though it's a borderline case.

I propose to add this to https://wiki.postgresql.org/wiki/Todo:

 * Multi-Language Support
 ** Windows: Cache MessageEncoding conversion for use outside transactions

Thanks,
nm