Thread: [bug fix] postgres.exe fails to start on Windows Server 2012 due to ASLR

Hello,

My customer reported the following problem on Windows.  I'm afraid this is a
serious problem, and I think we should provide a fix in the next minor
release.  I've attached a fix, and I would wish it to be back-ported.


[Problem]
The customer is using 64-bit PostgreSQL 9.1.x on Windows Server 2012 R2.  He
found the following messages output repeatedly (once per about 3 minutes) in
the event log.

LOG:  could not reserve shared memory region (addr=0000000001270000) for
child 0000000000001340: 487
LOG:  could not fork autovacuum worker process: A blocking operation was
interrupted by a call to WSACancelBlockingCall.

I found the same problem was reported by another user here:

Re: PG 924, Windows 2012, error code 487
http://www.postgresql.org/message-id/2C0926ABD16BB641A8E2F11A549200425471425A21@PHXCCRPRD01.adprod.bmc.com

In my customer's case, the failed process was autovacuum worker, so there
was no visible impact on their applications.  However, as the above mail
suggests, this can happen with any backend process.


[Cause]
This is caused by the improved ASLR (Address Space Layout Randomization) in
Windows 8/2012.  That is analyzed in the following mail last year, but the
problem was not addressed:

windows 8 RTM compatibility issue (could not reserve shared memory region
for child)
http://www.postgresql.org/message-id/5046CAEB.4010600@grammatech.com


[Fix]
Disable ASLR when building PostgreSQL modules.  This eliminated the log
output in my customer's environment.

I'll add this to the CommitFest if necessary.

Regards
MauMau

On 01/24/2014 06:42 PM, MauMau wrote:
> Hello,
> 
> My customer reported the following problem on Windows.  I'm afraid this
> is a serious problem, and I think we should provide a fix in the next
> minor release.  I've attached a fix, and I would wish it to be back-ported.
> 
> 
> [Problem]
> The customer is using 64-bit PostgreSQL 9.1.x

Which "x"?

Does this issue also occur on 9.3.2, or in 9.4 HEAD, when tested on Win2k12?

-- Craig Ringer                   http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services

I'm sorry for the late reply.  I was unable to access email.

From: "Craig Ringer" <craig@2ndquadrant.com>
> On 01/24/2014 06:42 PM, MauMau wrote:
>> The customer is using 64-bit PostgreSQL 9.1.x
>
> Which "x"?

9.1.6.

> Does this issue also occur on 9.3.2, or in 9.4 HEAD, when tested on 
> Win2k12?

I'm sure it should.  The release note doesn't have any reference to this 
issue.  Another user who reported this issue in pgsql-general experienced 
this with 9.2.4.

In addition, Another customer reported to me that psql failed to connect to 
PostgreSQL 9.1.9 running on Windows Server 2012 R2, about once out of 7 
times.  I believe this needs immediate fix.

Regards
MauMau

On 01/30/2014 08:14 PM, MauMau wrote:
>> Does this issue also occur on 9.3.2, or in 9.4 HEAD, when tested on
>> Win2k12?
> 
> I'm sure it should.  The release note doesn't have any reference to this
> issue.  Another user who reported this issue in pgsql-general
> experienced this with 9.2.4.

> In addition, Another customer reported to me that psql failed to connect
> to PostgreSQL 9.1.9 running on Windows Server 2012 R2, about once out of
> 7 times.  I believe this needs immediate fix.

I'm reasonably persuaded that there's a need for this, though IFEO (see
below) can be used at or after install-time as a workaround.

It looks like your patch sets the msbuild equivalent of the
/DYNAMICBASE:NO flag
(http://msdn.microsoft.com/en-us/library/bb384887.aspx). Is this known
to work (or be silently ignored) back to Windows SDK 7.1? (I'll test it
today and see).

I don't think we need to worry about Force ASLR
(http://support.microsoft.com/kb/2639308) as it seems it only applies
when an application loads modules, unless an admin goes playing in the
registry.


Users facing this problem can work around it without code changes by
setting IFEO in the registry to disable ASLR for postgres.exe. See:

http://msdn.microsoft.com/en-us/library/windows/desktop/ff190925(v=vs.85).aspx

http://support.microsoft.com/kb/2639308

http://msdn.microsoft.com/en-us/library/bb430720.aspx

It may be reasonable for EDB to consider releasing updated installers
that set IFEO flags to disable ASLR on postgres.exe .

-- Craig Ringer                   http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services

From: "Craig Ringer" <craig@2ndquadrant.com>
> I'm reasonably persuaded that there's a need for this, though IFEO (see
> below) can be used at or after install-time as a workaround.

I see.  And I also found it effective as another workaround to set the below 
registry key.  This disables ASLR for all executables, so it would be 
disliked.
   HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory 
Management\MoveImages


> It looks like your patch sets the msbuild equivalent of the
> /DYNAMICBASE:NO flag
> (http://msdn.microsoft.com/en-us/library/bb384887.aspx). Is this known
> to work (or be silently ignored) back to Windows SDK 7.1? (I'll test it
> today and see).

OK, I tried these versions at hand:

* Windows SDK 7.1, which happens to be associated with Visual Studio 2010 on 
my PC
link.exe accepts /dynamicbase:no, which takes effect.  Without 
/dynamicbase:no, dumpbin /headers displays the lines:
           8140 DLL characteristics                  Dynamic base                  NX compatible
TerminalServer Aware
 

On the other hand, with /dynamicbase:no, the second line above disappeared.

* Visual Studio 2008 Express
link.exe seems to silently ignore /dynamicbase:no.  dumpbin /headers does 
not display "Dynamic base" regardless of whether /dynamicbase:no is 
specified.


> I don't think we need to worry about Force ASLR
> (http://support.microsoft.com/kb/2639308) as it seems it only applies
> when an application loads modules, unless an admin goes playing in the
> registry.
>
>
> Users facing this problem can work around it without code changes by
> setting IFEO in the registry to disable ASLR for postgres.exe. See:

The problem is that users cannot discover the workaround.  As more users use 
PostgreSQL on Windows 8/2012, similar questions would be asked in 
pgsql-general and pgsql-bugs.

> It may be reasonable for EDB to consider releasing updated installers
> that set IFEO flags to disable ASLR on postgres.exe .

Not all users use PostgreSQL built by EnterpriseDB, so I think 
src\tools\msvc\build.bat should produce modules that are not affected by 
ASLR.

Regards
MauMau

On 02/01/2014 09:53 PM, MauMau wrote:
> 
> Not all users use PostgreSQL built by EnterpriseDB, so I think
> src\tools\msvc\build.bat should produce modules that are not affected by
> ASLR.

I completely agree; just saying that any installer can set the key.

I'm convinced that setting this flag is appropriate, at least while Pg
relies on having the shared memory segment mapped in the same zone in
every executable. Just pointing out that there's a workaround in the
mean time.

-- Craig Ringer                   http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services

From: "Craig Ringer" <craig@2ndquadrant.com>
> I completely agree; just saying that any installer can set the key.
>
> I'm convinced that setting this flag is appropriate, at least while Pg
> relies on having the shared memory segment mapped in the same zone in
> every executable. Just pointing out that there's a workaround in the
> mean time.

Please don't mind, I didn't misunderstand your intent.  I think we should 
apply this in the next minor release to avoid unnecessary confusion -- more 
new users would use PostgreSQL on Windows 8/2012 and hit this problem.

I added this patch to the CommitFest.

Regards
MauMau

On 02/04/2014 07:28 PM, MauMau wrote:
>>
> 
> Please don't mind, I didn't misunderstand your intent.  I think we
> should apply this in the next minor release to avoid unnecessary
> confusion -- more new users would use PostgreSQL on Windows 8/2012 and
> hit this problem.
> 
> I added this patch to the CommitFest.

It's really a bugfix suitable for backpatching IMO.

-- Craig Ringer                   http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services

Magnus Hagander <magnus@hagander.net> writes:
> I've applied this and backpatched to 9.3 and 9.2, which is as far back as
> it goes cleanly.

> In 9.1 the build system looked significantly different, which makes it
> strange since the original report in this thread was about 9.1 but the
> patch supplied clearly wasn't for that.

> Does somebody want to look at backpatching this to 9.1 and earlier, or
> should we just say that it's not fully supported on those Windows versions
> unless you apply the registry workaround?

The question I think needs to be asked is not that, but "what about the
mingw and cygwin builds?".
        regards, tom lane

From: "Magnus Hagander" <magnus@hagander.net>
> Unfortunately we missed the releases that have just been wrapped.

It's really unfortunate... I hope the next minor release will be soon.

> I've applied this and backpatched to 9.3 and 9.2, which is as far back as
> it goes cleanly.

Thank you.


> In 9.1 the build system looked significantly different, which makes it
> strange since the original report in this thread was about 9.1 but the
> patch supplied clearly wasn't for that.
>
> Does somebody want to look at backpatching this to 9.1 and earlier, or
> should we just say that it's not fully supported on those Windows versions
> unless you apply the registry workaround?

I'll try it at least for 9.1 in a few days when time permits.  The fix 
should be adding only one line in Project.pm.

Regards
MauMau

From: "Magnus Hagander" <magnus@hagander.net>
> Does somebody want to look at backpatching this to 9.1 and earlier, or
> should we just say that it's not fully supported on those Windows versions
> unless you apply the registry workaround?

Please use the attached patch.  It applies cleanly to both 9.1 and 9.0.

We don't need to consider 8.4, because ASLR became enabled by default in
Visual Studio 2008 and 8.4 doesn't support building with 2008.

I tested with Visual Studio 2008 Express.  You can check if ASLR is disabled
using dumpbin.  "dumpbin /headers <EXE_or_DLL_file>" outputs the following
lines.  If the "Dynamic base" line is displayed, ASLR is enabled.  If this
line disappears, ASLR is disabled.

            8140 DLL characteristics
                   Dynamic base
                   NX compatible
                   Terminal Server Aware

Regards
MauMau