Thread: Postgres Windows build system doesn't work with python installed inProgram Files

Collegues,

Accidently I've come over minor bug in the Mkvcbuild.pm.
It happens, that it doesn't tolerate spaces in the $config->{python}
path, because it want to call python in order to find out version,
prefix and so on, and doesn't properly quote command.

Fix is very simple, see attach.

Patch is made against REL_12_STABLE, but probably applicable to other
versions as well.
--

Attachment
On Thu, Apr 30, 2020 at 03:06:08PM +0300, Victor Wagner wrote:
> Fix is very simple, see attach.
>
> Patch is made against REL_12_STABLE, but probably applicable to other
> versions as well.

Indeed, thanks.

>          my $pythonprog = "import sys;print(sys.prefix);"
>            . "print(str(sys.version_info[0])+str(sys.version_info[1]))";
>          my $prefixcmd =
> -          $solution->{options}->{python} . "\\python -c \"$pythonprog\"";
> +        '"' . $solution->{options}->{python} . "\\python\" -c \"$pythonprog\"";
>          my $pyout = `$prefixcmd`;
>          die "Could not query for python version!\n" if $?;
>          my ($pyprefix, $pyver) = split(/\r?\n/, $pyout);

This reminds me of ad7595b.  Wouldn't it be better to use qq() here?
--
Michael

Attachment
В Fri, 1 May 2020 17:52:15 +0900
Michael Paquier <michael@paquier.xyz> пишет:

> On Thu, Apr 30, 2020 at 03:06:08PM +0300, Victor Wagner wrote:
> > Fix is very simple, see attach.
> >
> > Patch is made against REL_12_STABLE, but probably applicable to
> > other versions as well.
>
> Indeed, thanks.
>
> >          my $pythonprog = "import sys;print(sys.prefix);"
> >            .
> > "print(str(sys.version_info[0])+str(sys.version_info[1]))"; my
> > $prefixcmd =
> > -          $solution->{options}->{python} . "\\python -c
> > \"$pythonprog\"";
> > +        '"' . $solution->{options}->{python} . "\\python\"
> > -c \"$pythonprog\""; my $pyout = `$prefixcmd`;
> >          die "Could not query for python version!\n" if $?;
> >          my ($pyprefix, $pyver) = split(/\r?\n/, $pyout);
>
> This reminds me of ad7595b.  Wouldn't it be better to use qq() here?

Maybe. But probably original author of this code was afraid of using
too long chain of ->{} in the string substitution.

So, I left this style n place.

Nonetheless, using qq wouldn't save us from doubling backslashes.

--



Em qui., 30 de abr. de 2020 às 09:06, Victor Wagner <vitus@wagner.pp.ru> escreveu:

Collegues,

Accidently I've come over minor bug in the Mkvcbuild.pm.
It happens, that it doesn't tolerate spaces in the $config->{python}
path, because it want to call python in order to find out version,
prefix and so on, and doesn't properly quote command.

Fix is very simple, see attach.

Patch is made against REL_12_STABLE, but probably applicable to other
versions as well.

I don't know if it applies to the same case, but from the moment I installed python on the development machine, the Postgres build stopped working correctly.
Although perl, flex and bison are available in the path, the build does not generate files that depend on flex and bison.

Running bison on src/backend/bootstrap/bootparse.y
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  Running flex on src/backend/bootstrap/bootscanner.l
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  Running bison on src/backend/parser/gram.y
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
etc

Warning from build.pl
Use of uninitialized value $ARGV[0] in uc at build.pl line 44.
Use of uninitialized value $ARGV[0] in uc at build.pl line 48.

regards,
Ranier Vilela
On Sun, May 03, 2020 at 04:23:24PM -0300, Ranier Vilela wrote:
> I don't know if it applies to the same case, but from the moment I
> installed python on the development machine, the Postgres build stopped
> working correctly.
> Although perl, flex and bison are available in the path, the build does not
> generate files that depend on flex and bison.

Are you following the instructions of the documentation?  Here is a
link to them:
https://www.postgresql.org/docs/devel/install-windows-full.html

My guess is that you would be just missing a PATH configuration or
such because python enforced a new setting?

> Warning from build.pl
> Use of uninitialized value $ARGV[0] in uc at build.pl line 44.
> Use of uninitialized value $ARGV[0] in uc at build.pl line 48.

Hmm.  We have buildfarm machines using the MSVC scripts and Python,
see for example woodloose.  And note that @ARGV would be normally
defined, so your warning looks fishy to me.
--
Michael

Attachment

Re: Postgres Windows build system doesn't work with python installedin Program Files

From
Juan José Santamaría Flecha
Date:


On Mon, May 4, 2020 at 7:58 AM Michael Paquier <michael@paquier.xyz> wrote:

> Warning from build.pl
> Use of uninitialized value $ARGV[0] in uc at build.pl line 44.
> Use of uninitialized value $ARGV[0] in uc at build.pl line 48.

Hmm.  We have buildfarm machines using the MSVC scripts and Python,
see for example woodloose.  And note that @ARGV would be normally
defined, so your warning looks fishy to me.

I think these are two different issues, python PATH and build.pl warnings. For the later, you can check woodloose logs and see the warning after commit 8f00d84afc.

Regards,

Juan José Santamaría Flecha 
 
Em seg., 4 de mai. de 2020 às 02:58, Michael Paquier <michael@paquier.xyz> escreveu:
On Sun, May 03, 2020 at 04:23:24PM -0300, Ranier Vilela wrote:
> I don't know if it applies to the same case, but from the moment I
> installed python on the development machine, the Postgres build stopped
> working correctly.
> Although perl, flex and bison are available in the path, the build does not
> generate files that depend on flex and bison.

Are you following the instructions of the documentation?  Here is a
link to them:
https://www.postgresql.org/docs/devel/install-windows-full.html
My environment was ok and worked 100%, compiling with msvc 2019 (64 bits).
 


My guess is that you would be just missing a PATH configuration or
such because python enforced a new setting?
Perl and flex and bison, are in the path, no doubt.
 

> Warning from build.pl
> Use of uninitialized value $ARGV[0] in uc at build.pl line 44.
> Use of uninitialized value $ARGV[0] in uc at build.pl line 48.

Hmm.  We have buildfarm machines using the MSVC scripts and Python,
see for example woodloose.  And note that @ARGV would be normally
defined, so your warning looks fishy to me.
I'll redo from the beginning.
1. Make empty directory postgres
2. Clone postgres
3. Call msvc 2019 (64 bits) env batch
4. Setup path to perl, bison and flex
    set path=%path%;c:\perl;\bin;c:\bin

5. C:\dll>perl -V
Summary of my perl5 (revision 5 version 30 subversion 1) configuration:

  Platform:
    osname=MSWin32
    osvers=10.0.18363.476
    archname=MSWin32-x64-multi-thread
    uname='Win32 strawberry-perl 5.30.1.1 #1 Fri Nov 22 02:24:29 2019 x64'

6. C:\dll>bison -V
bison (GNU Bison) 2.7
Written by Robert Corbett and Richard Stallman.

7. C:\dll>flex -V
flex 2.6.4

8. cd\dll\postgres\src\tools\msvc
9. build

results:
...
PrepareForBuild:
  Creating directory ".\Release\postgres\".
  Creating directory ".\Release\postgres\postgres.tlog\".
InitializeBuildStatus:
  Creating ".\Release\postgres\postgres.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
CustomBuild:
  Running bison on src/backend/bootstrap/bootparse.y
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  Running flex on src/backend/bootstrap/bootscanner.l
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  Running bison on src/backend/parser/gram.y
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  Running flex on src/backend/parser/scan.l
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  Running bison on src/backend/replication/repl_gram.y
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  Running flex on src/backend/replication/repl_scanner.l
  Running bison on src/backend/replication/syncrep_gram.y
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  Running flex on src/backend/replication/syncrep_scanner.l
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  Running bison on src/backend/utils/adt/jsonpath_gram.y
  Running flex on src/backend/utils/adt/jsonpath_scan.l
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
  Running flex on src/backend/utils/misc/guc-file.l
  'perl' nao é reconhecido como um comando interno
  ou externo, um programa operável ou um arquivo em lotes.
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5): error MSB6006: "cmd.exe" exited with code 9009.
 [C:\dll\postgres\postgres.vcxproj]
Done Building Project "C:\dll\postgres\postgres.vcxproj" (default targets) -- FAILED.

Done Building Project "C:\dll\postgres\pgsql.sln" (default targets) -- FAILED.

...

Build FAILED.

"C:\dll\postgres\pgsql.sln" (default target) (1) ->
"C:\dll\postgres\cyrillic_and_mic.vcxproj" (default target) (2) ->
"C:\dll\postgres\postgres.vcxproj" (default target) (3) ->
(CustomBuild target) ->
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5): error MSB6006: "cmd.exe" exited with code 900
9. [C:\dll\postgres\postgres.vcxproj]


"C:\dll\postgres\pgsql.sln" (default target) (1) ->
"C:\dll\postgres\initdb.vcxproj" (default target) (32) ->
"C:\dll\postgres\libpgfeutils.vcxproj" (default target) (33) ->
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5): error MSB6006: "cmd.exe" exited with code 900
9. [C:\dll\postgres\libpgfeutils.vcxproj]


"C:\dll\postgres\pgsql.sln" (default target) (1) ->
"C:\dll\postgres\ecpg.vcxproj" (default target) (124) ->
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5): error MSB6006: "cmd.exe" exited with code 900
9. [C:\dll\postgres\ecpg.vcxproj]


"C:\dll\postgres\pgsql.sln" (default target) (1) ->
"C:\dll\postgres\isolationtester.vcxproj" (default target) (129) ->
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5): error MSB6006: "cmd.exe" exited with code 900
9. [C:\dll\postgres\isolationtester.vcxproj]

    0 Warning(s)
    4 Error(s)

No warnings, this time.

best regards,
Ranier Vilela
On Mon, May 04, 2020 at 09:45:54AM +0200, Juan José Santamaría Flecha wrote:
> I think these are two different issues, python PATH and build.pl warnings.
> For the later, you can check woodloose logs and see the warning after
> commit 8f00d84afc.

Oh, indeed.  I somewhat managed to miss these in the logs of the
buildfarm.  What if we refactored the code of build.pl so as we'd
check first if $ARGV[0] is defined or not?  If not defined, then we
need to have a release-quality build for all the components.  How does
that sound?  Something not documented is that using "release" as first
argument enforces also a release-quality build for all the components,
so we had better not break that part.
--
Michael

Attachment

Re: Postgres Windows build system doesn't work with python installedin Program Files

From
Juan José Santamaría Flecha
Date:

On Mon, May 4, 2020 at 2:18 PM Michael Paquier <michael@paquier.xyz> wrote:
On Mon, May 04, 2020 at 09:45:54AM +0200, Juan José Santamaría Flecha wrote:
> I think these are two different issues, python PATH and build.pl warnings.
> For the later, you can check woodloose logs and see the warning after
> commit 8f00d84afc.

Oh, indeed.  I somewhat managed to miss these in the logs of the
buildfarm.  What if we refactored the code of build.pl so as we'd
check first if $ARGV[0] is defined or not?  If not defined, then we
need to have a release-quality build for all the components.  How does
that sound?  Something not documented is that using "release" as first
argument enforces also a release-quality build for all the components,
so we had better not break that part.

+1, seems like the way to go to me.

Regards,

Juan José Santamaría Flecha
On Fri, May 01, 2020 at 12:48:17PM +0300, Victor Wagner wrote:
> Maybe. But probably original author of this code was afraid of using
> too long chain of ->{} in the string substitution.
>
> So, I left this style n place.
>
> Nonetheless, using qq wouldn't save us from doubling backslashes.

Looking at this part in more details, I find the attached much more
readable.  I have been able to test it on my own Windows environment
and the problem gets fixed (I have reproduced the original problem as
well).
--
Michael

Attachment
В Tue, 5 May 2020 15:45:48 +0900
Michael Paquier <michael@paquier.xyz> пишет:

> On Fri, May 01, 2020 at 12:48:17PM +0300, Victor Wagner wrote:
> > Maybe. But probably original author of this code was afraid of using
> > too long chain of ->{} in the string substitution.
> >
> > So, I left this style n place.
> >
> > Nonetheless, using qq wouldn't save us from doubling backslashes.
>
> Looking at this part in more details, I find the attached much more
> readable.  I have been able to test it on my own Windows environment

I agree, it is better.


> and the problem gets fixed (I have reproduced the original problem as
> well).
--



On Mon, May 04, 2020 at 03:42:20PM +0200, Juan José Santamaría Flecha wrote:
> +1, seems like the way to go to me.

Attached is a patch for that and I have gone with a simple solution,
with some bonus comments about the way things happen.  Here are the
patterns I tested for build.pl and the commands it generates, making
sure that we have the same commands with HEAD and the patch:
1) perl build.pl
msbuild pgsql.sln /verbosity:normal  /p:Configuration=Release
2) perl build.pl debug
msbuild pgsql.sln /verbosity:normal  /p:Configuration=Debug
3) perl build.pl release
msbuild pgsql.sln /verbosity:normal  /p:Configuration=Release
4) perl build.pl foo
msbuild foo.vcxproj /verbosity:normal  /p:Configuration=Release
5) perl build.pl debug foo
msbuild foo.vcxproj /verbosity:normal  /p:Configuration=Debug
6) perl build.pl release foo
msbuild foo.vcxproj /verbosity:normal  /p:Configuration=Release

The two warnings show up only in the first case, of course.
--
Michael

Attachment

Re: Postgres Windows build system doesn't work with python installedin Program Files

From
Juan José Santamaría Flecha
Date:

On Tue, May 5, 2020 at 4:06 PM Michael Paquier <michael@paquier.xyz> wrote:
On Mon, May 04, 2020 at 03:42:20PM +0200, Juan José Santamaría Flecha wrote:
> +1, seems like the way to go to me.

Attached is a patch for that and I have gone with a simple solution,
with some bonus comments about the way things happen. 

This solves the issue.

Please forgive me if I am being too nitpicky, but I find the comments a little too verbose, a usage format might be more visual and easier to explain:

Usage: build [[CONFIGURATION] COMPONENT]

The options are  case-insensitive.
CONFIGURATION sets the configuration to build, "debug" or "release" (by default).
COMPONENT defines a component to build. An empty option means all components.

Regards,

Juan José Santamaría Flecha
On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:
> I agree, it is better.

Thanks, applied and back-patched down to 9.5.  Now for the second
problem of this thread..
--
Michael

Attachment
Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier <michael@paquier.xyz> escreveu:
On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:
> I agree, it is better.

Thanks, applied and back-patched down to 9.5.  Now for the second
problem of this thread..
Sorry, no clue yet.
I hacked the perl sources, to hardcoded perl, bison and flex with path.It works like this.
For some reason, which I haven't yet discovered, msbuild is ignoring the path, where perl and bison and flex are.
Although it is being set, within the 64-bit compilation environment of msvc 2019.
I'm still investigating.

regards,
Ranier Vilela
Em qua., 6 de mai. de 2020 às 10:21, Ranier Vilela <ranier.vf@gmail.com> escreveu:
Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier <michael@paquier.xyz> escreveu:
On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:
> I agree, it is better.

Thanks, applied and back-patched down to 9.5.  Now for the second
problem of this thread..
Sorry, no clue yet.
I hacked the perl sources, to hardcoded perl, bison and flex with path.It works like this.
For some reason, which I haven't yet discovered, msbuild is ignoring the path, where perl and bison and flex are.
Although it is being set, within the 64-bit compilation environment of msvc 2019.
I'm still investigating.
In fact perl, it is found, otherwise, neither build.pl would be working. But within the perl environment, when the system call is made, in this case, neither perl, bison, nor flex is found.
 
regards,
Ranier Vilela
Em qua., 6 de mai. de 2020 às 10:25, Ranier Vilela <ranier.vf@gmail.com> escreveu:
Em qua., 6 de mai. de 2020 às 10:21, Ranier Vilela <ranier.vf@gmail.com> escreveu:
Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier <michael@paquier.xyz> escreveu:
On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:
> I agree, it is better.

Thanks, applied and back-patched down to 9.5.  Now for the second
problem of this thread..
Sorry, no clue yet.
I hacked the perl sources, to hardcoded perl, bison and flex with path.It works like this.
For some reason, which I haven't yet discovered, msbuild is ignoring the path, where perl and bison and flex are.
Although it is being set, within the 64-bit compilation environment of msvc 2019.
I'm still investigating.
In fact perl, it is found, otherwise, neither build.pl would be working. But within the perl environment, when the system call is made, in this case, neither perl, bison, nor flex is found.

I'm using it like this, for now.

File pgbison.pl:
system("c:\\bin\\bison $headerflag $input -o $output");
File pgflex.pl:
system("c:\\bin\\flex $flexflags -o$output $input");
                system("c:\\perl\\bin\\perl src\\tools\\fix-old-flex-code.pl $output");

File Solution.pm:
                system(
                system('perl generate-lwlocknames.pl lwlocknames.txt');
                system(
                system(
                system(
                system(
                system(
                system(
                system("perl create_help.pl ../../../doc/src/sgml/ref sql_help");
                system(
                system(
                system(
                system(
                system(
                system('perl parse.pl < ../../../backend/parser/gram.y > preproc.y');
                system(

C:\dll\postgres\src\tools\msvc>\bin\grep bison *pm
File MSBuildProject.pm:
      <Message Condition="'\$(Configuration)|\$(Platform)'=='Debug|$self->{platform}'">Running bison on $grammarFile</Message>
      <Command Condition="'\$(Configuration)|\$(Platform)'=='Debug|$self->{platform}'">c:\\perl\\bin\\perl "src\\tools\\msvc\\pgbison.pl" "$grammarFile"</Command>
      <Message Condition="'\$(Configuration)|\$(Platform)'=='Release|$self->{platform}'">Running bison on $grammarFile</Message>
      <Command Condition="'\$(Configuration)|\$(Platform)'=='Release|$self->{platform}'">c:\\perl\\bin\\perl "src\\tools\\msvc\\pgbison.pl" "$grammarFile"</Command>

C:\dll\postgres\src\tools\msvc>\bin\grep flex *pm
File MSBuildProject.pm:
      <Message Condition="'\$(Configuration)|\$(Platform)'=='Debug|$self->{platform}'">Running flex on $grammarFile</Message>
      <Command Condition="'\$(Configuration)|\$(Platform)'=='Debug|$self->{platform}'">c:\\perl\\bin\\perl "src\\tools\\msvc\\pgflex.pl" "$grammarFile"</Command>
      <Message Condition="'\$(Configuration)|\$(Platform)'=='Release|$self->{platform}'">Running flex on $grammarFile</Message>
      <Command Condition="'\$(Configuration)|\$(Platform)'=='Release|$self->{platform}'">c:\\perl\\bin\\perl "src\\tools\\msvc\\pgflex.pl" "$grammarFile"</Command>
 
regards,
Ranier Vilela
В Wed, 6 May 2020 10:21:41 -0300
Ranier Vilela <ranier.vf@gmail.com> пишет:

> Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier
> <michael@paquier.xyz> escreveu:
>
> > On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:
> > > I agree, it is better.
> >
> > Thanks, applied and back-patched down to 9.5.  Now for the second
> > problem of this thread..
> >
> Sorry, no clue yet.
> I hacked the perl sources, to hardcoded perl, bison and flex with
> path.It works like this.

Perl has "magic" variable $^X which expands to full path to perl
executable, I wonder why build.pl doesn't  use it to invoke secondary
perl scripts.

--



On 5/6/20 1:14 PM, Victor Wagner wrote:
> В Wed, 6 May 2020 10:21:41 -0300
> Ranier Vilela <ranier.vf@gmail.com> пишет:
>
>> Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier
>> <michael@paquier.xyz> escreveu:
>>
>>> On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote:  
>>>> I agree, it is better.  
>>> Thanks, applied and back-patched down to 9.5.  Now for the second
>>> problem of this thread..
>>>  
>> Sorry, no clue yet.
>> I hacked the perl sources, to hardcoded perl, bison and flex with
>> path.It works like this.
> Perl has "magic" variable $^X which expands to full path to perl
> executable, I wonder why build.pl doesn't  use it to invoke secondary 
> perl scripts.
>

We assume perl, flex and bison are in the PATH. That doesn't seem
unreasonable, it's worked well for quite a long time.


cheers


andrew


-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services




Em qua., 6 de mai. de 2020 às 14:14, Victor Wagner <vitus@wagner.pp.ru> escreveu:
В Wed, 6 May 2020 10:21:41 -0300
Ranier Vilela <ranier.vf@gmail.com> пишет:

> Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier
> <michael@paquier.xyz> escreveu:
>
> > On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote: 
> > > I agree, it is better. 
> >
> > Thanks, applied and back-patched down to 9.5.  Now for the second
> > problem of this thread..
> > 
> Sorry, no clue yet.
> I hacked the perl sources, to hardcoded perl, bison and flex with
> path.It works like this.

Perl has "magic" variable $^X which expands to full path to perl
executable, I wonder why build.pl doesn't  use it to invoke secondary
perl scripts.
I still don't think it's necessary, it was working well.
My main suspicions are:
1. Path with spaces;
2. Incompatibility with < symbol, some suggest use &quot;
<Exec Command="&quot;
3. Msbuid.exe It has been updated (version 16.5.0)
4. Perl scripts increased the level of security.
5. My user do not have administrator rights.
 
regards,
Ranier Vilela
Em qua., 6 de mai. de 2020 às 15:19, Ranier Vilela <ranier.vf@gmail.com> escreveu:
Em qua., 6 de mai. de 2020 às 14:14, Victor Wagner <vitus@wagner.pp.ru> escreveu:
В Wed, 6 May 2020 10:21:41 -0300
Ranier Vilela <ranier.vf@gmail.com> пишет:

> Em qua., 6 de mai. de 2020 às 09:53, Michael Paquier
> <michael@paquier.xyz> escreveu:
>
> > On Tue, May 05, 2020 at 10:16:23AM +0300, Victor Wagner wrote: 
> > > I agree, it is better. 
> >
> > Thanks, applied and back-patched down to 9.5.  Now for the second
> > problem of this thread..
> > 
> Sorry, no clue yet.
> I hacked the perl sources, to hardcoded perl, bison and flex with
> path.It works like this.

Perl has "magic" variable $^X which expands to full path to perl
executable, I wonder why build.pl doesn't  use it to invoke secondary
perl scripts.
I still don't think it's necessary, it was working well.
My main suspicions are:
1. Path with spaces;
2. Incompatibility with < symbol, some suggest use &quot;
<Exec Command="&quot;
3. Msbuid.exe It has been updated (version 16.5.0)
4. Perl scripts increased the level of security.
5. My user do not have administrator rights.
Cause found.

How it worked before
1. Call link from menu Visual Studio 2019: Auxiliary\Build\vcvars64.bat
    That create a console with settings to compile on 64 bits.
2. Adjusting the path manually
    set path=%path%;c:\perl\bin;c:\bin
3. Call build.bat

Hacking pgbison.pl, to print PATH, shows that the path inside pgbison.pl, returned to being the original, without the addition of c:\perl\bin;c:\bin.
my $out = $ENV{PATH};
print "Path after system call=$out\n";
Path after system call=...C:\Users\ranier\AppData\Local\Microsoft\WindowsApps;;
The final part lacks: c:\perl\bin;c:\bin

Now I need to find out why the path is being reset, within the perl scripts.

Cause: PATH being reset.

regards,
Ranier Vilela
On Wed, May 06, 2020 at 02:11:34PM -0400, Andrew Dunstan wrote:
> We assume perl, flex and bison are in the PATH. That doesn't seem
> unreasonable, it's worked well for quite a long time.

I recall that it is an assumption we rely on since MSVC scripts are
around, and that's rather easy to configure, so it seems to me that
changing things now would just introduce annoying changes for anybody
(developers, maintainers) using this stuff.
--
Michael

Attachment
Em qua., 6 de mai. de 2020 às 21:08, Michael Paquier <michael@paquier.xyz> escreveu:
On Wed, May 06, 2020 at 02:11:34PM -0400, Andrew Dunstan wrote:
> We assume perl, flex and bison are in the PATH. That doesn't seem
> unreasonable, it's worked well for quite a long time.

I recall that it is an assumption we rely on since MSVC scripts are
around, and that's rather easy to configure, so it seems to me that
changing things now would just introduce annoying changes for anybody
(developers, maintainers) using this stuff.
Ah yes, better to leave it as is. No problem for me, I already got around the difficulty.

regards,
Ranier Vilela
On Wed, May 06, 2020 at 03:58:15PM -0300, Ranier Vilela wrote:
> Hacking pgbison.pl, to print PATH, shows that the path inside pgbison.pl,
> returned to being the original, without the addition of c:\perl\bin;c:\bin.
> my $out = $ENV{PATH};
> print "Path after system call=$out\n";
> Path after system
> call=...C:\Users\ranier\AppData\Local\Microsoft\WindowsApps;;
> The final part lacks: c:\perl\bin;c:\bin
>
> Now I need to find out why the path is being reset, within the perl scripts.

FWIW, we have a buildfarm animal called drongo that runs with VS 2019,
that uses Python, and that is now happy.  One of my own machines uses
VS 2019 as well and I have yet to see what you are describing here.
Perhaps that's related to a difference in the version of perl you are
using and the version of that any others?
--
Michael

Attachment
Em qua., 6 de mai. de 2020 às 21:14, Michael Paquier <michael@paquier.xyz> escreveu:
On Wed, May 06, 2020 at 03:58:15PM -0300, Ranier Vilela wrote:
> Hacking pgbison.pl, to print PATH, shows that the path inside pgbison.pl,
> returned to being the original, without the addition of c:\perl\bin;c:\bin.
> my $out = $ENV{PATH};
> print "Path after system call=$out\n";
> Path after system
> call=...C:\Users\ranier\AppData\Local\Microsoft\WindowsApps;;
> The final part lacks: c:\perl\bin;c:\bin
>
> Now I need to find out why the path is being reset, within the perl scripts.

FWIW, we have a buildfarm animal called drongo that runs with VS 2019,
that uses Python, and that is now happy.  One of my own machines uses
VS 2019 as well and I have yet to see what you are describing here.
Perhaps that's related to a difference in the version of perl you are
using and the version of that any others?
I really don't know what to say, I know very little about perl.

The perl is:
Win32 strawberry-perl 5.30.1.1

regards,
Ranier VIlela
В Thu, 7 May 2020 09:14:33 +0900
Michael Paquier <michael@paquier.xyz> пишет:

> On Wed, May 06, 2020 at 03:58:15PM -0300, Ranier Vilela wrote:
> > Hacking pgbison.pl, to print PATH, shows that the path inside
> > pgbison.pl, returned to being the original, without the addition of
> > c:\perl\bin;c:\bin. my $out = $ENV{PATH};
> > print "Path after system call=$out\n";
> > Path after system
> > call=...C:\Users\ranier\AppData\Local\Microsoft\WindowsApps;;
> > The final part lacks: c:\perl\bin;c:\bin
> >
> > Now I need to find out why the path is being reset, within the perl
> > scripts.
>
> FWIW, we have a buildfarm animal called drongo that runs with VS 2019,
> that uses Python, and that is now happy.  One of my own machines uses
> VS 2019 as well and I have yet to see what you are describing here.
> Perhaps that's related to a difference in the version of perl you are
> using and the version of that any others?


I doubt so. I have different machines with perl from 5.22 to 5.30 but
none of tham exibits such weird behavoir.

Perhaps problem is that Ranier calls vcvars64.bat from the menu, and
then calls msbuild such way that is becames unrelated process.

Obvoisly buildfarm animal doesn't use menu and then starts build
process from same CMD.EXE process, that it called vcvarsall.but into.

It is same in all OSes - Windows, *nix and even MS-DOS - there is no
way to change environment of parent process. You can change environment
of current process (and if this process is command interpreter you can
do so by sourcing script into it. In windows this misleadingly called
'CALL', but it executes commands from command file in the current
shell, not in subshell) you can pass enivronment to the child
processes. But you can never affect environment of the parent or
sibling process.

The only exception is - if you know that some process would at startup
read environment vars from some file or registry, you can modify this
source in unrelated process.

So, if you want perl in path of msbuild, started from Visual Studio,
you should either first set path in CMD.EXE, then type command to start
Studio from this very  command window, or set path via control panel
dialog (which modified registry). Later is what I usially do on machines
wher I compile postgres.




--



> --
> Michael




В Wed, 6 May 2020 21:23:57 -0300
Ranier Vilela <ranier.vf@gmail.com> пишет:

>
> The perl is:
> Win32 strawberry-perl 5.30.1.1
>

This perl would have problems when compiling PL/Perl (see my letter
about week ago), but it have no problems running various build scripts
for Postgres. I'm using it with MSVisualStudio 2019 and only unexpected
thing I've encountered is that it comes with its own  patch.exe, which
doesn't like unix-style end-of-lines in patches (but OK with them in
patched files).


> regards,
> Ranier VIlela
--




On Wed, May 06, 2020 at 12:17:03AM +0200, Juan José Santamaría Flecha wrote:
> Please forgive me if I am being too nitpicky, but I find the comments a
> little too verbose, a usage format might be more visual and easier to
> explain:
>
> Usage: build [[CONFIGURATION] COMPONENT]
>
> The options are  case-insensitive.
> CONFIGURATION sets the configuration to build, "debug" or "release" (by
> default).
> COMPONENT defines a component to build. An empty option means all
> components.

Your comment makes sense to me.  What about the attached then?  On top
of documenting the script usage in the code, let's trigger it if it
gets called with more than 3 arguments.  What do you think?

FWIW, I forgot to mention that I don't think those warnings are worth
a backpatch.  No objections with improving things on HEAD of course.
--
Michael

Attachment
Em qui., 7 de mai. de 2020 às 02:04, Victor Wagner <vitus@wagner.pp.ru> escreveu:
В Thu, 7 May 2020 09:14:33 +0900
Michael Paquier <michael@paquier.xyz> пишет:

> On Wed, May 06, 2020 at 03:58:15PM -0300, Ranier Vilela wrote:
> > Hacking pgbison.pl, to print PATH, shows that the path inside
> > pgbison.pl, returned to being the original, without the addition of
> > c:\perl\bin;c:\bin. my $out = $ENV{PATH};
> > print "Path after system call=$out\n";
> > Path after system
> > call=...C:\Users\ranier\AppData\Local\Microsoft\WindowsApps;;
> > The final part lacks: c:\perl\bin;c:\bin
> >
> > Now I need to find out why the path is being reset, within the perl
> > scripts. 
>
> FWIW, we have a buildfarm animal called drongo that runs with VS 2019,
> that uses Python, and that is now happy.  One of my own machines uses
> VS 2019 as well and I have yet to see what you are describing here.
> Perhaps that's related to a difference in the version of perl you are
> using and the version of that any others?


I doubt so. I have different machines with perl from 5.22 to 5.30 but
none of tham exibits such weird behavoir.
The perl is the same,when it was working ok.
 

Perhaps problem is that Ranier calls vcvars64.bat from the menu, and
then calls msbuild such way that is becames unrelated process.
It also worked previously, using this same process, link menu and manual path configuration.
What has changed:
1 In the environment, the python installation, which added entries to the path.
2. Perl scripts: Use perl's $/ more idiomatically
commit beb2516e961490723fb1a2f193406afb3d71ea9c
3. Msbuild and others, have been updated.They are not the same ones that were working before.
 

Obvoisly buildfarm animal doesn't use menu and then starts build
process from same CMD.EXE process, that it called vcvarsall.but into.

It is same in all OSes - Windows, *nix and even MS-DOS - there is no
way to change environment of parent process. You can change environment
of current process (and if this process is command interpreter you can
do so by sourcing script into it. In windows this misleadingly called
'CALL', but it executes commands from command file in the current
shell, not in subshell) you can pass enivronment to the child
processes. But you can never affect environment of the parent or
sibling process.
Maybe that's what is happening, calling system, perl or msbuild, would be creating a new environment, transferring the path that is configured in Windows, and not the path that is in the environment that was manually configured.
 

The only exception is - if you know that some process would at startup
read environment vars from some file or registry, you can modify this
source in unrelated process.

So, if you want perl in path of msbuild, started from Visual Studio,
you should either first set path in CMD.EXE, then type command to start
Studio from this very  command window, or set path via control panel
dialog (which modified registry). Later is what I usially do on machines
wher I compile postgres.
buidfarm aninal, uses a more secure and reliable process, the path is already configured and does not change.
Perhaps this is the way for me and for others.

It would then remain to document, to warn that to work correctly, the path must be configured before entering the compilation environment.

regards,
Ranier Vilela
Hi Michael,

I performed a quick test for the path "msvc-build-init-v2.patch" using
below cases:

1. perl build.pl
2. perl build.pl debug psql
3. perl build.pl RELEASE psql
4. perl build.pl deBUG psql
5. perl build.pl psql
The above cases (case-insensitive) are all working great without any
warning for latest master branch.

When build with more than 3 parameters, yes, I got below expected
message as well.

c:\Users\david\Downloads\postgres\src\tools\msvc>perl build.pl DEbug
psql pg_baseback
Usage: build.pl [ [ <configuration> ] <component> ]
Options are case-insensitive.
configuration: Release | Debug. This sets the configuration
to build. Default is Release.
component: name of component to build. An empty value means
to build all components.


However, if I ask for help in a typical Windows' way, i.e, "perl
build.pl -help" or "perl build.pl /?", I got some messages like below,

c:\Users\david\Downloads\postgres\src\tools\msvc>perl build.pl -help
Detected hardware platform: Win32
Files src/bin/pgbench/exprscan.l
Files src/bin/pgbench/exprparse.y
Files src/bin/psql/psqlscanslash.l
Files contrib/cube/cubescan.l
Files contrib/cube/cubeparse.y
Files contrib/seg/segscan.l
Files contrib/seg/segparse.y
Generating configuration headers...
Microsoft (R) Build Engine version 16.5.1+4616136f8 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1001: Unknown switch.
Switch: -help.vcxproj

For switch syntax, type "MSBuild -help"

c:\Users\david\Downloads\postgres\src\tools\msvc>perl build.pl /?
Detected hardware platform: Win32
Files src/bin/pgbench/exprscan.l
Files src/bin/pgbench/exprparse.y
Files src/bin/psql/psqlscanslash.l
Files contrib/cube/cubescan.l
Files contrib/cube/cubeparse.y
Files contrib/seg/segscan.l
Files contrib/seg/segparse.y
Generating configuration headers...
Microsoft (R) Build Engine version 16.5.1+4616136f8 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1001: Unknown switch.
Switch: /?.vcxproj

For switch syntax, type "MSBuild -help"

It would be a bonus if the build.pl can support the "help" in Windows' way.


Thanks,

David

On 2020-05-06 10:45 p.m., Michael Paquier wrote:
> On Wed, May 06, 2020 at 12:17:03AM +0200, Juan José Santamaría Flecha wrote:
>> Please forgive me if I am being too nitpicky, but I find the comments a
>> little too verbose, a usage format might be more visual and easier to
>> explain:
>>
>> Usage: build [[CONFIGURATION] COMPONENT]
>>
>> The options are  case-insensitive.
>> CONFIGURATION sets the configuration to build, "debug" or "release" (by
>> default).
>> COMPONENT defines a component to build. An empty option means all
>> components.
> Your comment makes sense to me.  What about the attached then?  On top
> of documenting the script usage in the code, let's trigger it if it
> gets called with more than 3 arguments.  What do you think?
>
> FWIW, I forgot to mention that I don't think those warnings are worth
> a backpatch.  No objections with improving things on HEAD of course.
> --
> Michael
--
David

Software Engineer
Highgo Software Inc. (Canada)
www.highgo.ca




Re: Postgres Windows build system doesn't work with python installed in Program Files

From
Juan José Santamaría Flecha
Date:

On Tue, Jun 2, 2020 at 2:06 AM David Zhang <david.zhang@highgo.ca> wrote:

On 2020-05-06 10:45 p.m., Michael Paquier wrote:
> On Wed, May 06, 2020 at 12:17:03AM +0200, Juan José Santamaría Flecha wrote:
>> Please forgive me if I am being too nitpicky, but I find the comments a
>> little too verbose, a usage format might be more visual and easier to
>> explain:
>>
>> Usage: build [[CONFIGURATION] COMPONENT]
>>
>> The options are  case-insensitive.
>> CONFIGURATION sets the configuration to build, "debug" or "release" (by
>> default).
>> COMPONENT defines a component to build. An empty option means all
>> components.
> Your comment makes sense to me.  What about the attached then?  On top
> of documenting the script usage in the code, let's trigger it if it
> gets called with more than 3 arguments.  What do you think?
>
> FWIW, I forgot to mention that I don't think those warnings are worth
> a backpatch.  No objections with improving things on HEAD of course.

It would be a bonus if the build.pl can support the "help" in Windows' way.

Going through the open items in the commitfest, I see that this patch has not been pushed. It still applies and solves the warning so, I am marking it as RFC.

Adding a help option is a new feature, that can have its own patch without delaying this one any further.

Regards,

Juan José Santamaría Flecha
On Sat, Jul 04, 2020 at 09:17:52PM +0200, Juan José Santamaría Flecha wrote:
> Going through the open items in the commitfest, I see that this patch has
> not been pushed. It still applies and solves the warning so, I am marking
> it as RFC.

Thanks, applied.  I was actually waiting to see if you had more
comments.

> Adding a help option is a new feature, that can have its own patch without
> delaying this one any further.

Yep.  And I am not sure if that's worth worrying either.
--
Michael

Attachment