Thread: It seems no Windows buildfarm members are running find_typedefs

It seems no Windows buildfarm members are running find_typedefs

From
Tom Lane
Date:
The current typedefs list seems to be lacking any Windows-only typedefs.
Noticed while trying to pgindent postmaster.c.
        regards, tom lane



Re: It seems no Windows buildfarm members are running find_typedefs

From
Andrew Dunstan
Date:
On 04/01/2014 08:53 PM, Tom Lane wrote:
> The current typedefs list seems to be lacking any Windows-only typedefs.
> Noticed while trying to pgindent postmaster.c.
>
>             



Hmm. odd. will check.

cheers

andrew




Re: It seems no Windows buildfarm members are running find_typedefs

From
Andrew Dunstan
Date:
On 04/01/2014 09:22 PM, Andrew Dunstan wrote:
>
> On 04/01/2014 08:53 PM, Tom Lane wrote:
>> The current typedefs list seems to be lacking any Windows-only typedefs.
>> Noticed while trying to pgindent postmaster.c.
>>
>>
>
>
>
> Hmm. odd. will check.
>
>

It's apparently causing the buildfarm to crash, which is why I must have 
disabled it. I'll chase that down tomorrow.

cheers

andrew





Re: It seems no Windows buildfarm members are running find_typedefs

From
Andrew Dunstan
Date:
On 04/02/2014 12:25 AM, Andrew Dunstan wrote:
>
> On 04/01/2014 09:22 PM, Andrew Dunstan wrote:
>>
>> On 04/01/2014 08:53 PM, Tom Lane wrote:
>>> The current typedefs list seems to be lacking any Windows-only 
>>> typedefs.
>>> Noticed while trying to pgindent postmaster.c.
>>>
>>>
>>
>>
>>
>> Hmm. odd. will check.
>>
>>
>
> It's apparently causing the buildfarm to crash, which is why I must 
> have disabled it. I'll chase that down tomorrow.


OK, we're back: 
<http://www.pgbuildfarm.org/cgi-bin/show_stage_log.pl?nm=frogmouth&dt=2014-04-02%2016%3A08%3A12&stg=typedefs>


BTW, three animals are currently trying to contribute typedefs but 
aren't in fact contributing anything: okapi, dromedary and prairiedog. 
See <http://www.pgbuildfarm.org/cgi-bin/typedefs.pl?show_list=1>

I can't really help much on these as my Gentoo facilities are 
non-existent, and my OSX facilities are not much better. I do recall 
trying to find a way to get typedefs on OSX a few years ago, without 
success.


cheers

andrew


>
> cheers
>
> andrew
>
>
>
>




Re: It seems no Windows buildfarm members are running find_typedefs

From
Tom Lane
Date:
Andrew Dunstan <andrew@dunslane.net> writes:
> BTW, three animals are currently trying to contribute typedefs but 
> aren't in fact contributing anything: okapi, dromedary and prairiedog. 
> See <http://www.pgbuildfarm.org/cgi-bin/typedefs.pl?show_list=1>

Man, that's a short list.  I wonder if we need to encourage more people
to do this.

> I can't really help much on these as my Gentoo facilities are 
> non-existent, and my OSX facilities are not much better. I do recall 
> trying to find a way to get typedefs on OSX a few years ago, without 
> success.

I poked around a bit, and so far as I can tell, OS X does not store debug
symbol tables in executables.  It looks like gdb goes to the .o files when
it wants debug info.  What's in the .o files is good ol' DWARF (at least
in reasonably recent OS X releases), so it's not any harder to pull out
the typedef names than it is on Linux.  The problem is that you gotta
iterate over all the .o files in the build tree rather than the installed
executables.  I looked at fixing find_typedefs but it seems like it's
pretty fixated on the latter approach; any thoughts on how to revise it?
        regards, tom lane



Re: It seems no Windows buildfarm members are running find_typedefs

From
Andrew Dunstan
Date:
On 04/02/2014 08:43 PM, Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> BTW, three animals are currently trying to contribute typedefs but
>> aren't in fact contributing anything: okapi, dromedary and prairiedog.
>> See <http://www.pgbuildfarm.org/cgi-bin/typedefs.pl?show_list=1>
> Man, that's a short list.  I wonder if we need to encourage more people
> to do this.
>
>> I can't really help much on these as my Gentoo facilities are
>> non-existent, and my OSX facilities are not much better. I do recall
>> trying to find a way to get typedefs on OSX a few years ago, without
>> success.
> I poked around a bit, and so far as I can tell, OS X does not store debug
> symbol tables in executables.  It looks like gdb goes to the .o files when
> it wants debug info.  What's in the .o files is good ol' DWARF (at least
> in reasonably recent OS X releases), so it's not any harder to pull out
> the typedef names than it is on Linux.  The problem is that you gotta
> iterate over all the .o files in the build tree rather than the installed
> executables.  I looked at fixing find_typedefs but it seems like it's
> pretty fixated on the latter approach; any thoughts on how to revise it?
>
>             


Well, the reason it's that way is that that's the way it was done before 
the buildfarm took over the task. But it's not holy writ. Doing 
something else would be a SMOC.

Essentially, I think the part that would need to change is this:
        foreach my $bin (            glob("$installdir/bin/*"),            glob("$installdir/lib/*"),
glob("$installdir/lib/postgresql/*")         )
 

For OSX we'd construct the list via File::Find to recurse through the 
directories.

So, something like this:

   my $using_osx = [some test for OSX];   my @testfiles;   my $obj_wanted = sub {        /^.*\.o\z/s &&
push(@testfiles,$File::Find::name);   };   if ($using_osx)   {        File::Find::find($obj_wanted,$pgsql);   }   else
{     @testfiles = (            glob("$installdir/bin/*"),            glob("$installdir/lib/*"),
glob("$installdir/lib/postgresql/*");  }   foreach my $bin (@testfiles)
 


cheers

andrew



Re: It seems no Windows buildfarm members are running find_typedefs

From
Wim Lewis
Date:
On 2 Apr 2014, at 5:43 PM, Tom Lane wrote:
> I poked around a bit, and so far as I can tell, OS X does not store debug
> symbol tables in executables.  It looks like gdb goes to the .o files when
> it wants debug info.  What's in the .o files is good ol' DWARF (at least
> in reasonably recent OS X releases), so it's not any harder to pull out
> the typedef names than it is on Linux.  The problem is that you gotta
> iterate over all the .o files in the build tree rather than the installed
> executables.  I looked at fixing find_typedefs but it seems like it's
> pretty fixated on the latter approach; any thoughts on how to revise it?

The Apple development tools gather the debug information during the final link stage (the one that produces the
executableor shared object) using "dsymutil", which simply iterates over all of the .o files and links the debug info
intoa separate object, foo.dSYM. Apple's gdb and lldb then find the relevant .dSYM file using a per-build UUID embedded
inthe executable/library/debug symbol file. 

The dSYM file is a normal object file which has the DWARF sections but not the usual text/data sections, so if it can
begenerated/found, it should be possible to dump the DWARF data from it and look for typedefs that way. 

(I'm pretty sure that if you were to run dsymutil and then merge the resulting object file's sections into the
executable/shlib,you'd get a perfectly functional and debuggable result without having to look for or cart around the
extradSYM file--- I haven't tried this though.) 





Re: It seems no Windows buildfarm members are running find_typedefs

From
Tom Lane
Date:
Wim Lewis <wiml@omnigroup.com> writes:
> On 2 Apr 2014, at 5:43 PM, Tom Lane wrote:
>> I poked around a bit, and so far as I can tell, OS X does not store debug
>> symbol tables in executables.

> The Apple development tools gather the debug information during the final link stage (the one that produces the
executableor shared object) using "dsymutil", which simply iterates over all of the .o files and links the debug info
intoa separate object, foo.dSYM. Apple's gdb and lldb then find the relevant .dSYM file using a per-build UUID embedded
inthe executable/library/debug symbol file.
 

Ah.  I've forgotten the details, but I'm pretty sure that we have
deliberately arranged our build process so that the .dSYM files don't get
built during link steps.  Debugging seems to work all right anyway, at
least if the build tree is available, so I think Apple's gdb is able to
work from the symbol tables in the .o files.

While perhaps that approach should be rethought, I'm disinclined to
mess with it just for the benefit of find_typedefs.
        regards, tom lane



Re: It seems no Windows buildfarm members are running find_typedefs

From
Tom Lane
Date:
Andrew Dunstan <andrew@dunslane.net> writes:
> For OSX we'd construct the list via File::Find to recurse through the
> directories.
> So, something like this:

Thanks for the tips.  The attached patch against buildfarm 4.11 seems
to work, cf
http://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=dromedary&dt=2014-04-03%2003%3A33%3A16&stg=typedefs
I tried it on current OSX (10.9.2) as well as dromedary's 10.6.8.
It does not work on prairiedog (10.4.11) --- the debug info format
seems to be different that far back.  Can't say that's worth worrying
about.

            regards, tom lane

*** build-farm-4.11/run_build.pl~    Fri Jun 14 09:05:52 2013
--- build-farm-4.11/run_build.pl    Wed Apr  2 23:31:27 2014
***************
*** 1506,1520 ****
      my $objdump = $host || 'objdump';
      my @err = `$objdump -W 2>&1`;
      my @readelferr = `readelf -w 2>&1`;
      my %syms;
      my @dumpout;
      my @flds;

!     foreach my $bin (
!         glob("$installdir/bin/*"),
!         glob("$installdir/lib/*"),
!         glob("$installdir/lib/postgresql/*")
!       )
      {
          next if $bin =~ m!bin/(ipcclean|pltcl_)!;
          next unless -f $bin;
--- 1506,1536 ----
      my $objdump = $host || 'objdump';
      my @err = `$objdump -W 2>&1`;
      my @readelferr = `readelf -w 2>&1`;
+     my $using_osx = (`uname` eq "Darwin\n");
+     my @testfiles;
      my %syms;
      my @dumpout;
      my @flds;

!     if ($using_osx)
!     {
!         # On OS X, we need to examine the .o files
!         my $obj_wanted = sub {
!             /^.*\.o\z/s && push(@testfiles, $File::Find::name);
!         };
!
!         File::Find::find($obj_wanted,$pgsql);
!     }
!     else
!     {
!         # Elsewhere, look at the installed executables and shared libraries
!         @testfiles = (
!             glob("$installdir/bin/*"),
!             glob("$installdir/lib/*"),
!             glob("$installdir/lib/postgresql/*")
!         );
!     }
!     foreach my $bin (@testfiles)
      {
          next if $bin =~ m!bin/(ipcclean|pltcl_)!;
          next unless -f $bin;
***************
*** 1534,1540 ****
          }
          elsif ( @readelferr > 10 )
          {
-
              # FreeBSD, similar output to Linux
              @dumpout =
  `readelf -w $bin 2>/dev/null | egrep -A3 DW_TAG_typedef 2>/dev/null`;
--- 1550,1555 ----
***************
*** 1546,1551 ****
--- 1561,1579 ----
                  $syms{$flds[-1]} =1;
              }
          }
+         elsif ($using_osx)
+         {
+             @dumpout =
+ `dwarfdump $bin 2>/dev/null | egrep -A2 TAG_typedef 2>/dev/null`;
+             foreach (@dumpout)
+             {
+                 @flds = split;
+                 next unless (@flds == 3);
+                 next unless ($flds[0] eq "AT_name(");
+                 next unless ($flds[1] =~ m/^"(.*)"$/);
+                 $syms{$1} =1;
+             }
+         }
          else
          {
              @dumpout = `$objdump --stabs $bin 2>/dev/null`;

Re: It seems no Windows buildfarm members are running find_typedefs

From
Tom Lane
Date:
I wrote:
> Thanks for the tips.  The attached patch against buildfarm 4.11 seems
> to work, cf
> http://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=dromedary&dt=2014-04-03%2003%3A33%3A16&stg=typedefs

BTW, after looking a bit more closely at what this added to the typedefs
list, I realize that this mechanism also collects typedefs that appear in
files that are compiled but never installed, for example src/timezone/zic
and the ecpg regression tests.  This seems like a Good Thing, since
certainly pgindent is going to hit the source files for those programs
too.  I wonder if we ought to switch over to scanning the .o files on
all platforms?
        regards, tom lane



Re: It seems no Windows buildfarm members are running find_typedefs

From
Andrew Dunstan
Date:
On 04/03/2014 12:01 AM, Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
>> For OSX we'd construct the list via File::Find to recurse through the
>> directories.
>> So, something like this:
> Thanks for the tips.  The attached patch against buildfarm 4.11 seems
> to work, cf
> http://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=dromedary&dt=2014-04-03%2003%3A33%3A16&stg=typedefs
> I tried it on current OSX (10.9.2) as well as dromedary's 10.6.8.
> It does not work on prairiedog (10.4.11) --- the debug info format
> seems to be different that far back.  Can't say that's worth worrying
> about.
>
>         


Thanks, applied.

> BTW, after looking a bit more closely at what this added to the typedefs
> list, I realize that this mechanism also collects typedefs that appear in
> files that are compiled but never installed, for example src/timezone/zic
> and the ecpg regression tests.  This seems like a Good Thing, since
> certainly pgindent is going to hit the source files for those programs
> too.  I wonder if we ought to switch over to scanning the .o files on
> all platforms?

Sure, if someone wants to work out what's involved. Do the MSVC tools 
have some gadget for extracting symbols from .o files in a similar way? 
It would be nice to get them into the mix too.

cheers

andrew