Thread: Compiler warnings with --enable-dtrace

Hi hackers,

--enable-dtrace produces compiler warnings about const correctness,
except on macOS.  That's because Apple's dtrace produces function
declarations in probes.h that take strings as const char * whereas
AFAIK on all other operating systems they take char * (you can see
that possibly recent difference in Apple's version of dt_header_decl()
in dt_program.c).  People have complained before[1].

Maybe we should do what the Perl people do[2] and post-process the
generated header file to add const qualifiers?  Please see attached.

I have just added --enable-dtrace to my build farm animal elver so
these warnings should appear at the next build.  I wonder if the
owners of damselfly, castoroides, protosciurus (CCed) would consider
adding it for them too so that we could get some coverage of this
build option on Illumos and Solaris.

[1] https://www.postgresql.org/message-id/flat/38D06FCCB225BA1C6699D4E7%40amenophis
[2] https://github.com/Perl/perl5/blob/a385812b685b3164e706880a72ee60c9cc9573e4/Makefile.SH#L870

-- 
Thomas Munro
http://www.enterprisedb.com

Thomas Munro <thomas.munro@enterprisedb.com> writes:
> --enable-dtrace produces compiler warnings about const correctness,
> except on macOS.  That's because Apple's dtrace produces function
> declarations in probes.h that take strings as const char * whereas
> AFAIK on all other operating systems they take char * (you can see
> that possibly recent difference in Apple's version of dt_header_decl()
> in dt_program.c).  People have complained before[1].

Yeah, it's a bit annoying, although AFAICT hardly anyone uses dtrace.

> Maybe we should do what the Perl people do[2] and post-process the
> generated header file to add const qualifiers?  Please see attached.

+1 for the idea.  I notice that Perl's version of this is careful
not to munge lines that already contain "const" ... do we need to
worry about that?

            regards, tom lane

I wrote:
> Thomas Munro <thomas.munro@enterprisedb.com> writes:
>> Maybe we should do what the Perl people do[2] and post-process the
>> generated header file to add const qualifiers?  Please see attached.

> +1 for the idea.  I notice that Perl's version of this is careful
> not to munge lines that already contain "const" ... do we need to
> worry about that?

Oh, I take that back --- on closer look, I see that you're getting
the same effect by checking for a preceding paren or comma.  That's
arguably better than their way because it works if there's a mix of
const and not-const parameters on one input line, though likely no
dtrace implementation actually emits such things.

            regards, tom lane

On Mon, May 7, 2018 at 9:42 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@enterprisedb.com> writes:
>> --enable-dtrace produces compiler warnings about const correctness,
>> except on macOS.  That's because Apple's dtrace produces function
>> declarations in probes.h that take strings as const char * whereas
>> AFAIK on all other operating systems they take char * (you can see
>> that possibly recent difference in Apple's version of dt_header_decl()
>> in dt_program.c).  People have complained before[1].
>
> Yeah, it's a bit annoying, although AFAICT hardly anyone uses dtrace.

You're probably right about that, but the "--enable-dtrace" configure
option is another matter. I started to use it for release builds on my
personal Linux system a few months back, though not because I am a
SystemTap user.

lwn.net had a great (subscriber only) article just today about new
Linux tooling for USDT probes [1]. The BPF/BCC + USDT stuff only
became available in the last year or so. It seems like a technology
that has the potential to be enormously useful for debugging complex
production issues, while still being relatively easy to use. The BCC
"trace" utility [2] can be used to produce simple one-liners that spit
out interesting information about a running Postgres instance. It
seems to be surprisingly low overhead in many cases.

I've been meaning to do a write-up on all of this to make it more
accessible, though it's already quite accessible.

[1] https://lwn.net/Articles/753601/
[2] https://github.com/iovisor/bcc/blob/master/tools/trace_example.txt
-- 
Peter Geoghegan

Re: Peter Geoghegan 2018-05-12 <CAH2-WznHEfw6jJiyH8mBLPT=VKcLyOfwjXEQCHidCkDNv72PSA@mail.gmail.com>
> It seems to be surprisingly low overhead in many cases.

I was pondering adding --enable-dtrace in the Debian packages, but
Andres advised against it, "because it affects code generation".

So far, perf seems to be the tool of choice.

Christoph

On Fri, May 11, 2018 at 11:46 PM, Christoph Berg <myon@debian.org> wrote:
> Re: Peter Geoghegan 2018-05-12 <CAH2-WznHEfw6jJiyH8mBLPT=VKcLyOfwjXEQCHidCkDNv72PSA@mail.gmail.com>
>> It seems to be surprisingly low overhead in many cases.
>
> I was pondering adding --enable-dtrace in the Debian packages, but
> Andres advised against it, "because it affects code generation".
>
> So far, perf seems to be the tool of choice.

Even perf has support for USDT probes from kernel 4.8 on, as the
article mentions. BCC isn't something that you can install just as
easily as perf at the moment, but I think that that's just a matter of
time. As you probably know, BCC can do quite a lot of interesting
things that perf cannot. It deserves to become very popular, and there
are already a lot of early adopters. If this sounds improbable to
anyone, then they should consider that there are already cute comics
about USDT probes that are quite popular [1]. The unnecessarily
complicated way in which BCC/eBPF is distributed and integrated with
the kernel doesn't seem to be holding it back too much.

Does Debian take any position on what packagers should do with USDT
probes in general?

I don't think that this should be an urgent issue for you. I expect it
to become important in the next couple of years, though.

[1] https://jvns.ca/blog/2017/07/05/linux-tracing-systems/#zine
-- 
Peter Geoghegan

On Sun, May 13, 2018 at 8:40 AM, Peter Geoghegan <pg@bowt.ie> wrote:
> On Fri, May 11, 2018 at 11:46 PM, Christoph Berg <myon@debian.org> wrote:
>> Re: Peter Geoghegan 2018-05-12 <CAH2-WznHEfw6jJiyH8mBLPT=VKcLyOfwjXEQCHidCkDNv72PSA@mail.gmail.com>
>>> It seems to be surprisingly low overhead in many cases.
>>
>> I was pondering adding --enable-dtrace in the Debian packages, but

I was initially confused about how --enable-dtrace was being used to
build USDT probes on Linux.  I see that's because systemtap-std-dev
installs its own fake /usr/bin/dtrace that understands the same
switches.  Huh.

>> Andres advised against it, "because it affects code generation".

Right, it inserts a bunch of NOPs and may cause nearby code to be
rearranged slightly.  It'd be interesting to know if it actually makes
any difference to performance, considering the current set of probe
locations.  Whether they're useful for analysing production systems,
I'm not sure anyway -- when I've worked with (real) dtrace I've
typically been adding throwaway probes of my own for testing patches.
I think it's a pretty interesting technique for investigating parallel
query efficiency, for things that simple user stack sampling can't
tell you.

> [1] https://jvns.ca/blog/2017/07/05/linux-tracing-systems/#zine

Thanks.  I do love all this introspection and dynamic tracing tech,
but wow, it's like a bowl of alphabet soup on Linux at this point.

-- 
Thomas Munro
http://www.enterprisedb.com

Re: Peter Geoghegan 2018-05-12 <CAH2-Wz=n=M0_6X46qoLjgf2ADU0x0etM1yjbG8TYMcwHeRxjkw@mail.gmail.com>
> Does Debian take any position on what packagers should do with USDT
> probes in general?

I'm not aware of any guidelines - I guess this stuff is still too new
to make general statements about it. Which other software offers
custom probes?

Christoph

On Wed, May 23, 2018 at 5:17 AM, Christoph Berg <myon@debian.org> wrote:
> I'm not aware of any guidelines - I guess this stuff is still too new
> to make general statements about it. Which other software offers
> custom probes?

As the LWN article mentions, user space probes are available for a
number of other widely used open source software projects, including
MySQL and even glibc. I'm pretty sure that these are not new; they
were added for DTrace quite a few years back, and then perhaps
forgotten about. I'm pretty sure that these are generally not enabled
on standard Linux packages. I bet they're enabled on packages for
Solaris/illumos, though, since DTrace is considered an important part
of the Solaris ecosystem.

In recent years, the Linux kernel gained the ability to expose probes
that can be used in much the same way as the --enable-dtrace
user-space probes, as well as the ability to run a custom eBPF
JIT-compiled program in kernel space, to do the actual
instrumentation. BCC is an externally maintained toolkit that is used
to write most eBPF-based instrumentation in practice, since it makes
the process far easier. BCC isn't a separate technology to eBPF,
though.

BCC/eBPF make the old DTrace/SystemTap probes much more useful on
Linux for a number of reasons. SystemTap had problems with reliability
in the past, owing to the fact that it builds kernel modules rather
than building eBPF code that runs in the eBPF in-kernel virtual
machine (eBPF code is generally considered "safe", which is a huge
advantage). Also, eBPF has less overhead for certain types of
aggregation which are apparently quite important, and benefits from
being JIT compiled (there is an LLVM backend, which is what BCC uses).
Finally, eBPF looks like it might emerge as a kind of defacto standard
on Linux, which SystemTap clearly never managed to do.

The ecosystem questions clearly aren't worked out yet, but there is a
lot of investment in and momentum behind eBPF. There is good reason to
expect that these teething issues be worked out. Linux should be able
to do this stuff without the user having to read a tedious explanation
of how the different parts fit together.

-- 
Peter Geoghegan