On Sun, Jul 26, 2015 at 11:15 PM, Peter Geoghegan <pg@heroku.com> wrote:
> The SQLite people have been using a tool like this for some time.
> They've also had luck finding bugs with a generic fuzz-testing tool
> called "american fuzzy lop" (yes, seriously, that's what it's called),
> which apparently is the state of the art.
>
> I myself ran that tool against Postgres. I didn't spend enough time to
> tweak it in a way that might have been effective. I also didn't figure
> out a way to make iterations fast enough for the tool to be effective,
> because I was invoking Postgres in single-user mode. I might pick it
> up again in the future, but probably for a more targeted case.
I've been poking at this. Like you I ran AFL against Postgres in
single-user mode. It was slowed down by the Postgres startup time and
worse, it was very limiting being in single-user-mode.
What I think holds more promise is Libfuzzer from LLVM. It's not as
clever about generating test cases and it doesn't have a pretty curses
interface, but it's much more amenable to integrating into a
client/server architecture.
In fact this is the interface I have now:
stark=> SELECT fuzz(1000000, 'select $1::timestamptz')
Pretty slick, eh? :)
It's also blindingly fast. Even with a subtransaction around each
fuzzing call it's still getting up to 5-8k/s executions for simpler
functions. The main disadvantage is that it's more fragile because
it's not isolated from the program it's testing. So a bug that causes
infinite recursion or consumes lots of memory can cause it to crash
and it can take a while to do so.
I hope to package it up as a contrib module but it does require the
Libfuzzer library which is built as part of LLVM but only when you
build LLVM itself with the coverage sanitizer so it doesn't end up in
any llvm binary packages. And of course building LLVM takes forever
and a day... Maybe I'll include a copy of the llvm source files in the
module -- the license looks to be compatible.
Anyways, first non-security trophy it found:
stark=# SELECT 'doy'::timestamptz;
ERROR: unexpected dtype 33 while parsing timestamptz "doy"
LINE 1: SELECT 'doy'::timestamptz; ^
stark=# SELECT 'dow'::timestamptz;
ERROR: unexpected dtype 32 while parsing timestamptz "dow"
LINE 1: SELECT 'dow'::timestamptz; ^
This looks to be a quite an old bug. It dates to 2000 and the code
hasn't been touched since Tom wrote it -- not that Tom but Tom
Lockhart. I haven't quite grokked the ParseDateTime code but it looks
to me like the "dow" and "doy" keywords are miscategorized and should
be type UNITS not type RESERV. The corresponding code in extract_date
would then have to move to the other branch leaving "epoch" as the
only reserved word which serves double duty as a unit in extract date.
--
greg
