Thread: C testing for Postgres

C testing for Postgres

From
Adam Berlin
Date:

During the unconference at PGCon in Ottawa, I asked about writing C-based tests for Postgres. There was interest in trying a tool and also some hesitation to depend on a third-party library. So, I wrote a tool that I'd like to contribute to Postgres. I’ve been calling it cexpect [1].

cexpect is a general-use library for creating test suites in C. It includes:

- a core library for creating and running suites of tests.

- a standard set of test expectations for C

- a default formatter (dots)

- an extensible matcher framework

- an extensible formatter framework

Why add a testing framework for C to Postgres? 

An xUnit-style test framework [2] is a tool for writing tests that is not currently an option for people hacking on Postgres. 

  • C-based tests could help increase code coverage in parts of the codebase that are difficult to reach with a regress-style test (for example: gin posting list compression [3]). 

  • Writing tests for internal components help developers become more familiar with a codebase. 

  • Writing C-based tests go beyond providing regression value by providing feedback on design decisions like modularity and dependencies. 

  • Test suites already existing in `src/test/modules` could benefit from having a consistent way to declare expectations, run suites, and display results.

  • Forks and extensions that write tests could benefit from a testing framework provided by the upstream project. An extensible framework will allow forks and extensions to create their own matchers and formatters without changing the core framework.



The extensible matcher framework has benefits for decoupled, isolated unit tests, and also  high-level system tests. The complexity of a common assertions can be packaged into a matcher - abstracting the complexity and making the assertion easier to reuse.

For example, there could be expectation matchers specifically crafted for the domain of Postgres:

`expect(xlog, to_have_xlog_record(XLOG_XACT_PREPARE))`

`expect(“postgresadmin”, to_be_an_admin_user())`

The matchers that come with cexpect out of the box are for C datatypes:

`expect(1, is_int_equal_to(1))`

`expect(1 == 2, is_false())`

`expect(“some random string”, is_string_containing(“and”))’

… and more, with the goal of having a matcher for all standard data types.

The extensible formatter framework could be used to create a test-anything protocol (TAP) output, familiar to Postgres hackers. It could also be used to insert test results into a database table for later analysis, or create a consistent way of reporting results from a user-defined function - the pattern often used for creating tests under `src/test/modules`.

How does it work?

Create an executable that links to the core shared library, include the core library headers, create a suite, add some tests, and run it.

test.c:


```

#include "cexpect.h"

#include "cexpect_cmatchers.h"

void some_passing_test(Test *test) {

    expect(test, 1, is_int_equal_to(1));

}

int main(int argc, char *args[]) {

    Suite *suite = create_suite("Example test");

    add_test(suite, some_passing_test);

    start_cexpect(suite);

}

```

Running test.c:


```bash

export DYLD_LIBRARY_PATH=$path_to_cexpect_library

export compile_flags=”-Wno-int-conversion -Wno-pointer-to-int-cast test.c -L $path_to_cexpect_library -I $path_to_cexpect_headers”

gcc $compile_flags -l cexpect -o test.o

./test.o

Running suite: Example test

.

Summary:

Ran 1 test(s).

1 passed, 0 failed, 0 pending

```

Rather than post a patch, I'd rather start a conversation first. I'm guessing there are some improvements that we'd want to make (for example: the Makefile) before commiting a patch. Let's iterate on improvements before creating a formal patch.


Thoughts?


Thanks,

Adam Berlin

Software Engineer at Pivotal Greenplum

[1] https://github.com/berlin-ab/cexpect

[2] https://en.wikipedia.org/wiki/XUnit

[3] https://coverage.postgresql.org/src/backend/access/gin/ginpostinglist.c.gcov.html

Re: C testing for Postgres

From
Dmitry Dolgov
Date:
> On Fri, Jun 28, 2019 at 11:38 AM Adam Berlin <aberlin@pivotal.io> wrote:
>
> During the unconference at PGCon in Ottawa, I asked about writing C-based
> tests for Postgres. There was interest in trying a tool and also some
> hesitation to depend on a third-party library. So, I wrote a tool that I'd
> like to contribute to Postgres. I’ve been calling it cexpect [1].

Cool, thanks!

> Rather than post a patch, I'd rather start a conversation first. I'm guessing
> there are some improvements that we'd want to make (for example: the
> Makefile) before commiting a patch. Let's iterate on improvements before
> creating a formal patch.

Just to mention, there were similar discussions already in the past ([1], [2]),
with some concerns being raised, but looks like without any visible results.

[1]:
https://www.postgresql.org/message-id/flat/CAEepm%3D2heu%2B5zwB65jWap3XY-UP6PpJZiKLQRSV2UQH9BmVRXQ%40mail.gmail.com
[2]: https://www.postgresql.org/message-id/flat/Pine.LNX.4.58.0410111044030.14840%40linuxworld.com.au



Re: C testing for Postgres

From
Adam Berlin
Date:
Here are my takeaways from the previous discussions:

* there *is* interest in testing
* we shouldn't take it too far
* there are already tests being written under `src/test/modules`, but without a consistent way of describing expectations and displaying results
* no tool was chosen

If we were to use this tool, would the community want to vendor the framework in the Postgres repository, or keep it in a separate repository that produces a versioned shared library?

On Fri, Jun 28, 2019 at 5:57 AM Dmitry Dolgov <9erthalion6@gmail.com> wrote:
> On Fri, Jun 28, 2019 at 11:38 AM Adam Berlin <aberlin@pivotal.io> wrote:
>
> During the unconference at PGCon in Ottawa, I asked about writing C-based
> tests for Postgres. There was interest in trying a tool and also some
> hesitation to depend on a third-party library. So, I wrote a tool that I'd
> like to contribute to Postgres. I’ve been calling it cexpect [1].

Cool, thanks!

> Rather than post a patch, I'd rather start a conversation first. I'm guessing
> there are some improvements that we'd want to make (for example: the
> Makefile) before commiting a patch. Let's iterate on improvements before
> creating a formal patch.

Just to mention, there were similar discussions already in the past ([1], [2]),
with some concerns being raised, but looks like without any visible results.

[1]: https://www.postgresql.org/message-id/flat/CAEepm%3D2heu%2B5zwB65jWap3XY-UP6PpJZiKLQRSV2UQH9BmVRXQ%40mail.gmail.com
[2]: https://www.postgresql.org/message-id/flat/Pine.LNX.4.58.0410111044030.14840%40linuxworld.com.au

Re: C testing for Postgres

From
David Fetter
Date:
On Fri, Jun 28, 2019 at 09:42:54AM -0400, Adam Berlin wrote:
> Here are my takeaways from the previous discussions:
> 
> * there *is* interest in testing

Yep.

> * we shouldn't take it too far
> * there are already tests being written under `src/test/modules`, but
> without a consistent way of describing expectations and displaying results

This is a giant problem.

> * no tool was chosen

If there's a way to get this in the tree, assuming people agree it
should be there, that'd be fantastic.

Our current system has been creaking for years.

Best,
David.
-- 
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate



Re: C testing for Postgres

From
Jesse Zhang
Date:
On Fri, Jun 28, 2019 at 10:37 AM Adam Berlin <aberlin@pivotal.io> wrote:
>
> If we were to use this tool, would the community want to vendor the
> framework in the Postgres repository, or keep it in a separate
> repository that produces a versioned shared library?
>

If the library is going to actively evolve, we should bring it into the
tree. For a project like this, a "versioned shared library" is a massive
pain in the rear for both the consumer of such libraries and for their
maintainers.

Cheers,
Jesse



Re: C testing for Postgres

From
Michael Paquier
Date:
On Fri, Jun 28, 2019 at 09:42:54AM -0400, Adam Berlin wrote:
> If we were to use this tool, would the community want to vendor the
> framework in the Postgres repository, or keep it in a separate repository
> that produces a versioned shared library?

Well, my take is that having a base infrastructure for a fault
injection framework is something that would prove to be helpful, and
that I am not against having something in core.  While working on
various issues, I have found myself doing many times crazy stat()
calls on an on-disk file to enforce an elog(ERROR) or elog(FATAL), and
by experience fault points are things very *hard* to place correctly
because they should not be single-purpose things.

Now, we don't want to finish with an infinity of fault points in the
tree, but being able to enforce a failure in a point added for a patch
using a SQL command can make the integration of tests in a patch
easier for reviewers, for example isolation tests with elog(ERROR)
(like what has been discussed for b4721f3).
--
Michael

Attachment

Re: C testing for Postgres

From
Ashwin Agrawal
Date:

On Mon, Jul 1, 2019 at 11:26 PM Michael Paquier <michael@paquier.xyz> wrote:
On Fri, Jun 28, 2019 at 09:42:54AM -0400, Adam Berlin wrote:
> If we were to use this tool, would the community want to vendor the
> framework in the Postgres repository, or keep it in a separate repository
> that produces a versioned shared library?

Well, my take is that having a base infrastructure for a fault
injection framework is something that would prove to be helpful, and
that I am not against having something in core.  While working on
various issues, I have found myself doing many times crazy stat()
calls on an on-disk file to enforce an elog(ERROR) or elog(FATAL), and
by experience fault points are things very *hard* to place correctly
because they should not be single-purpose things.

Now, we don't want to finish with an infinity of fault points in the
tree, but being able to enforce a failure in a point added for a patch
using a SQL command can make the integration of tests in a patch
easier for reviewers, for example isolation tests with elog(ERROR)
(like what has been discussed for b4721f3).

Just to clarify what Adam is proposing in this thread is *not* a fault injection framework.

Re: C testing for Postgres

From
Adam Berlin
Date:
Just to clarify what Adam is proposing in this thread is *not* a fault injection framework.

Yes, thanks for clarifying Ashwin. 

Sorry Michael, this testing framework is more like these other frameworks:

Java with Junit + Hamcrest: http://hamcrest.org/JavaHamcrest/tutorial
Javascript with Jasmine: https://jasmine.github.io/