Testing pg_terminate_backend() - Mailing list pgsql-patches

bruce wrote:
> Tom Lane wrote:
> > Bruce Momjian <bruce@momjian.us> writes:
> > > Tom Lane wrote:
> > >> The closest thing I can think of to an automated test is to run repeated
> > >> sets of the parallel regression tests, and each time SIGTERM a randomly
> > >> chosen backend at a randomly chosen time.  Then see if anything "funny"
> >
> > > Yep, that was my plan, plus running the parallel regression tests you
> > > get the possibility of >2 backends.
> >
> > I was intentionally suggesting only one kill per test cycle.  Multiple
> > kills will probably create an O(N^2) explosion in the set of possible
> > downstream-failure deltas.  I doubt you'd really get any improvement
> > in testing coverage to justify the much larger amount of hand validation
> > needed.
> >
> > It also strikes me that you could make some simple alterations to the
> > regression tests to reduce the set of observable downstream deltas.
> > For example, anyplace where a test loads a table with successive INSERTs
> > and that table is used by later tests, wrap the INSERT sequence with
> > BEGIN/END.  Then there is only one possible downstream delta (empty
> > table) and not N different possibilities for an N-row table.
>
> I have added pg_terminate_backend() to use SIGTERM and will start
> running tests as discussed with Tom.  I will post my scripts too.

Attached is my test script.   I ran it for 14 hours (asserts on),
running 450 regression tests, with up to seven backends killed per
regression test.

I have processed the combined regression.diffs files by pickouting out
all the new error messages.  I don't see anything unusual in there.

Should I run it differently?

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +
#!/bin/bash

REGRESSION_DURATION=80    # average duration of regression test in seconds
OUTFILE=/rtmp/regression.sigterm

# To analyze output, use:
# grep '^\+ *[A-Z][A-Z]*:' /rtmp/regression.sigterm | sort | uniq | less


cd /pg/test/regress

while :
do
    (
        SLEEP=`expr $RANDOM \* $REGRESSION_DURATION / 32767`
        echo "Sleeping $SLEEP seconds"
        sleep "$SLEEP"
        echo "Trying kill"
        # send up to 7 kill signals
        for X in 1 2 3 4 5 6 7
        do
            psql -p 55432 -qt -c "
                SELECT pg_terminate_backend(stat.procpid)
                FROM (SELECT procpid FROM pg_stat_activity
                ORDER BY random() LIMIT 1) AS stat
                " template1 2> /dev/null
            if [ "$?" -eq 0 ]
            then    echo "Kill sent"
            fi
            sleep 5
        done
    ) &
    gmake check
    wait
    [ -s regression.diffs ] && cat regression.diffs >> "$OUTFILE"
done
+ CONTEXT:  COPY bool_test, line 1: "TRUE    null    FALSE    null"
+ CONTEXT:  COPY create_table_test, line 1: "5    10"
+ CONTEXT:  COPY x, line 0: ""
+ CONTEXT:  COPY x, line 1: "1    test_1"
+ CONTEXT:  COPY x, line 1: "4000:\X:C:\X:\X"
+ CONTEXT:  SQL function "declares_cursor"
+ CONTEXT:  SQL function "max_xacttest"
+ ERROR:  INSERT has more expressions than target columns
+ ERROR:  cannot drop table test1 because other objects depend on it
+ ERROR:  column "a" of relation "xacttest" does not exist
+ ERROR:  column "f1" does not exist
+ ERROR:  column "fooid" does not exist
+ ERROR:  current transaction is aborted, commands ignored until end of transaction block
+ ERROR:  cursor "foo25" does not exist
+ ERROR:  function getfoo(integer) does not exist
+ ERROR:  index "onek_nulltest" does not exist
+ ERROR:  index "wowidx" does not exist
+ ERROR:  no such savepoint
+ ERROR:  prepared statement "q5" does not exist
+ ERROR:  relation "a_star" does not exist
+ ERROR:  relation "aggtest" does not exist
+ ERROR:  relation "array_index_op_test" does not exist
+ ERROR:  relation "array_op_test" does not exist
+ ERROR:  relation "b_star" does not exist
+ ERROR:  relation "bt_f8_heap" does not exist
+ ERROR:  relation "bt_i4_heap" does not exist
+ ERROR:  relation "bt_name_heap" does not exist
+ ERROR:  relation "bt_txt_heap" does not exist
+ ERROR:  relation "c_star" does not exist
+ ERROR:  relation "d_star" does not exist
+ ERROR:  relation "e_star" does not exist
+ ERROR:  relation "emp" does not exist
+ ERROR:  relation "equipment_r" does not exist
+ ERROR:  relation "f_star" does not exist
+ ERROR:  relation "fast_emp4000" does not exist
+ ERROR:  relation "foo" already exists
+ ERROR:  relation "gcircle_tbl" does not exist
+ ERROR:  relation "gpolygon_tbl" does not exist
+ ERROR:  relation "hash_f8_heap" does not exist
+ ERROR:  relation "hash_i4_heap" does not exist
+ ERROR:  relation "hash_name_heap" does not exist
+ ERROR:  relation "hash_txt_heap" does not exist
+ ERROR:  relation "hobbies_r" does not exist
+ ERROR:  relation "ihighway" does not exist
+ ERROR:  relation "int2_tbl" does not exist
+ ERROR:  relation "int8_tbl" does not exist
+ ERROR:  relation "onek" does not exist
+ ERROR:  relation "onek2" does not exist
+ ERROR:  relation "onek_unique1" does not exist
+ ERROR:  relation "onek_with_null" does not exist
+ ERROR:  relation "pcachetest" does not exist
+ ERROR:  relation "pcacheview" does not exist
+ ERROR:  relation "person" does not exist
+ ERROR:  relation "pg_toast_stud_emp" does not exist
+ ERROR:  relation "polygon_tbl" does not exist
+ ERROR:  relation "ramp" does not exist
+ ERROR:  relation "random_tbl" does not exist
+ ERROR:  relation "real_city" does not exist
+ ERROR:  relation "road" does not exist
+ ERROR:  relation "shighway" does not exist
+ ERROR:  relation "six" does not exist
+ ERROR:  relation "slow_emp4000" does not exist
+ ERROR:  relation "stud_emp" does not exist
+ ERROR:  relation "student" does not exist
+ ERROR:  relation "tenk1" does not exist
+ ERROR:  relation "tenk2" does not exist
+ ERROR:  relation "test1" already exists
+ ERROR:  relation "test2" already exists
+ ERROR:  relation "test_tsvector" does not exist
+ ERROR:  relation "tmp" already exists
+ ERROR:  relation "tmp" does not exist
+ ERROR:  relation "tmp_onek_unique1" does not exist
+ ERROR:  relation "tmp_view" does not exist
+ ERROR:  relation "toyemp" does not exist
+ ERROR:  relation "xacttest" does not exist
+ ERROR:  rule "foorule" for relation "foo" does not exist
+ ERROR:  table "onek_with_null" does not exist
+ ERROR:  table "pcachetest" does not exist
+ ERROR:  table "tmp" does not exist
+ ERROR:  table "tmp1" does not exist
+ ERROR:  type "city_budget" is only a shell
+ ERROR:  type "hobbies_r" does not exist
+ ERROR:  type "widget" is only a shell
+ ERROR:  type emp does not exist
+ ERROR:  type hobbies_r does not exist
+ ERROR:  type person does not exist
+ ERROR:  view "tmp_view_new" does not exist
+ ERROR:  view "vw_getfoo" does not exist
+ FATAL:  terminating connection due to administrator command
+ HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
+ HINT:  Use DROP ... CASCADE to drop the dependent objects too.
+ NOTICE:  argument type widget is only a shell
+ NOTICE:  rule _RETURN on view v_test1 depends on table test1
+ NOTICE:  view v_test1 depends on rule _RETURN on view v_test1