[sqlsmith] Infinite recursion in bitshift - Mailing list pgsql-hackers

Hi,

sqlsmith just found another crasher:

    select bit '1' >> (-2^31)::int;

This is due to an integer overflow in bitshiftright()/bitshiftleft()
leading to them recursively calling each other.  Patch attached.

regards,
Andreas
>From cfdc425f75da268e1c2af08f936c59f34b69e577 Mon Sep 17 00:00:00 2001
From: Andreas Seltenreich <seltenreich@gmx.de>
Date: Fri, 14 Oct 2016 20:52:52 +0200
Subject: [PATCH] Fix possible infinite recursion on bitshift.

bitshiftright() and bitshiftleft() would recursively call each other
infinitely when the user passed a MIN_INT for the shift amount due to
an integer overflow.  This patch reduces a negative shift amount to
-VARBITMAXLEN to avoid overflow.
---
 src/backend/utils/adt/varbit.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/src/backend/utils/adt/varbit.c b/src/backend/utils/adt/varbit.c
index 75e6a46..d8ecfb6 100644
--- a/src/backend/utils/adt/varbit.c
+++ b/src/backend/utils/adt/varbit.c
@@ -1387,9 +1387,15 @@ bitshiftleft(PG_FUNCTION_ARGS)

     /* Negative shift is a shift to the right */
     if (shft < 0)
+    {
+        /* Protect against overflow */
+        if (shft < -VARBITMAXLEN)
+            shft = -VARBITMAXLEN;
+
         PG_RETURN_DATUM(DirectFunctionCall2(bitshiftright,
                                             VarBitPGetDatum(arg),
                                             Int32GetDatum(-shft)));
+    }

     result = (VarBit *) palloc(VARSIZE(arg));
     SET_VARSIZE(result, VARSIZE(arg));
@@ -1447,9 +1453,15 @@ bitshiftright(PG_FUNCTION_ARGS)

     /* Negative shift is a shift to the left */
     if (shft < 0)
+    {
+        /* Protect against overflow */
+        if (shft < -VARBITMAXLEN)
+            shft = -VARBITMAXLEN;
+
         PG_RETURN_DATUM(DirectFunctionCall2(bitshiftleft,
                                             VarBitPGetDatum(arg),
                                             Int32GetDatum(-shft)));
+    }

     result = (VarBit *) palloc(VARSIZE(arg));
     SET_VARSIZE(result, VARSIZE(arg));
--
2.9.3