Minor fail in realfailN scanner rules - Mailing list pgsql-hackers

While fooling with John Naylor's ecpg lexer sync patch, my attention
was drawn to the "realfail1" flex rule, which triggers when we see
digits immediately followed by "e", but no exponent after that:

{realfail1}        {
                    /*
                     * throw back the [Ee], and treat as {decimal}.  Note
                     * that it is possible the input is actually {integer},
                     * but since this case will almost certainly lead to a
                     * syntax error anyway, we don't bother to distinguish.
                     */
                    yyless(yyleng - 1);
                    SET_YYLLOC();
                    yylval->str = pstrdup(yytext);
                    return FCONST;
                }

I think that code and comment are mine, but I realized that it's overly
optimistic to suppose that the situation can't happen.  Consider

SELECT 42efoo, 45 ebar;
 efoo | ebar 
------+------
   42 |   45
(1 row)

The first target item is lexed as FCONST then IDENT because of what
realfail1 has done, while the second one is lexed as ICONST then IDENT.

This is not great.  It happens to work anyway -- that is, the first
column is deemed to be int4 not numeric -- because make_const() is very
paranoid about what it might find in a T_Float constant.  But it might
well be that there are other syntactic contexts in which returning FCONST
would result in parse errors or unexpected behavior.

Fortunately, this doesn't really take any extra code to fix; we can
do something like the attached.  psql and ecpg should be corrected
to match, although it's certainly just cosmetic for psql, and probably
also for ecpg.

            regards, tom lane

diff --git a/src/backend/parser/scan.l b/src/backend/parser/scan.l
index 6c6a6e3..74e34df 100644
--- a/src/backend/parser/scan.l
+++ b/src/backend/parser/scan.l
@@ -1005,22 +1005,18 @@ other            .
                 }
 {realfail1}        {
                     /*
-                     * throw back the [Ee], and treat as {decimal}.  Note
-                     * that it is possible the input is actually {integer},
-                     * but since this case will almost certainly lead to a
-                     * syntax error anyway, we don't bother to distinguish.
+                     * throw back the [Ee], and figure out whether what
+                     * remains is an {integer} or {decimal}.
                      */
                     yyless(yyleng - 1);
                     SET_YYLLOC();
-                    yylval->str = pstrdup(yytext);
-                    return FCONST;
+                    return process_integer_literal(yytext, yylval);
                 }
 {realfail2}        {
                     /* throw back the [Ee][+-], and proceed as above */
                     yyless(yyleng - 2);
                     SET_YYLLOC();
-                    yylval->str = pstrdup(yytext);
-                    return FCONST;
+                    return process_integer_literal(yytext, yylval);
                 }


@@ -1255,6 +1251,10 @@ litbufdup(core_yyscan_t yyscanner)
     return new;
 }

+/*
+ * Process {integer}.  Note this will also do the right thing with {decimal},
+ * ie digits and a decimal point.
+ */
 static int
 process_integer_literal(const char *token, YYSTYPE *lval)
 {
@@ -1265,7 +1265,7 @@ process_integer_literal(const char *token, YYSTYPE *lval)
     val = strtoint(token, &endptr, 10);
     if (*endptr != '\0' || errno == ERANGE)
     {
-        /* integer too large, treat it as a float */
+        /* integer too large (or contains decimal pt), treat it as a float */
         lval->str = pstrdup(token);
         return FCONST;
     }