Re: BUG #15519: Casting float4 into int4 gets the wrong sign insteadof "integer out of range" error - Mailing list pgsql-bugs
| From | Victor Petrovykh |
|---|---|
| Subject | Re: BUG #15519: Casting float4 into int4 gets the wrong sign insteadof "integer out of range" error |
| Date | |
| Msg-id | CAJ-A5aOFf8XDiLhEtaKHkZYK-nNcvbrBMARSSOtGqMK1RMdPqA@mail.gmail.com Whole thread Raw |
| In response to | Re: BUG #15519: Casting float4 into int4 gets the wrong sign instead of "integer out of range" error (Tom Lane <tgl@sss.pgh.pa.us>) |
| Responses |
Re: BUG #15519: Casting float4 into int4 gets the wrong sign instead of "integer out of range" error
|
| List | pgsql-bugs |
Thanks for the clarification.
I don't think I make any unsafe architecture assumptions in the above. As far as I can tell the only such assumption is about likely values of X and Y being 8, 16, 32, 64, 128, etc. and that the mantissa of a float is using more than half of its bits. I'm assuming that casting function knows the 2 actual types so it can make a deterministic decision about the comparison it must use.
I was thinking about the assumption whether or not INT_MAX is representable perfectly as a float. Consider any combination of types int_X and float_Y where X and Y denote the number of bits used for representation:
- for any type int_X it is always true that its maximum value cannot be represented by a float_Y if X >= Y because at least one bit of the float_Y must be used to represent the exponent part of the float (else the float_Y is indistinguishable from int_Y)
- for any type int_X it is always possible to represent the maximum value (and by extension any other value) exactly as a float_Y if X <= M(Y), meaning that if the mantissa of the float_Y has at least as many bits as X. In practice X and Y will be some form of 8, 16, 32, 64, etc. so 2X <= Y and unless the mantissa is less than half of the significant bits of the float we will have x <= M(Y) for any practical X < Y
- for any int_X and float_Y there exists a specific value FLOATY, such that for all possible float_Y values y < FLOATY it is also true that y <= MAX_INT_X and for all y >= FLOATY it is also true that y > MAX_INT (basically I can always pick a float threshold above which all numbers would be above MAX_INT and therefore out of range and below it would be guaranteed to be representable as ints). I conjecture that this special value FLOATY = (float_Y)MAX_INT_X for any X >= Y. I think I can formally prove this conjecture, basically it has to do with sparseness of float values when compared to MAX_INT_X + 1 and MAX_INT_X - 1.
So it seems to me that the rule for casting would depend on whether the float has same or fewer bits than the int or not:
- when the int has same or more bits (e.g. float4 -> int4 or float4 -> int8)
if (num < INT_MIN || num >= INT_MAX || isnan(num))
- when the int has fewer bits (e.g. float4 -> int2 or float8 -> int4)
if (num < INT_MIN || num > INT_MAX || isnan(num))
I have one more related question: is a fix for this likely to appear in the next Postgres release?
On Fri, Nov 23, 2018 at 8:42 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Victor Petrovykh <victor@magic.io> writes:
> Am I missing something in thinking that the cast of a float to int should
> produce an error if the sign of the original value doesn't match the sign
> of the cast result?
Well, yeah, our traditional way of coding this overflow check would
probably have compared the sign of the result to the sign of the input,
but (a) we'd still have needed some ad-hoc range check to avoid getting
fooled by inputs large enough to have wrapped around an even number of
times, and (b) this approach depends on the compiler not taking any
liberties based on an assumption that the program doesn't provoke
integer overflow. (b) gets more worrisome with each year that goes by,
because the compiler guys keep finding ever-more-creative ways to break
your code if it violates C-spec semantics. So we really want to write
a test that will fail exactly when the integer coercion would overflow,
not do the coercion and then check to see if it overflowed.
regards, tom lane
pgsql-bugs by date: