Re: Greatest Common Divisor - Mailing list pgsql-hackers

From Dean Rasheed
Subject Re: Greatest Common Divisor
Date
Msg-id CAEZATCX9YgkWupVFcJ5ixkEJMyf2X5ZecN8K5r12oOB8mDHzWQ@mail.gmail.com
Whole thread Raw
In response to Re: Greatest Common Divisor  (Fabien COELHO <coelho@cri.ensmp.fr>)
Responses Re: Greatest Common Divisor
Re: Greatest Common Divisor
Re: Greatest Common Divisor
List pgsql-hackers
Out of curiosity, what was the original use-case for this?

I'm not objecting to adding it, I'm just curious. In fact, I think
that if we do add this, then we should probably add lcm() at the same
time, since handling its overflow cases is sufficiently non-trivial to
justify not requiring users to have to implement it themselves.

I don't like the INT_MIN handling though:

select gcd(-2147483648,0);
     gcd
-------------
 -2147483648
(1 row)

select gcd(-2147483648,-2147483648);
     gcd
-------------
 -2147483648
(1 row)

Normally gcd() returns a positive integer, and gcd(a,0) = gcd(a,a) =
abs(a). But since abs(INT_MIN) cannot be represented as a 32-bit
integer, both those cases should throw an integer-out-of-range error.

In addition, the following case should produce 1, but for me it
produces an error. This is actually going to be platform-dependent as
it is currently implemented (see the comments in int4div and int4mod):

select gcd(-2147483648,-1);
ERROR:  floating-point exception
DETAIL:  An invalid floating-point operation was signaled. This
probably means an out-of-range result or an invalid operation, such as
division by zero.

so there needs to be some special-case INT_MIN handling at the start
to deal with these cases.

AFAIK, what gcd(0,0) should be is not well defined, but most common
implementations seem to return 0 (I checked Matlab, python and Java's
standard libraries). This seems like a reasonable extrapolation of the
rule gcd(a,0) = gcd(0,a) = gcd(a,a) = abs(a), so I don't have a
problem with doing the same here, but I think that it should be
documented (e.g., see [1]), if for no other reason than users might
expect it to be safe to divide by the result.

Regards,
Dean

[1] https://www.mathworks.com/help/matlab/ref/gcd.html



pgsql-hackers by date:

Previous
From: Stephen Frost
Date:
Subject: Re: TRUNCATE on foreign tables
Next
From: Stephen Frost
Date:
Subject: Re: Greatest Common Divisor