Thread: BUG #18240: Undefined behaviour in cash_mul_flt8() and friends
The following bug has been logged on the website:
Bug reference: 18240
Logged by: Alexander Lakhin
Email address: exclusion@gmail.com
PostgreSQL version: 16.1
Operating system: Ubuntu 22.04
Description:
The following multiplication:
SELECT 1_000_000_000::money * 1_000_000_000::float8;
gives the incorrect result:
-$92,233,720,368,547,758.08
UBSan detects undefined behaviour:
cash.c:669:11: runtime error: 1e+20 is outside the range of representable
values of type 'long'
#0 0x55d66011b73a in cash_mul_flt8
.../src/backend/utils/adt/cash.c:669:11
The same is observed with float4:
cash.c:719:11: runtime error: 1e+20 is outside the range of representable
values of type 'long'
#0 0x55f2adc46072 in cash_mul_flt4
.../src/backend/utils/adt/cash.c:719:11
And with float8 * money...
Reproduced on REL9_6_0 (but the defect is much older, AFAICS) .. HEAD.
On Mon, Dec 11, 2023 at 06:00:02AM +0000, PG Bug reporting form wrote: > The following multiplication: > SELECT 1_000_000_000::money * 1_000_000_000::float8; > > gives the incorrect result: > -$92,233,720,368,547,758.08 Yep, good catch. Reproduced here. -- Michael
Attachment
Michael Paquier <michael@paquier.xyz> writes:
> On Mon, Dec 11, 2023 at 06:00:02AM +0000, PG Bug reporting form wrote:
>> The following multiplication:
>> SELECT 1_000_000_000::money * 1_000_000_000::float8;
>> gives the incorrect result:
>> -$92,233,720,368,547,758.08
> Yep, good catch. Reproduced here.
Yeah, approximately none of cash.c pays any attention to the risks
of overflow/underflow. Improving that situation would be a good
finger exercise for some aspiring hacker, perhaps. Although I bet
somebody will ask again why it is that we continue to support the
money type.
regards, tom lane
Hello Michael and Tom,
11.12.2023 18:05, Tom Lane wrote:
> Michael Paquier <michael@paquier.xyz> writes:
>> On Mon, Dec 11, 2023 at 06:00:02AM +0000, PG Bug reporting form wrote:
>>> The following multiplication:
>>> SELECT 1_000_000_000::money * 1_000_000_000::float8;
>>> gives the incorrect result:
>>> -$92,233,720,368,547,758.08
>> Yep, good catch. Reproduced here.
> Yeah, approximately none of cash.c pays any attention to the risks
> of overflow/underflow. Improving that situation would be a good
> finger exercise for some aspiring hacker, perhaps. Although I bet
> somebody will ask again why it is that we continue to support the
> money type.
Thank you for looking at that!
I've tried a simple operator-testing query (with already known problematic
operators spelled out):
SELECT format('SELECT ''2000000000''::%s %s ''2000000000''::%s;',
tl.typname, oprname, tr.typname) FROM pg_operator
INNER JOIN pg_type tl ON tl.oid = oprleft
INNER JOIN pg_type tr ON tr.oid = oprright WHERE (
/* 2000000000 xxx 2000000000 */
NOT(tl.typname = 'money' AND oprname = '*' AND tr.typname = 'float4') AND
NOT(tl.typname = 'float4' AND oprname = '*' AND tr.typname = 'money') AND
NOT(tl.typname = 'money' AND oprname = '*' AND tr.typname = 'float8') AND
NOT(tl.typname = 'float8' AND oprname = '*' AND tr.typname = 'money') AND
NOT(tl.typname = 'int4' AND oprname = '<<' AND tr.typname = 'int4') AND
NOT(tl.typname = 'int4' AND oprname = '>>' AND tr.typname = 'int4') AND
NOT(tl.typname = 'int8' AND oprname = '<<' AND tr.typname = 'int4') AND
NOT(tl.typname = 'int8' AND oprname = '>>' AND tr.typname = 'int4') AND
/* 2000000000 xxx 0.0000000002 */
NOT(tl.typname = 'money' AND oprname = '/' AND tr.typname = 'float4') AND
NOT(tl.typname = 'money' AND oprname = '/' AND tr.typname = 'float8') AND
/* 32767 xxx 32767 */
NOT(tl.typname = 'int2' AND oprname = '<<' AND tr.typname = 'int4') AND
NOT(tl.typname = 'int2' AND oprname = '>>' AND tr.typname = 'int4')
)
\gexec
and found no new UBSan-detected anomalies with "extraordinary" numbers.
Best regards,
Alexander
On Mon, Dec 11, 2023 at 10:05:53AM -0500, Tom Lane wrote: > Yeah, approximately none of cash.c pays any attention to the risks > of overflow/underflow. Improving that situation would be a good > finger exercise for some aspiring hacker, perhaps. Although I bet > somebody will ask again why it is that we continue to support the > money type. AFAIK, we discourage the use of money in the wiki for quite a few years: https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money And numeric has much better code coverage and support. I am wondering whether we've reached the point where it would be better to remove it entirely from the tree, and just tell people to use numeric. This has a cost for upgrades, where we should cross check for its use but there is already check_for_data_type_usage() to do this job so the facility is there. -- Michael
Attachment
On Tue, Dec 12, 2023 at 10:03:43AM +0100, Michael Paquier wrote: > On Mon, Dec 11, 2023 at 10:05:53AM -0500, Tom Lane wrote: > > Yeah, approximately none of cash.c pays any attention to the risks > > of overflow/underflow. Improving that situation would be a good > > finger exercise for some aspiring hacker, perhaps. Although I bet > > somebody will ask again why it is that we continue to support the > > money type. > > AFAIK, we discourage the use of money in the wiki for quite a few > years: > https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money > > And numeric has much better code coverage and support. I am wondering > whether we've reached the point where it would be better to remove it > entirely from the tree, and just tell people to use numeric. This has > a cost for upgrades, where we should cross check for its use but there > is already check_for_data_type_usage() to do this job so the facility > is there. Yes, probably. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, 12 Dec 2023 at 22:03, Michael Paquier <michael@paquier.xyz> wrote: > AFAIK, we discourage the use of money in the wiki for quite a few > years: > https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money > > And numeric has much better code coverage and support. I am wondering > whether we've reached the point where it would be better to remove it > entirely from the tree, and just tell people to use numeric. This has > a cost for upgrades, where we should cross check for its use but there > is already check_for_data_type_usage() to do this job so the facility > is there. We could extract it into a contrib module. That might help reduce new usages of it and would also allow people who have large tables using the money type but can't realistically wait out a table rewrite to upgrade to a newer version of Postgres. David
David Rowley <dgrowleyml@gmail.com> writes:
> On Tue, 12 Dec 2023 at 22:03, Michael Paquier <michael@paquier.xyz> wrote:
>> And numeric has much better code coverage and support. I am wondering
>> whether we've reached the point where it would be better to remove it
>> entirely from the tree, and just tell people to use numeric.
> We could extract it into a contrib module.
Perhaps, but ...
> That might help reduce new usages of it and would also allow people
> who have large tables using the money type but can't realistically
> wait out a table rewrite to upgrade to a newer version of Postgres.
... I doubt that a contrib module would solve the problem for people
who can't afford a rewrite. pg_upgrade requires that datatype OIDs
stay the same, which is something I don't believe a contrib module
could manage. We've run into that before if memory serves. Is it
time to do something about that? Perhaps we could allow extension
modules to use binary_upgrade_set_next_pg_type_oid, and then somehow
reserve the money and _money OIDs forever?
regards, tom lane
On Tue, Dec 12, 2023 at 09:37:58AM -0500, Tom Lane wrote: > ... I doubt that a contrib module would solve the problem for people > who can't afford a rewrite. pg_upgrade requires that datatype OIDs > stay the same, which is something I don't believe a contrib module > could manage. We've run into that before if memory serves. Is it > time to do something about that? Perhaps we could allow extension > modules to use binary_upgrade_set_next_pg_type_oid, and then somehow > reserve the money and _money OIDs forever? You mean with an integration in binary dumps? I don't see why it would not work and this facility may prove to be useful for out-of-core extension, assuming that this can be made transparent based on some new .control properties on the upgrade cluster, I guess.. Now, I am not sure that we're making users a favor in keeping around a data type that's kind of obsolete these days, particularly when all of them basically require handling of fractional cents when doing serious calculations. That's a trap in disguise, even if tying data to a locale for the unit sounds appealing for some applications. -- Michael
Attachment
Michael Paquier <michael@paquier.xyz> writes:
> On Tue, Dec 12, 2023 at 09:37:58AM -0500, Tom Lane wrote:
>> ... Perhaps we could allow extension
>> modules to use binary_upgrade_set_next_pg_type_oid, and then somehow
>> reserve the money and _money OIDs forever?
> Now, I am not sure that we're making users a favor in keeping around a
> data type that's kind of obsolete these days, particularly when all of
> them basically require handling of fractional cents when doing serious
> calculations. That's a trap in disguise, even if tying data to a
> locale for the unit sounds appealing for some applications.
Yeah, I wouldn't propose this if money were the only type we were
thinking of getting rid of; likely not worth the trouble. But I've
got one eye on the geometric types as well --- I think it would be
great all around if we could kick those out of the core distribution
and let them be maintained independently. But I fear we can't do that
without some method whereby pg_upgrade is still possible across the
change.
regards, tom lane
On 12/16/23 10:19 AM, Tom Lane wrote: > Michael Paquier <michael@paquier.xyz> writes: >> On Tue, Dec 12, 2023 at 09:37:58AM -0500, Tom Lane wrote: >>> ... Perhaps we could allow extension >>> modules to use binary_upgrade_set_next_pg_type_oid, and then somehow >>> reserve the money and _money OIDs forever? > >> Now, I am not sure that we're making users a favor in keeping around a >> data type that's kind of obsolete these days, particularly when all of >> them basically require handling of fractional cents when doing serious >> calculations. That's a trap in disguise, even if tying data to a >> locale for the unit sounds appealing for some applications. > > Yeah, I wouldn't propose this if money were the only type we were > thinking of getting rid of; likely not worth the trouble. But I've > got one eye on the geometric types as well --- I think it would be > great all around if we could kick those out of the core distribution > and let them be maintained independently. But I fear we can't do that > without some method whereby pg_upgrade is still possible across the > change. "money" poses a bit more risk than the geometric types (I do want to come back to geometric types later), and it'd be good if we could make some headway on removing it. Just this past week I came across a case where "money" was being discussed as being valid, and had to give the "it's kinda, sorta deprecated" talk. I'm of the strong opinion that we should get rid of money. I personally haven't encountered it in the wild -- I'm sure it's there, but it seems limited -- and most apps that seriously deal with money will either user either "numeric" or an integer-based type. I am sensitive to the upgrade piece, as we don't want someone relying on that behavior to be indefinitely stuck. But perhaps part of the deprecation plan is to just keep the casts around[1] for a few releases, without exposing the type, and prevent new creations of the type? We do state that "A money value can be cast to numeric without loss of precision"[1], but I do agree with the concerns on a pg_upgrade operation being stuck on a costly rewrite operation. Would it make sense to block an upgrade via pg_upgrade if we detect a money type unless it's rewritten to a different type? Users should still be able to user other upgrades methods if need be, and handle the rewrite during that time. Jonathan [1] https://www.postgresql.org/docs/current/datatype-money.html
Attachment
"Jonathan S. Katz" <jkatz@postgresql.org> writes:
> I'm of the strong opinion that we should get rid of money. I personally
> haven't encountered it in the wild -- I'm sure it's there, but it seems
> limited -- and most apps that seriously deal with money will either user
> either "numeric" or an integer-based type.
Yeah, maybe we should just do it. I was pleasantly surprised by how
little push-back we got from nuking the 32-bit datetime types a few
releases ago; perhaps this one would likewise not have much of a
constituency.
> I am sensitive to the upgrade piece, as we don't want someone relying on
> that behavior to be indefinitely stuck. But perhaps part of the
> deprecation plan is to just keep the casts around[1] for a few releases,
> without exposing the type, and prevent new creations of the type?
I don't really see a way to do that, especially not if we don't want
to put a large amount of effort into it. We can certainly make
pg_upgrade reject "money" columns, and tell people they need to
rewrite those before they upgrade not after.
regards, tom lane
On Sun, Dec 24, 2023 at 01:21:15PM -0500, Tom Lane wrote: > "Jonathan S. Katz" <jkatz@postgresql.org> writes: >> I'm of the strong opinion that we should get rid of money. I personally >> haven't encountered it in the wild -- I'm sure it's there, but it seems >> limited -- and most apps that seriously deal with money will either user >> either "numeric" or an integer-based type. > > Yeah, maybe we should just do it. I was pleasantly surprised by how > little push-back we got from nuking the 32-bit datetime types a few > releases ago; perhaps this one would likewise not have much of a > constituency. Yeah, that's hard to say. Perhaps it would a good idea to ask more widely how much people have been relying on that and how large these data sets are. Another thing that we could attempt is doing something about the the table rewrites on type changes to require an EOL. Like more CONCURRENTLY flavors in ALTER TABLE? >> I am sensitive to the upgrade piece, as we don't want someone relying on >> that behavior to be indefinitely stuck. But perhaps part of the >> deprecation plan is to just keep the casts around[1] for a few releases, >> without exposing the type, and prevent new creations of the type? > > I don't really see a way to do that, especially not if we don't want > to put a large amount of effort into it. We can certainly make > pg_upgrade reject "money" columns, and tell people they need to > rewrite those before they upgrade not after. At least the infra to be able to achieve that should be here. -- Michael
Attachment
On Sun, 2023-12-24 at 13:21 -0500, Tom Lane wrote: > "Jonathan S. Katz" <jkatz@postgresql.org> writes: > > I'm of the strong opinion that we should get rid of money. I personally > > haven't encountered it in the wild -- I'm sure it's there, but it seems > > limited -- and most apps that seriously deal with money will either user > > either "numeric" or an integer-based type. > > Yeah, maybe we should just do it. I was pleasantly surprised by how > little push-back we got from nuking the 32-bit datetime types a few > releases ago; perhaps this one would likewise not have much of a > constituency. There are questions on Stackoverflow: https://stackoverflow.com/search?q=%5Bpostgresql%5D+money But of course it is hard to say if these are production uses or just experiments. Yours, Laurenz Albe
On Mon, Dec 25, 2023 at 09:52:53AM +0900, Michael Paquier wrote: > At least the infra to be able to achieve that should be here. Looking at that, I can see that Peter has added a few tests to test the predictability of plans generated with non-hashable types, and that these are based on money. See 6dd8b0080787. One possible pick to replace money for these tests is tsvector, that cannot be hashed and has an equality operator. Perhaps these had better be replaced anyway? -- Michael
Attachment
Michael Paquier <michael@paquier.xyz> writes:
> Looking at that, I can see that Peter has added a few tests to test
> the predictability of plans generated with non-hashable types, and
> that these are based on money. See 6dd8b0080787.
> One possible pick to replace money for these tests is tsvector, that
> cannot be hashed and has an equality operator. Perhaps these had
> better be replaced anyway?
Hm. A quick check in pg_opclass shows that these are the types that
currently have btree but not hash opclasses:
# select ob.opcintype::regtype from (select * from pg_opclass where opcmethod = 403) ob
left join (select * from pg_opclass where opcmethod = 405) oh using(opcintype) where oh.opcintype is null;
opcintype
-------------
bit
bit varying
money
tsvector
tsquery
(5 rows)
I'm a little nervous about using tsvector or tsquery, as it seems
pretty plausible that somebody would get around to making hash
support for them someday. Perhaps the same argument could be made
about bit or varbit, but I'd bet a lot less on that happening,
as those are backwater-ish types (not even in the standard
anymore, IIRC). So I'd think about using one of those.
Or we could build a for-test-purposes-only datatype, but that
could require a lot of scaffolding work.
regards, tom lane
On Mon, Dec 25, 2023 at 10:58:59AM -0500, Tom Lane wrote: > I'm a little nervous about using tsvector or tsquery, as it seems > pretty plausible that somebody would get around to making hash > support for them someday. Perhaps the same argument could be made > about bit or varbit, but I'd bet a lot less on that happening, > as those are backwater-ish types (not even in the standard > anymore, IIRC). So I'd think about using one of those. Okay, I've used varbit then. > Or we could build a for-test-purposes-only datatype, but that > could require a lot of scaffolding work. It took me some time to come back to this thread, apologies for the delay. I have been looking at all that, and finished for now with the attached patch that removes the dependency to "money" in the main regression test suite in all the relevant places without losing coverage, switching these to use varbit (well, mostly): - Tests for hashing and hash-based plans (6dd8b0080787). - A set of tests in rules used money historically but these can be switched to numeric. - Two tests in stats_ext relied on cash_words() to generate different strings, but the same variance can be achieved without it. - One test in create_table relied on money as its cast is not immutable. Here it is possible to use a timestamptz to achieve the same, with something like to_date() to force a function execution when defining a partition boundary for one of the partitions (from 7c079d7417a8). With this patch, the tests related to money that remain are for type_sanity, window.sql and of course money which are all related to the data type so they could be wiped out once the type is itself removed. While looking at the whole picture, an issue with the direct removal of money is how we should handle btree_gin and btree_gist which have operators based on money. We try to keep things compatible at run-time, but could this be worth a hard break in these modules, dropping the older sql scripts used in the modules if we don't have access to money anymore at runtime? These are not popular modules.. Any thoughts about that? For now, please find attached a patch to adjust the regression tests to depend less on money, which does not depend on the type removal. Comments are welcome. -- Michael
Attachment
On 1/10/24 8:12 PM, Michael Paquier wrote:
> While looking at the whole picture, an issue with the direct removal
> of money is how we should handle btree_gin and btree_gist which have
> operators based on money. We try to keep things compatible at
> run-time, but could this be worth a hard break in these modules,
> dropping the older sql scripts used in the modules if we don't have
> access to money anymore at runtime? These are not popular modules..
> Any thoughts about that?
Both modules are pretty popular. Personally, I used it in scheduling
apps that involved range types + exclusion constraints. The data I've
seen suggests btree_gist / btree_gin are widely deployed.
That said, I don't know how much of these modules are used with the
money type specifically. My guess is that it's more common to combine it
with something like an {int,bool}/range type than a money type.
It sounds like we'd have to tread a bit lightly because of this, even if
money is not frequently (or at all) used with btree_gist/gin?
Jonathan
Attachment
"Jonathan S. Katz" <jkatz@postgresql.org> writes:
> On 1/10/24 8:12 PM, Michael Paquier wrote:
>> While looking at the whole picture, an issue with the direct removal
>> of money is how we should handle btree_gin and btree_gist which have
>> operators based on money. We try to keep things compatible at
>> run-time, but could this be worth a hard break in these modules,
>> dropping the older sql scripts used in the modules if we don't have
>> access to money anymore at runtime? These are not popular modules..
>> Any thoughts about that?
> Both modules are pretty popular.
Yeah, I think it's a serious error to assume they aren't used plenty.
Not the money part, but other scalar types.
> It sounds like we'd have to tread a bit lightly because of this, even if
> money is not frequently (or at all) used with btree_gist/gin?
What'd have to happen is that people would have to upgrade to a
version of btree_gin/btree_gist that deletes its money support
before they could pg_upgrade into a core version that lacks money.
So we'd have to ship that version at least one major release before
nuking the core support.
I think the shortest timeline we could possibly do this on is:
v17: label money as deprecated and due for removal in the SGML docs
v18: remove support in btree_gin/btree_gist (and any other affected
extensions)
v19: remove it from core
Note that in v18, people could still use money even in
btree_gin/btree_gist, just by installing a non-default extension
version. So their C code for money would have to stay.
regards, tom lane
On 1/12/24 11:33 AM, Tom Lane wrote: > "Jonathan S. Katz" <jkatz@postgresql.org> writes: >> It sounds like we'd have to tread a bit lightly because of this, even if >> money is not frequently (or at all) used with btree_gist/gin? > > What'd have to happen is that people would have to upgrade to a > version of btree_gin/btree_gist that deletes its money support > before they could pg_upgrade into a core version that lacks money. > So we'd have to ship that version at least one major release before > nuking the core support. Hm -- what about people who skip versions (e.g. 16 => 19)? Would they have to stop at 18 first to perform the upgrade? And does this only affect people who use btree_gist/gin for moeny, or all btree_gist/gin users? > I think the shortest timeline we could possibly do this on is: > > v17: label money as deprecated and due for removal in the SGML docs We had apparently deprecation notice as late as 8.2[1], but this warning is missing in the docs :( > v18: remove support in btree_gin/btree_gist (and any other affected > extensions) If the btree_gist/gin issue only affects people who use it in combination with money, I'd at least think we consider this for v17, and make clear that a users with money have to take action before upgrading. If we stick with just adding the deprecation notice in the v17 docs, I'd also suggest we emit a log warning when loading the catalog if the user has an active use of a "money" type in a table. (I understand that may a pain to do, but at least wanted to suggested it). > v19: remove it from core > > Note that in v18, people could still use money even in > btree_gin/btree_gist, just by installing a non-default extension > version. So their C code for money would have to stay. Yeah, but for what we support directly in PostgreSQL, we will have made best effort. If someone _really_ wants to do something with "money" in the way you describe above, then they're committed to it. Thanks, Jonathan [1] https://www.postgresql.org/docs/8.2/datatype-money.html
Attachment
On Fri, Jan 12, 2024 at 10:50:25PM -0500, Jonathan S. Katz wrote: > On 1/12/24 11:33 AM, Tom Lane wrote: >> What'd have to happen is that people would have to upgrade to a >> version of btree_gin/btree_gist that deletes its money support >> before they could pg_upgrade into a core version that lacks money. >> So we'd have to ship that version at least one major release before >> nuking the core support. > > Hm -- what about people who skip versions (e.g. 16 => 19)? Would they have > to stop at 18 first to perform the upgrade? And does this only affect people > who use btree_gist/gin for moeny, or all btree_gist/gin users? That's exactly why I am worried about a schedule being too aggressive here. Less runs of pg_upgrade means less headaches and less downtime. I'd guess that we should have at least 3~4 years between a deprecation in the docs and the actual removal. It seems like this is what we should do now on HEAD, aiming at a complete removal in 21~22 when 17 gets out of support (even if pg_upgrade support is down to 10 years). >> v19: remove it from core >> >> Note that in v18, people could still use money even in >> btree_gin/btree_gist, just by installing a non-default extension >> version. So their C code for money would have to stay. > > Yeah, but for what we support directly in PostgreSQL, we will have made best > effort. If someone _really_ wants to do something with "money" in the way > you describe above, then they're committed to it. By the way, let me know if you have any objections about the patch I have posted upthread that reduces the dependency to money in the main regression test suite, replacing things with varbit while keeping coverage the same. I have a few second thoughts about the change in create_table, so I'm tempted to leave that part out for now, switching the rest. -- Michael
Attachment
On Sun, Jan 14, 2024 at 09:55:03AM +0900, Michael Paquier wrote: > By the way, let me know if you have any objections about the patch I > have posted upthread that reduces the dependency to money in the main > regression test suite, replacing things with varbit while keeping > coverage the same. I have a few second thoughts about the change in > create_table, so I'm tempted to leave that part out for now, switching > the rest. This one has been backpatched as 31acee4b66f9 for now, but I've left out the test case in create_table with partitions. I kind of felt that I was missing something, and it did not prevent to switch all the others. I'll spawn a thread on -hackers about what to do next, with probably a patch to propose a deprecation notice in the docs for v17. -- Michael
Attachment
On Mon, Jan 15, 2024 at 04:54:51PM +0900, Michael Paquier wrote: > I'll spawn a thread on -hackers about what to do next, with probably a > patch to propose a deprecation notice in the docs for v17. Did this thread ever materialize? I searched around a bit and couldn't find anything. -- nathan
For future reference, this bug should be fixed by commit 22b0ccd. -- nathan