Thread: [HACKERS] CUBE seems a bit confused about ORDER BY

Hi,

I've noticed this suspicious behavior of "cube" data type with ORDER BY,
which I believe is a bug in the extension (or the GiST index support).
The following example comes directly from regression tests added by
33bd250f (so CC Teodor and Stas, who are mentioned in the commit).

This query should produce results with ordering "ascending by 2nd
coordinate or upper right corner". To make it clear, I've added the
"c~>4" expression to the query, otherwise it's right from the test.

test=# SELECT c~>4 "c~>4", * FROM test_cube ORDER BY c~>4 LIMIT 15;c~>4 |             c
------+---------------------------  50 | (30333, 50),(30273, 6)  75 | (43301, 75),(43227, 43) 142 | (19650,
142),(19630,51) 160 | (2424, 160),(2424, 81) 171 | (3449, 171),(3354, 108) 155 | (18037, 155),(17941, 109) 208 |
(28511,208),(28479, 114) 217 | (19946, 217),(19941, 118) 191 | (16906, 191),(16816, 139) 187 | (759, 187),(662, 163)
266| (22684, 266),(22656, 181) 255 | (24423, 255),(24360, 213) 249 | (45989, 249),(45910, 222) 377 | (11399,
377),(11360,294) 389 | (12162, 389),(12103, 309)
 
(15 rows)

As you can see, it's not actually sorted by the c~>4 coordinate (but by
c~>2, which it the last number).

Moreover, disabling index scans fixes the ordering:

test=# set enable_indexscan = off;
SET
test=# SELECT c~>4, * FROM test_cube ORDER BY c~>4 LIMIT 15; --
ascending by 2nd coordinate or upper right corner?column? |             c
----------+---------------------------      50 | (30333, 50),(30273, 6)      75 | (43301, 75),(43227, 43)     142 |
(19650,142),(19630, 51)     155 | (18037, 155),(17941, 109)     160 | (2424, 160),(2424, 81)     171 | (3449,
171),(3354,108)     187 | (759, 187),(662, 163)     191 | (16906, 191),(16816, 139)     208 | (28511, 208),(28479, 114)
   217 | (19946, 217),(19941, 118)     249 | (45989, 249),(45910, 222)     255 | (24423, 255),(24360, 213)     266 |
(22684,266),(22656, 181)     367 | (31018, 367),(30946, 333)     377 | (11399, 377),(11360, 294)
 
(15 rows)


Seems like a bug somewhere in gist_cube_ops, I guess?


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On 10/30/2017 03:04 PM, Alexander Korotkov wrote:
> On Sun, Oct 29, 2017 at 1:30 AM, Alexander Korotkov
> <a.korotkov@postgrespro.ru <mailto:a.korotkov@postgrespro.ru>> wrote:
...
> 
> Thus, I heard no objection to my approach.  Attached patch changes ~>
> operator in the proposed way and fixes KNN-GiST search for that.  I'm
> going to register this patch to the nearest commitfest.
> 

Seems fine to me, although perhaps it should be split into two parts.
One with the cube_coord_llur fixes, and then g_cube_distance changes
adding support for negative coordinates.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Tue, Nov 21, 2017 at 7:07 AM, Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> On Mon, Nov 20, 2017 at 1:59 PM, Alexander Korotkov
> <a.korotkov@postgrespro.ru> wrote:
>>
>> On Mon, Nov 20, 2017 at 4:09 AM, Tomas Vondra
>> <tomas.vondra@2ndquadrant.com> wrote:
>>>
>>> Seems fine to me, although perhaps it should be split into two parts.
>>> One with the cube_coord_llur fixes, and then g_cube_distance changes
>>> adding support for negative coordinates.
>>
>>
>> Thank you for your feedback.  I'll split this patch into two.
>
>
> Please, find two patches attached.

This got no reviews, so moved to next CF.
-- 
Michael

On 11/29/2017 06:13 AM, Michael Paquier wrote:
> On Tue, Nov 21, 2017 at 7:07 AM, Alexander Korotkov
> <a.korotkov@postgrespro.ru> wrote:
>> On Mon, Nov 20, 2017 at 1:59 PM, Alexander Korotkov
>> <a.korotkov@postgrespro.ru> wrote:
>>>
>>> On Mon, Nov 20, 2017 at 4:09 AM, Tomas Vondra
>>> <tomas.vondra@2ndquadrant.com> wrote:
>>>>
>>>> Seems fine to me, although perhaps it should be split into two parts.
>>>> One with the cube_coord_llur fixes, and then g_cube_distance changes
>>>> adding support for negative coordinates.
>>>
>>>
>>> Thank you for your feedback.  I'll split this patch into two.
>>
>>
>> Please, find two patches attached.
> 
> This got no reviews, so moved to next CF.
> 

That is somewhat inaccurate, I believe. I won't claim it received enough
reviews, but it certainly received some. And splitting it into two does
not invalidate that, I guess.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Wed, Nov 29, 2017 at 7:59 PM, Alexander Korotkov
<a.korotkov@postgrespro.ru> wrote:
> Sure, patch got some review.  I've no objection against moving this to the
> next commitfest though.

Please note that as this is qualified as a bug fix, I was not going to
mark it as returned with feedback or such. We want to keep track of
fixes that are bugs and potentially need back-patching.

> Since, these patches include bug fix, it's possible that someone will commit
> it before next commitfest.

Sure. Committers are free to commit patches they see suited for
integration even out of the CF time.
-- 
Michael

The following review has been posted through the commitfest application:
make installcheck-world:  tested, passed
Implements feature:       tested, passed
Spec compliant:           tested, passed
Documentation:            tested, passed

Hi! I've reviewed the patch.
Patch works as expected, documented and tested.

The code does not use LL_COORD and UR_COORD used all around. But the code still is easily readable. 

LL_COORD and UR_COORD are always combined with Min or Max function, so LL (Lower Left) and UR (Upper Right) are
somewhatirrelevant names...
 
BTW if we swap implementation of these macro-funcs and fix cube_out, almost all tests pass again. This is evidence of
goodcompatibility and shows that compatibility overhead is added everywhere.
 

I think that this patch is ready for committer.

Best regards, Andrey Borodin.

The new status of this patch is: Ready for Committer

> Yes, the thing is that we change behavior of existing ~> operator.  In general, 
> this is not good idea because it could affect existing users whose already use 
> this operator.  Typically in such situation, we could leave existing operator as 
> is, and invent new operator with new behavior.  However, in this particular 
> case, I think there are reasons to make an exception to the rules.  The reasons 
> are following:
> 1) The ~> operator was designed especially for knn gist.
> 2) Knn gist support for current behavior is broken by design and can't be 
> fixed.  Most we can do to fix existing ~> operator behavior as is to drop knn 
> gist support.  But then ~> operator would be left useless.
> 3) It doesn't seems that ~> operator have many users yet, because an error 
> wasn't reported during whole release cycle.
> 
> I hope these reasons justify altering behavior of existing operator as an 
> exception to the rules.  Another question is whether we should backpatch it.  
> But I think we could leave this decision to committer.
> 
>     I think that this patch is ready for committer.
I'm agree with changing behavior of existing ~> operator, but is any objection 
here? Current implementation is not fixable and I hope that users which use this 
operator will understand why we change it. Fortunately, the fix doesn't require 
changes in system catalog.

The single question here is about index over expression with this operator, they 
will need to reindex, which should be noted in release notes.

-- 
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
                                                    WWW: http://www.sigaev.ru/

On 12/12/2017 01:52 PM, Alexander Korotkov wrote:
> On Tue, Dec 12, 2017 at 3:49 PM, Teodor Sigaev <teodor@sigaev.ru
> <mailto:teodor@sigaev.ru>> wrote:
> 
>         Yes, the thing is that we change behavior of existing ~>
>         operator.  In general, this is not good idea because it could
>         affect existing users whose already use this operator. 
>         Typically in such situation, we could leave existing operator as
>         is, and invent new operator with new behavior.  However, in this
>         particular case, I think there are reasons to make an exception
>         to the rules.  The reasons are following:
>         1) The ~> operator was designed especially for knn gist.
>         2) Knn gist support for current behavior is broken by design and
>         can't be fixed.  Most we can do to fix existing ~> operator
>         behavior as is to drop knn gist support.  But then ~> operator
>         would be left useless.
>         3) It doesn't seems that ~> operator have many users yet,
>         because an error wasn't reported during whole release cycle.
> 
>         I hope these reasons justify altering behavior of existing
>         operator as an exception to the rules.  Another question is
>         whether we should backpatch it.  But I think we could leave this
>         decision to committer.
> 
>             I think that this patch is ready for committer.
> 
>     I'm agree with changing behavior of existing ~> operator, but is any
>     objection here? Current implementation is not fixable and I hope
>     that users which use this operator will understand why we change it.
>     Fortunately, the fix doesn't require changes in system catalog.
> 
>     The single question here is about index over expression with this
>     operator, they will need to reindex, which should be noted in
>     release notes.
> 
> 
> Yes.  I bet only few users have built indexes over ~> operator if any. 
> Ask them to reindex in the release notes seems OK for me.
> 

Is there a good way to detect such cases? Either in pg_upgrade, so that
we can print warnings, or at least manually (which would be suitable for
release notes).

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

>> Yes.  I bet only few users have built indexes over ~> operator if any.
>> Ask them to reindex in the release notes seems OK for me.
>>
> 
> Is there a good way to detect such cases? Either in pg_upgrade, so that
> we can print warnings, or at least manually (which would be suitable for
> release notes).

Hmm, suppose, fix should be backpatched (because now it's unusable) and 
pg_upgrade  should not do anything. Just add release note to 10.0 and 11.0

Oh, check expression is affected too, users will need to reinsert data.

-- 
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
                                                    WWW: http://www.sigaev.ru/

> SELECT dep.classid::regclass AS class,
>    CASE WHEN dep.classid = 'pg_catalog.pg_class'::regclass THEN 
> dep.objid::regclass::text
>         WHEN dep.classid = 'pg_catalog.pg_constraint'::regclass THEN (SELECT 
> conname FROM pg_catalog.pg_constraint WHERE oid = dep.objid)
>    ELSE NULL
>    END AS name
> FROM
>    pg_catalog.pg_extension e
>    JOIN pg_catalog.pg_depend edep ON edep.refclassid = 
> 'pg_catalog.pg_extension'::regclass AND edep.refobjid = e.oid AND deptype = 'e' 
> AND edep.classid = 'pg_catalog.pg_operator'::regclass
>    JOIN pg_catalog.pg_operator o ON o.oid = edep.objid AND o.oprname = '~>'
>    JOIN pg_catalog.pg_depend dep ON dep.refclassid = 
> 'pg_catalog.pg_operator'::regclass AND dep.refobjid = o.oid
> WHERE
>    e.extname = 'cube' AND dep.classid IN ('pg_catalog.pg_constraint'::regclass, 
> 'pg_catalog.pg_class'::regclass);
> 
> On the below data schema
> 
> create table tmp (c cube, check ((c ~> 0 > 0)));
> create index tmp_idx on tmp ((c~>0));
> 
> it gives following result
> 
>       class     |    name
> ---------------+-------------
>   pg_class      | tmp_idx
>   pg_constraint | tmp_c_check
> (2 rows)

SQL-query seems too huge for release notes and isn't looking for materialized 
view (fixable) and functional indexes with function which contains this operator 
somewhere inside (not fixable by this query). I think, just words is enough.

-- 
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
                                                    WWW: http://www.sigaev.ru/

Teodor Sigaev wrote:

> SQL-query seems too huge for release notes and isn't looking for
> materialized view (fixable) and functional indexes with function which
> contains this operator somewhere inside (not fixable by this query). I
> think, just words is enough.

But the query can be made a little bit shorter and more comprehensible:

SELECT pg_describe_object(dep.classid, dep.objid, dep.objsubid)
FROM pg_catalog.pg_extension ext
   JOIN pg_catalog.pg_depend edep ON edep.refobjid = ext.oid
   JOIN pg_catalog.pg_operator oper ON oper.oid = edep.objid
   JOIN pg_catalog.pg_depend dep ON dep.refobjid = oper.oid
WHERE
   ext.extname = 'cube' AND
   edep.refclassid = 'pg_catalog.pg_extension'::regclass AND
   edep.classid = 'pg_catalog.pg_operator'::regclass AND
   edep.deptype = 'e' AND
   oper.oprname = '~>' AND
   dep.refclassid = 'pg_catalog.pg_operator'::regclass
;

which returns the following

                                                 pg_describe_object                                                 
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 regla «_RETURN» en vista materializada f
 índice tmp_idx
 restricción «tmp_c_check» en tabla tmp
 operador 15 (cube, integer) de familia de operadores gist_cube_ops para el método de acceso gist: ~>(cube,integer)
(4 filas)

(after 
create materialized view f as select * from tmp where c~>1 > 1;
)

I think this is useful enough.  The fact remains that we can't check
very well for functions; maybe suggest a LIKE clause to look for ~>
anywhere in function source code?


(It looks like you could get rid of the 'deptype' qual and
dep.refclassid also)

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Thanks to all, pushed

>     I think this is useful enough.  The fact remains that we can't check
>     very well for functions; maybe suggest a LIKE clause to look for ~>
>     anywhere in function source code?Seems, it's overengineering. It could give a false positive results and doesn't

cover all cases such as C-functions, library functions for perl/python/etc 
languages.

-- 
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
                                                    WWW: http://www.sigaev.ru/

Teodor Sigaev wrote:
> 
> >     I think this is useful enough.  The fact remains that we can't check
> >     very well for functions; maybe suggest a LIKE clause to look for ~>
> >     anywhere in function source code?

> Seems, it's overengineering. It could give a false positive results
> and doesn't cover all cases such as C-functions, library functions for
> perl/python/etc languages.

That's true, and I agree we don't necessarily have to find everything.
I still think we should print the pg_depend query in the relnotes,
because those would be the most common cases of objects that need to be
rebuilt.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> That's true, and I agree we don't necessarily have to find everything.
> I still think we should print the pg_depend query in the relnotes,
> because those would be the most common cases of objects that need to be
> rebuilt.

Wait. A. Minute.  This patch seems totally unacceptable for backpatching.

>> Since behavior of ~> (cube, int) operator is changed, depending entities
>> must be refreshed after upgrade. Such as, expression indexes using this
>> operator must be reindexed, materialized views must be rebuilt, stored
>> procedures and client code must be revised to correctly use new behavior.
>> That should be mentioned in release notes.

Was there any real discussion of whether we could get away with that
in the back branches?  My opinion is no.  It's not even clear to me
that this is acceptable in HEAD --- isn't it going to create huge
problems for pg_upgrade?

Perhaps some solution to the compatibility problems could be found
based on giving the cube extension a new version number, and applying
the behavioral change only upon doing ALTER EXTENSION UPDATE.  But
it doesn't look like the patch even bumped the extension's version
number.

I do not think this patch was ready to commit.

            regards, tom lane

Tom Lane wrote:

> >> Since behavior of ~> (cube, int) operator is changed, depending entities
> >> must be refreshed after upgrade. Such as, expression indexes using this
> >> operator must be reindexed, materialized views must be rebuilt, stored
> >> procedures and client code must be revised to correctly use new behavior.
> >> That should be mentioned in release notes.
> 
> Was there any real discussion of whether we could get away with that
> in the back branches?  My opinion is no.  It's not even clear to me
> that this is acceptable in HEAD --- isn't it going to create huge
> problems for pg_upgrade?

This was discussed upthread and the solution found was "objects need to
be rebuilt, indexes need to be reindexed".  The point of Alexander's
query was to find affected objects that need such treatment.  Teodor
explicitly disregarded any change in pg_upgrade because the database
you're upgrading *from* is supposed to have gotten indexes reindexed,
etc.

> Perhaps some solution to the compatibility problems could be found
> based on giving the cube extension a new version number, and applying
> the behavioral change only upon doing ALTER EXTENSION UPDATE.

Hmm, that's an idea worth exploring, keeping in mind that the bug
affects an operator.  I'm not sure it's possible to redefine an operator
(change its underlying function) without dropping it, which would cause
its OID to change.  Maybe that's okay, or maybe there's a way around it.

> But it doesn't look like the patch even bumped the extension's version
> number.

As I understand, extension version numbers should change if the SQL
interface changes (the set of objects differ), but should not change if
only the underlying C code changes.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> Tom Lane wrote:
>> Was there any real discussion of whether we could get away with that
>> in the back branches?  My opinion is no.  It's not even clear to me
>> that this is acceptable in HEAD --- isn't it going to create huge
>> problems for pg_upgrade?

> This was discussed upthread and the solution found was "objects need to
> be rebuilt, indexes need to be reindexed".  The point of Alexander's
> query was to find affected objects that need such treatment.  Teodor
> explicitly disregarded any change in pg_upgrade because the database
> you're upgrading *from* is supposed to have gotten indexes reindexed,
> etc.

I don't think that is really going to be acceptable.  People do not like
minor version updates that break their databases.  If we start doing
that we're going to find people refusing to apply minor updates, which
is not a place we want to be.

What we've done in the past for comparable situations is to make the
change in a new major version and teach pg_upgrade to detect and report
the need for changes --- in this case, it might do something like create
a script of REINDEX commands for the affected indexes.  See e.g.
old_9_6_invalidate_hash_indexes().

            regards, tom lane

Tom Lane wrote:
> Alvaro Herrera <alvherre@alvh.no-ip.org> writes:

> > This was discussed upthread and the solution found was "objects need to
> > be rebuilt, indexes need to be reindexed".  The point of Alexander's
> > query was to find affected objects that need such treatment.  Teodor
> > explicitly disregarded any change in pg_upgrade because the database
> > you're upgrading *from* is supposed to have gotten indexes reindexed,
> > etc.
> 
> I don't think that is really going to be acceptable.  People do not like
> minor version updates that break their databases.  If we start doing
> that we're going to find people refusing to apply minor updates, which
> is not a place we want to be.

I was uncomfortable with the idea of not doing anything to fix old
databases.

> What we've done in the past for comparable situations is to make the
> change in a new major version and teach pg_upgrade to detect and report
> the need for changes --- in this case, it might do something like create
> a script of REINDEX commands for the affected indexes.  See e.g.
> old_9_6_invalidate_hash_indexes().

I think the whole point is that any knn-gist indexes using this operator
are completely broken (i.e. they return the wrong answer), so leaving
them as-is in existing branches is not cool.

The idea of an extension update being the trigger for a fix sounds
reasonable.  Maybe we can have the current function under ~> that throws
a WARNING each time it is invoked, inviting people to upgrade the
extension; and the new extension would contain a new ~> with the right
semantics.  Then all the magic to rebuild objects has to occur in the
upgrade .sql script.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

>> This was discussed upthread and the solution found was "objects need to
>> be rebuilt, indexes need to be reindexed".  The point of Alexander's
>> query was to find affected objects that need such treatment.  Teodor
>> explicitly disregarded any change in pg_upgrade because the database
>> you're upgrading *from* is supposed to have gotten indexes reindexed,
>> etc.
>
> I don't think that is really going to be acceptable.  People do not like
> minor version updates that break their databases.  If we start doing
> that we're going to find people refusing to apply minor updates, which
> is not a place we want to be.
That's true, but we have choice of bad solutions. Current index could 
not support operator before patch. So we can:
1) Change operator to support existing index. That is what Alexander
    did. And yes, it changes returning order for both sequential and
    index scans, but makes them synced. Actually, user should not
    reindex existing indexes but should be ready for order changing
2) Change index structure which isn't obvious how. At least, it's
    possible to  add new operator class (so, upgrade script is needed)
    Mandatory reindex and order changes for index scans
3) Remove index support for this operator at all. And introduce new
    operator in HEAD with index support. This will need an upgrade script
    in minor versions

-- 
Teodor Sigaev                      E-mail: teodor@sigaev.ru
                                       WWW: http://www.sigaev.ru/