Thread: Re: PATCH: jsonpath string methods: lower, upper, initcap, l/r/btrim, replace, split_part

Florents Tselai <florents.tselai@gmail.com> writes:
> This patch is a follow-up and generalization to [0].
> It adds the following jsonpath methods:  lower, upper, initcap, l/r/btrim,
> replace, split_part.

How are you going to deal with the fact that this makes jsonpath
operations not guaranteed immutable?  (See commit cb599b9dd
for some context.)  Those are all going to have behavior that's
dependent on the underlying locale.

We have the kluge of having separate "_tz" functions to support
non-immutable datetime operations, but that way doesn't seem like
it's going to scale well to multiple sources of mutability.

            regards, tom lane

On Thu, Sep 26, 2024 at 12:04 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Florents Tselai <florents.tselai@gmail.com> writes:
> > This patch is a follow-up and generalization to [0].
> > It adds the following jsonpath methods:  lower, upper, initcap, l/r/btrim,
> > replace, split_part.
>
> How are you going to deal with the fact that this makes jsonpath
> operations not guaranteed immutable?  (See commit cb599b9dd
> for some context.)  Those are all going to have behavior that's
> dependent on the underlying locale.
>
> We have the kluge of having separate "_tz" functions to support
> non-immutable datetime operations, but that way doesn't seem like
> it's going to scale well to multiple sources of mutability.

While inventing "_tz" functions I was thinking about jsonpath methods
and operators defined in standard then.  Now I see huge interest on
extending that.  I wonder if we can introduce a notion of flexible
mutability?  Imagine that jsonb_path_query() function (and others) has
another function which analyzes arguments and reports mutability.  If
jsonpath argument is constant and all methods inside are safe then
jsonb_path_query() is immutable otherwise it is stable.  I was
thinking about that back working on jsonpath, but that time problem
seemed too limited for this kind of solution.  Now, it's possibly time
to shake off the dust from this idea.  What do you think?

------
Regards,
Alexander Korotkov
Supabase

On Sep 26, 2024, at 13:59, Florents Tselai <florents.tselai@gmail.com> wrote:

> Speaking of extensible: the jsonpath standard does mention function extensions [1] ,
> so it looks like we're covered by the standard, and the mutability aspect is an implementation detail. No?

That’s not the standard used for Postgres jsonpath. Postgres follows the SQL/JSON standard in the SQL standard, which
isnot publicly available, but a few people on the list have copies they’ve purchased and so could provide some context. 

In a previous post I wondered if the SQL standard had some facility for function extensions, but I suspect not. Maybe
inthe next iteration? 

> And having said that, the whole jsonb/jsonpath parser/executor infrastructure is extremely powerful
> and kinda under-utilized if we use it "only" for jsonpath.
> Tbh, I can see it supporting more specific DSLs and even offering hooks for extensions.
> And I know for certain I'm not the only one thinking about this.
> See [2] for example where they've lifted, shifted and renamed the jsonb/jsonpath infra to build a separate language
forgraphs 

I’m all for extensibility, though jsonpath does need to continue to comply with the SQL standard. Do you have some idea
ofthe sorts of hooks that would allow extension authors to use some of that underlying capability? 

Best,

David

> On 27 Sep 2024, at 12:45 PM, David E. Wheeler <david@justatheory.com> wrote:
>
> On Sep 26, 2024, at 13:59, Florents Tselai <florents.tselai@gmail.com> wrote:
>
>> Speaking of extensible: the jsonpath standard does mention function extensions [1] ,
>> so it looks like we're covered by the standard, and the mutability aspect is an implementation detail. No?
>
> That’s not the standard used for Postgres jsonpath. Postgres follows the SQL/JSON standard in the SQL standard, which
isnot publicly available, but a few people on the list have copies they’ve purchased and so could provide some context. 
>
> In a previous post I wondered if the SQL standard had some facility for function extensions, but I suspect not. Maybe
inthe next iteration? 
>
>> And having said that, the whole jsonb/jsonpath parser/executor infrastructure is extremely powerful
>> and kinda under-utilized if we use it "only" for jsonpath.
>> Tbh, I can see it supporting more specific DSLs and even offering hooks for extensions.
>> And I know for certain I'm not the only one thinking about this.
>> See [2] for example where they've lifted, shifted and renamed the jsonb/jsonpath infra to build a separate language
forgraphs 
>
> I’m all for extensibility, though jsonpath does need to continue to comply with the SQL standard. Do you have some
ideaof the sorts of hooks that would allow extension authors to use some of that underlying capability? 

Re-tracing what I had to do

1. Define a new JsonPathItemType jpiMyExtType and map it to a JsonPathKeyword
2. Add a new JsonPathKeyword and make the lexer and parser aware of that,
3. Tell the main executor executeItemOptUnwrapTarget what to do when the new type is matched.

I think 1, 2 are the trickiest because they require hooks to  jsonpath_scan.l and parser jsonpath_gram.y

3. is the meat of a potential hook, which would be something like
extern JsonPathExecResult executeOnMyJsonpathItem(JsonPathExecContext *cxt, JsonbValue *jb, JsonValueList *found);
This should be called by the main executor executeItemOptUnwrapTarget when it encounters case jpiMyExtType

It looks like quite an endeavor, to be honest.

On Tue, May 13, 2025 at 11:00 PM David E. Wheeler <david@justatheory.com> wrote:
> On May 13, 2025, at 16:24, Florents Tselai <florents.tselai@gmail.com> wrote:
> > As Robert said—and I agree—renaming the existing _tz family would be more trouble than it’s worth, given the need
fordeprecations, migration paths, etc. If we were designing this today, suffixes like _stable or _volatile might have
beenmore appropriate, but at this point, we’re better off staying consistent with the _tz family. 
>
> I get the pragmatism, and don’t want to over-bike-shed, but what a wart to live with. [I just went back and re-read
Robert’spost, and didn’t realize he used exactly the same expression!] Would it really be too effortful to create
_stableor _volatile functions and leave the _tz functions as a sort of legacy? 

No, that wouldn't be too much work, but the issue is that people will
keep using the _tz versions and when we eventually try to remove them
those people will complain no matter how prominent we make the
deprecation notice. If we want to go this route, maybe we should do
something like:

1. Add the new versions with a _s suffix or whatever.

2. Invent a GUC jsonb_tz_warning = { on | off } that advises you to
use the new functions instead, whenever you use the old ones.

3. After N years, flip the default value from off to on.

4. After M additional years, remove the old functions and the GUC.

5. Still get complaints.

--
Robert Haas
EDB: http://www.enterprisedb.com

"David E. Wheeler" <david@justatheory.com> writes:
> On May 21, 2025, at 14:06, Robert Haas <robertmhaas@gmail.com> wrote:
>> If we want to go this route, maybe we should do
>> something like:
>> ...
>> 5. Still get complaints.

> Complainers gonna complain.

Yeah.  I do not see the point of that amount of effort.

> Any idea how widespread the use of the function is? It was added in 17, and I’ve met few who have really dug into the
jonpathstuff yet, let alone needed the time zone conversion functionality. 

That's a good point.  We should also remember that if somebody really
really doesn't want to fix their app, they can trivially create a
wrapper function with the old name.

Having said that, what's wrong with inventing some improved function
names and never removing the old ones?

            regards, tom lane

On Wed, May 21, 2025 at 2:31 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Having said that, what's wrong with inventing some improved function
> names and never removing the old ones?

I don't particularly like the clutter, but if the consensus is that
the clutter doesn't matter, fair enough.

--
Robert Haas
EDB: http://www.enterprisedb.com