Home > mailing lists

Thread: JsonbValue to Jsonb conversion

JsonbValue to Jsonb conversion

From

Dmitry Dolgov

Date:

23 September 2014, 19:23:40

Hi all,

I'm faced with some troubles about the jsonb implementation, and I hope I'll get little advice =)

If I understand correctly, an abstract function for jsonb modification should have the following stages:

Jsonb -> JsonbValue -> Modification -> JsonbValue -> Jsonb

One can convert the JsonbValue to the Jsonb only by JsonbValueToJsonb function. So, my question is can be JsonbValue, that contains few jbvBinary elements, converted to Jsonb by this function? It will be very useful, if you want modify only small part of your JsonbValue (e.g. replace value of some key). But when I'm trying to do this, an exception "unknown type of jsonb container" appears. Maybe I missed something? Or is there another approach to do this conversion?

Re: JsonbValue to Jsonb conversion

From

Andrew Dunstan

Date:

24 September 2014, 03:20:31

On 09/23/2014 12:23 PM, Dmitry Dolgov wrote:
> Hi all,
>
> I'm faced with some troubles about the jsonb implementation, and I
> hope I'll get little advice =)
> If I understand correctly, an abstract function for jsonb modification
> should have the following stages:
>
> Jsonb -> JsonbValue -> Modification -> JsonbValue -> Jsonb
>
> One can convert the *JsonbValue* to the *Jsonb* only by
> *JsonbValueToJsonb* function. So, my question is can be *JsonbValue*,
> that contains few *jbvBinary* elements, converted to *Jsonb* by this
> function? It will be very useful, if you want modify only small part
> of your JsonbValue (e.g. replace value of some key). But when I'm
> trying to do this, an exception "unknown type of jsonb container"
> appears. Maybe I missed something? Or is there another approach to do
> this conversion?


If you can come up with a way of handling the jbvBinary values then by
all means send a patch.

But this problem is fairly easily worked around by using an iterator
over the binary value. The attached patch, which is work in progress for
adding in the currently missing json functions for jsonb, contains a
sort of example of doing this in jsonb_agg_transfn.

cheers

andrew

Attachment

jsonbmissingfuncs.patch

Re: JsonbValue to Jsonb conversion

From

Pavel Stehule

Date:

13 October 2014, 13:42:42

I am working on review of this patch.

There is new warnings:

jsonb.c: In function ‘jsonb_agg_transfn’:
jsonb.c:1540:20: warning: assignment makes pointer from integer without a cast
      v.val.numeric = DirectFunctionCall1(numeric_uplus, NumericGetDatum(v.val.numeric));
                    ^
jsonb.c: In function ‘jsonb_object_agg_transfn’:
jsonb.c:1745:20: warning: assignment makes pointer from integer without a cast
      v.val.numeric = DirectFunctionCall1(numeric_uplus, NumericGetDatum(v.val.numeric));

[pavel@localhost postgresql]$ gcc --version
gcc (GCC) 4.9.1 20140930 (Red Hat 4.9.1-11)

Check fails

parallel group (19 tests): alter_table plancache temp domain prepare limit plpgsql conversion sequence copy2 rangefuncs returning truncate xml with without_oid largeobject polymorphism rowtypes
     plancache                ... FAILED (test process exited with exit code 2)
     limit                    ... FAILED (test process exited with exit code 2)
     plpgsql                  ... FAILED (test process exited with exit code 2)
     copy2                    ... FAILED (test process exited with exit code 2)
     temp                     ... FAILED (test process exited with exit code 2)
     domain                   ... FAILED (test process exited with exit code 2)
     rangefuncs               ... FAILED (test process exited with exit code 2)
     prepare                  ... FAILED (test process exited with exit code 2)
     without_oid              ... FAILED (test process exited with exit code 2)
     conversion               ... FAILED (test process exited with exit code 2)
     truncate                 ... FAILED (test process exited with exit code 2)
     alter_table              ... FAILED (test process exited with exit code 2)
     sequence                 ... FAILED (test process exited with exit code 2)
     polymorphism             ... FAILED (test process exited with exit code 2)
     rowtypes                 ... FAILED (test process exited with exit code 2)
     returning                ... FAILED (test process exited with exit code 2)
     largeobject              ... FAILED (test process exited with exit code 2)
     with                     ... FAILED (test process exited with exit code 2)
     xml                      ... FAILED (test process exited with exit code 2)
test stats                    ... FAILED (test process exited with exit code 2)

[pavel@localhost postgresql]$ uname -a
Linux localhost.localdomain 3.16.3-302.fc21.x86_64 #1 SMP Fri Sep 26 14:27:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

backtrace

Core was generated by `postgres: pavel regression [local] SELECT                                     '.
Program terminated with signal SIGSEGV, Segmentation fault.

(gdb) bt
#0 0x0000000001e95300 in ?? ()
#1 0x00000000007c048b in parse_object_field (lex=0x1ede9d8, sem=0x7fff3c3c4660) at json.c:398
#2 0x00000000007c0524 in parse_object (lex=0x1ede9d8, sem=0x7fff3c3c4660) at json.c:430
#3 0x00000000007c0214 in pg_parse_json (lex=0x1ede9d8, sem=0x7fff3c3c4660) at json.c:297
#4 0x00000000007c5d91 in datum_to_jsonb (val=32118224, is_null=0 '\000', result=0x7fff3c3c4800, tcategory=JSONBTYPE_JSON,
    outfuncoid=322, key_scalar=0 '\000') at jsonb.c:789
#5 0x00000000007c68be in add_jsonb (val=32118224, is_null=0 '\000', result=0x7fff3c3c4800, val_type=114, key_scalar=0 '\000')
    at jsonb.c:1050
#6 0x00000000007c6d08 in jsonb_build_object (fcinfo=0x1edcb80) at jsonb.c:1155
#7 0x000000000060bfc5 in ExecMakeFunctionResultNoSets (fcache=0x1edcb10, econtext=0x1edc920, isNull=0x1edd568 "", isDone=0x1edd680)
    at execQual.c:1992
#8 0x000000000060c8bc in ExecEvalFunc (fcache=0x1edcb10, econtext=0x1edc920, isNull=0x1edd568 "", isDone=0x1edd680)
    at execQual.c:2383
#9 0x0000000000612869 in ExecTargetList (targetlist=0x1edd650, econtext=0x1edc920, values=0x1edd550, isnull=0x1edd568 "",
    itemIsDone=0x1edd680, isDone=0x7fff3c3c4a84) at execQual.c:5265
#10 0x0000000000612e9d in ExecProject (projInfo=0x1edd580, isDone=0x7fff3c3c4a84) at execQual.c:5480
#11 0x000000000062c046 in ExecResult (node=0x1edc810) at nodeResult.c:155
#12 0x0000000000608997 in ExecProcNode (node=0x1edc810) at execProcnode.c:373
#13 0x000000000060696e in ExecutePlan (estate=0x1edc700, planstate=0x1edc810, operation=CMD_SELECT, sendTuples=1 '\001',
    numberTuples=0, direction=ForwardScanDirection, dest=0x1ea18b0) at execMain.c:1481
#14 0x0000000000604de8 in standard_ExecutorRun (queryDesc=0x1edc2f0, direction=ForwardScanDirection, count=0) at execMain.c:308
#15 0x0000000000604ce5 in ExecutorRun (queryDesc=0x1edc2f0, direction=ForwardScanDirection, count=0) at execMain.c:256
#16 0x000000000075615a in PortalRunSelect (portal=0x1eda2e0, forward=1 '\001', count=0, dest=0x1ea18b0) at pquery.c:946
#17 0x0000000000755e34 in PortalRun (portal=0x1eda2e0, count=9223372036854775807, isTopLevel=1 '\001', dest=0x1ea18b0,
    altdest=0x1ea18b0, completionTag=0x7fff3c3c4dc0 "") at pquery.c:790
#18 0x00000000007502c2 in exec_simple_query (
    query_string=0x1ea0050 "SELECT jsonb_build_object('e',json '{\"x\": 3, \"y\": [1,2,3]}');") at postgres.c:1045
#19 0x0000000000754284 in PostgresMain (argc=1, argv=0x1e45448, dbname=0x1e452f8 "postgres", username=0x1e452e0 "pavel")
    at postgres.c:4010
#20 0x00000000006e7954 in BackendRun (port=0x1e64d20) at postmaster.c:4118
#21 0x00000000006e70ac in BackendStartup (port=0x1e64d20) at postmaster.c:3793
#22 0x00000000006e3ba9 in ServerLoop () at postmaster.c:1572
#23 0x00000000006e327c in PostmasterMain (argc=3, argv=0x1e44540) at postmaster.c:1219

Problematic statement:

SELECT jsonb_build_array('a',1,'b',1.2,'c',true,'d',null,'e',json '{"x": 3, "y": [1,2,3]}');

or SELECT jsonb_build_object('e',json '{"x": 3, "y": [1,2,3]}');

Tested on HEAD

Regards

Pavel

2014-09-24 2:20 GMT+02:00 Andrew Dunstan <andrew@dunslane.net>:

On 09/23/2014 12:23 PM, Dmitry Dolgov wrote:
Hi all,

I'm faced with some troubles about the jsonb implementation, and I hope I'll get little advice =)
If I understand correctly, an abstract function for jsonb modification should have the following stages:

Jsonb -> JsonbValue -> Modification -> JsonbValue -> Jsonb

One can convert the *JsonbValue* to the *Jsonb* only by *JsonbValueToJsonb* function. So, my question is can be *JsonbValue*, that contains few *jbvBinary* elements, converted to *Jsonb* by this function? It will be very useful, if you want modify only small part of your JsonbValue (e.g. replace value of some key). But when I'm trying to do this, an exception "unknown type of jsonb container" appears. Maybe I missed something? Or is there another approach to do this conversion?

If you can come up with a way of handling the jbvBinary values then by all means send a patch.

But this problem is fairly easily worked around by using an iterator over the binary value. The attached patch, which is work in progress for adding in the currently missing json functions for jsonb, contains a sort of example of doing this in jsonb_agg_transfn.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: JsonbValue to Jsonb conversion

From

Andrew Dunstan

Date:

13 October 2014, 16:45:53

On 10/13/2014 06:41 AM, Pavel Stehule wrote:
> Hi
>
> I am working on review of this patch.

The patch attached to the message you are replying to was never intended 
to be reviewed. It was only given by way of illustration of a technique.

The original patch to be reviewed is on the message 
<http://www.postgresql.org/message-id/5425D277.4030804@dunslane.net> as 
shown on the commitfest app. I have just submitted a revised patch to 
fix the compiler warnings you complained of, at 
<http://www.postgresql.org/message-id/543BD598.4020809@dunslane.net> I 
have not found any segfaults in the regression tests.

And please don't top-post on the PostgreSQL lists.

cheers

andrew

Re: JsonbValue to Jsonb conversion

From

Pavel Stehule

Date:

13 October 2014, 17:15:17

2014-10-13 15:45 GMT+02:00 Andrew Dunstan <andrew@dunslane.net>:

On 10/13/2014 06:41 AM, Pavel Stehule wrote:
Hi

I am working on review of this patch.

The patch attached to the message you are replying to was never intended to be reviewed. It was only given by way of illustration of a technique.

The original patch to be reviewed is on the message <http://www.postgresql.org/message-id/5425D277.4030804@dunslane.net> as shown on the commitfest app. I have just submitted a revised patch to fix the compiler warnings you complained of, at <http://www.postgresql.org/message-id/543BD598.4020809@dunslane.net> I have not found any segfaults in the regression tests.

I checked this last version - warning is out, but SIGFAULT on jsonb test is there .. I rechecked it with clang compiler, but result is same

I try to search why

And please don't top-post on the PostgreSQL lists.

I am sorry

Regards

Pavel

cheers

andrew

Re: JsonbValue to Jsonb conversion

From

Tom Lane

Date:

13 October 2014, 17:19:06

Pavel Stehule <pavel.stehule@gmail.com> writes:
> I checked this last version - warning is out, but SIGFAULT on jsonb test is
> there .. I rechecked it with clang compiler, but result is same

Stack trace please?
        regards, tom lane

Re: JsonbValue to Jsonb conversion

From

Pavel Stehule

Date:

13 October 2014, 17:22:21

2014-10-13 16:19 GMT+02:00 Tom Lane <tgl@sss.pgh.pa.us>:

Pavel Stehule <pavel.stehule@gmail.com> writes:
> I checked this last version - warning is out, but SIGFAULT on jsonb test is
> there .. I rechecked it with clang compiler, but result is same

Stack trace please?

(gdb) bt
#0 0x0000000000000072 in ?? ()
#1 0x000000000087d598 in parse_array_element (lex=0x2880118, sem=0x7fffb4f02508) at json.c:461
#2 0x0000000000878da7 in parse_array (lex=0x2880118, sem=0x7fffb4f02508) at json.c:505
#3 0x000000000087d837 in parse_object_field (lex=0x2880118, sem=0x7fffb4f02508) at json.c:391
#4 0x0000000000878cb2 in parse_object (lex=0x2880118, sem=0x7fffb4f02508) at json.c:432
#5 0x000000000087831c in pg_parse_json (lex=0x2880118, sem=0x7fffb4f02508) at json.c:297
#6 0x000000000087f484 in datum_to_jsonb (val=42202912, is_null=0 '\000', result=0x7fffb4f02800,
    tcategory=JSONBTYPE_JSON, outfuncoid=322, key_scalar=0 '\000') at jsonb.c:789
#7 0x000000000087fce7 in add_jsonb (val=42202912, is_null=0 '\000', result=0x7fffb4f02800, val_type=114,
    key_scalar=0 '\000') at jsonb.c:1050
#8 0x000000000087fbcc in jsonb_build_object (fcinfo=0x287e2c0) at jsonb.c:1155
#9 0x000000000066d179 in ExecMakeFunctionResultNoSets (fcache=0x287e250, econtext=0x287e060, isNull=0x287eca8 "",
    isDone=0x287edc0) at execQual.c:1992
#10 0x000000000066776f in ExecEvalFunc (fcache=0x287e250, econtext=0x287e060, isNull=0x287eca8 "", isDone=0x287edc0)
    at execQual.c:2383
#11 0x000000000066c3bb in ExecTargetList (targetlist=0x287ed90, econtext=0x287e060, values=0x287ec90,
    isnull=0x287eca8 "", itemIsDone=0x287edc0, isDone=0x7fffb4f02aac) at execQual.c:5265
#12 0x000000000066c2c2 in ExecProject (projInfo=0x287ecc0, isDone=0x7fffb4f02aac) at execQual.c:5480
#13 0x0000000000689ceb in ExecResult (node=0x287df50) at nodeResult.c:155
#14 0x0000000000661987 in ExecProcNode (node=0x287df50) at execProcnode.c:373
#15 0x000000000065dd46 in ExecutePlan (estate=0x287de40, planstate=0x287df50, operation=CMD_SELECT,
    sendTuples=1 '\001', numberTuples=0, direction=ForwardScanDirection, dest=0x283fa00) at execMain.c:1481
#16 0x000000000065dc70 in standard_ExecutorRun (queryDesc=0x2809d50, direction=ForwardScanDirection, count=0)
    at execMain.c:308
#17 0x000000000065db3f in ExecutorRun (queryDesc=0x2809d50, direction=ForwardScanDirection, count=0)
    at execMain.c:256
#18 0x00000000007ec70c in PortalRunSelect (portal=0x2807bc0, forward=1 '\001', count=0, dest=0x283fa00)
    at pquery.c:946
#19 0x00000000007ec229 in PortalRun (portal=0x2807bc0, count=9223372036854775807, isTopLevel=1 '\001',
    dest=0x283fa00, altdest=0x283fa00, completionTag=0x7fffb4f02ec0 "") at pquery.c:790
#20 0x00000000007e7f7c in exec_simple_query (
    query_string=0x283e1a0 "SELECT jsonb_build_object('e',json '{\"x\": 3, \"y\": [1,2,3]}');") at postgres.c:1045
#21 0x00000000007e72cb in PostgresMain (argc=1, argv=0x27e5838, dbname=0x27e56e8 "postgres",
---Type <return> to continue, or q <return> to quit---q
username=0x27e56d0 "paveQuit

Regards

Pavel

regards, tom lane

Re: JsonbValue to Jsonb conversion

From

Pavel Stehule

Date:

13 October 2014, 17:40:46

A JsonSemAction sem is not well initialized

a array_element_start is not initialized and enforces sigfault on my comp

*** ./utils/adt/jsonb.c.orig    2014-10-13 16:37:00.479708142 +0200
--- ./utils/adt/jsonb.c    2014-10-13 16:36:33.704650644 +0200
***************
*** 786,791 ****
--- 786,793 ----
                    sem.scalar = jsonb_in_scalar;
                    sem.object_field_start = jsonb_in_object_field_start;

+                     sem.array_element_start = NULL;
+
                    pg_parse_json(lex, &sem);

                }

I am not sure, if this fix is valid, but all tests are passed now

Regards

Pavel

2014-10-13 16:21 GMT+02:00 Pavel Stehule <pavel.stehule@gmail.com>:

2014-10-13 16:19 GMT+02:00 Tom Lane <tgl@sss.pgh.pa.us>:
Pavel Stehule <pavel.stehule@gmail.com> writes:
> I checked this last version - warning is out, but SIGFAULT on jsonb test is
> there .. I rechecked it with clang compiler, but result is same

Stack trace please?

(gdb) bt
#0 0x0000000000000072 in ?? ()
#1 0x000000000087d598 in parse_array_element (lex=0x2880118, sem=0x7fffb4f02508) at json.c:461
#2 0x0000000000878da7 in parse_array (lex=0x2880118, sem=0x7fffb4f02508) at json.c:505
#3 0x000000000087d837 in parse_object_field (lex=0x2880118, sem=0x7fffb4f02508) at json.c:391
#4 0x0000000000878cb2 in parse_object (lex=0x2880118, sem=0x7fffb4f02508) at json.c:432
#5 0x000000000087831c in pg_parse_json (lex=0x2880118, sem=0x7fffb4f02508) at json.c:297
#6 0x000000000087f484 in datum_to_jsonb (val=42202912, is_null=0 '\000', result=0x7fffb4f02800,
    tcategory=JSONBTYPE_JSON, outfuncoid=322, key_scalar=0 '\000') at jsonb.c:789
#7 0x000000000087fce7 in add_jsonb (val=42202912, is_null=0 '\000', result=0x7fffb4f02800, val_type=114,
    key_scalar=0 '\000') at jsonb.c:1050
#8 0x000000000087fbcc in jsonb_build_object (fcinfo=0x287e2c0) at jsonb.c:1155
#9 0x000000000066d179 in ExecMakeFunctionResultNoSets (fcache=0x287e250, econtext=0x287e060, isNull=0x287eca8 "",
    isDone=0x287edc0) at execQual.c:1992
#10 0x000000000066776f in ExecEvalFunc (fcache=0x287e250, econtext=0x287e060, isNull=0x287eca8 "", isDone=0x287edc0)
    at execQual.c:2383
#11 0x000000000066c3bb in ExecTargetList (targetlist=0x287ed90, econtext=0x287e060, values=0x287ec90,
    isnull=0x287eca8 "", itemIsDone=0x287edc0, isDone=0x7fffb4f02aac) at execQual.c:5265
#12 0x000000000066c2c2 in ExecProject (projInfo=0x287ecc0, isDone=0x7fffb4f02aac) at execQual.c:5480
#13 0x0000000000689ceb in ExecResult (node=0x287df50) at nodeResult.c:155
#14 0x0000000000661987 in ExecProcNode (node=0x287df50) at execProcnode.c:373
#15 0x000000000065dd46 in ExecutePlan (estate=0x287de40, planstate=0x287df50, operation=CMD_SELECT,
    sendTuples=1 '\001', numberTuples=0, direction=ForwardScanDirection, dest=0x283fa00) at execMain.c:1481
#16 0x000000000065dc70 in standard_ExecutorRun (queryDesc=0x2809d50, direction=ForwardScanDirection, count=0)
    at execMain.c:308
#17 0x000000000065db3f in ExecutorRun (queryDesc=0x2809d50, direction=ForwardScanDirection, count=0)
    at execMain.c:256
#18 0x00000000007ec70c in PortalRunSelect (portal=0x2807bc0, forward=1 '\001', count=0, dest=0x283fa00)
    at pquery.c:946
#19 0x00000000007ec229 in PortalRun (portal=0x2807bc0, count=9223372036854775807, isTopLevel=1 '\001',
    dest=0x283fa00, altdest=0x283fa00, completionTag=0x7fffb4f02ec0 "") at pquery.c:790
#20 0x00000000007e7f7c in exec_simple_query (
    query_string=0x283e1a0 "SELECT jsonb_build_object('e',json '{\"x\": 3, \"y\": [1,2,3]}');") at postgres.c:1045
#21 0x00000000007e72cb in PostgresMain (argc=1, argv=0x27e5838, dbname=0x27e56e8 "postgres",
---Type <return> to continue, or q <return> to quit---q
username=0x27e56d0 "paveQuit

Regards

Pavel

regards, tom lane

Re: JsonbValue to Jsonb conversion

From

Pavel Stehule

Date:

13 October 2014, 17:54:55

2014-10-13 15:45 GMT+02:00 Andrew Dunstan <andrew@dunslane.net>:

On 10/13/2014 06:41 AM, Pavel Stehule wrote:
Hi

I am working on review of this patch.

The patch attached to the message you are replying to was never intended to be reviewed. It was only given by way of illustration of a technique.

The original patch to be reviewed is on the message <http://www.postgresql.org/message-id/5425D277.4030804@dunslane.net> as shown on the commitfest app. I have just submitted a revised patch to fix the compiler warnings you complained of, at <http://www.postgresql.org/message-id/543BD598.4020809@dunslane.net> I have not found any segfaults in the regression tests.

And please don't top-post on the PostgreSQL lists.

Attached small fix of uninitialized local variable

Regards

Pavel

cheers

andrew

Attachment

jsonbmissingfuncs3-2.patch

Re: JsonbValue to Jsonb conversion

From

Andrew Dunstan

Date:

13 October 2014, 18:03:14

On 10/13/2014 10:39 AM, Pavel Stehule wrote:
> Hi
>
> A JsonSemAction sem is not well initialized
>
> a array_element_start is not initialized and enforces sigfault on my comp
>
> *** ./utils/adt/jsonb.c.orig    2014-10-13 16:37:00.479708142 +0200
> --- ./utils/adt/jsonb.c    2014-10-13 16:36:33.704650644 +0200
> ***************
> *** 786,791 ****
> --- 786,793 ----
>                       sem.scalar = jsonb_in_scalar;
>                       sem.object_field_start = 
> jsonb_in_object_field_start;
>
> +                     sem.array_element_start = NULL;
> +
>                       pg_parse_json(lex, &sem);
>
>                   }
>
> I am not sure, if this fix is valid, but all tests are passed now
>
>

Good find. I think what we should probably do is initialize the whole 
thing with:
    memset(&sem, 0, sizeof(JsonSemAction));

before assigning anything to its fields. That would be consistent with 
what we do elsewhere.

I'll make that change and submit a new patch.

Please stop using this thread, however. It's inappropriate for reviewing 
this patch.

cheers

andrew