Thread: XPATH evaluation

Hello,
During review of https://commitfest.postgresql.org/action/patch_view?id=580 I found following problems with XPath.
1.SELECT (XPATH('/root/*', '<root xmlns:o="http://olacle.com/db"
xmlns:p="http://postgresql.org/db"><o:db><a><b></b></a></o:db><p:db></p:db></root>'));Produces:"{"<o:db> <a>    <b/>
</a></o:db>",<p:db/>}"Inabove <b></b> was reduced to <b/> this is different infoset then input, and those notations are
differentlyinterpreted e.g. by XML Binding & WebServices. The 1st one will may be mapped to empty string, and 2nd one
toto null.
 
As well result was formatted which produces again different infoset.
Both of above may cause problems with XML digesting.
2.SELECT (XPATH('/root/*', '<root xmlns:o="http://olacle.com/db"
xmlns:p="http://postgresql.org/db"><o:db></o:db><p:db></p:db></root>'));"{<o:db/>,<p:db/>}"Inabove I missing
namespaces.
I may take on assignment 1st (fix is simple), but for 2nd I have mixed fillings. I think 2nd should transfer namespaces
insome way to client.
 
What do You think?
Regards,Radosław Smogura

On Jun17, 2011, at 11:09 , Radosław Smogura wrote:
> 1.
> SELECT (XPATH('/root/*', '<root xmlns:o="http://olacle.com/db"
xmlns:p="http://postgresql.org/db"><o:db><a><b></b></a></o:db><p:db></p:db></root>'));
> Produces:
> "{"<o:db>
>  <a>
>    <b/>
>  </a>
> </o:db>",<p:db/>}"
> In above <b></b> was reduced to <b/> this is different infoset then input, and those notations are differently
interpretede.g. by XML Binding & WebServices. The 1st one will may be mapped to empty string, and 2nd one to to null. 

Oh, joy :-(

Does this happen only with my patch applied or also with unpatched HEAD?

> 2.
> SELECT (XPATH('/root/*', '<root xmlns:o="http://olacle.com/db"
xmlns:p="http://postgresql.org/db"><o:db></o:db><p:db></p:db></root>'));
> "{<o:db/>,<p:db/>}"
> In above I missing namespaces.

Hm, that's a hard problem a think. Your problem (1) basically tells us that
ideally we'd return the matching parts of an XML document unmodified. Now,
(2) tells us that isn't to most sensible thing to do either.

> I may take on assignment 1st (fix is simple)

Whats your proposed fix for (1)?

> , but for 2nd I have mixed fillings. I think 2nd should transfer namespaces
> in some way to client.

I don't see how XPATH() can do that without breaking it's API. The only
thing we could do AFAICS is the define a second XPATH evaluation function
which returns a list of namespace declarations (prefix and uri) for every
node.

best regards,
Florian Pflug

On 06/17/2011 05:41 AM, Florian Pflug wrote:
> On Jun17, 2011, at 11:09 , Radosław Smogura wrote:
>> 1.
>> SELECT (XPATH('/root/*', '<root xmlns:o="http://olacle.com/db"
xmlns:p="http://postgresql.org/db"><o:db><a><b></b></a></o:db><p:db></p:db></root>'));
>> Produces:
>> "{"<o:db>
>>   <a>
>>     <b/>
>>   </a>
>> </o:db>",<p:db/>}"
>> In above<b></b>  was reduced to<b/>  this is different infoset then input, and those notations are differently
interpretede.g. by XML Binding&  WebServices. The 1st one will may be mapped to empty string, and 2nd one to to null. 
> Oh, joy :-(

I thought these were basically supposed to be the same.

The XML Information Set for example specifically excludes:

    The difference between the two forms of an empty element: |<foo/> |
    and |<foo></foo>|.||||

|
See <http://www.w3.org/TR/2004/REC-xml-infoset-20040204/> Appendix D.
Note that this implies that <foo></foo> does not have content of an
empty string, but that it has no content.

|
cheers

andrew

Andrew Dunstan <andrew@dunslane.net> Friday 17 of June 2011 15:47:04
> On 06/17/2011 05:41 AM, Florian Pflug wrote:
> > On Jun17, 2011, at 11:09 , Radosław Smogura wrote:
> >> 1.
> >> SELECT (XPATH('/root/*', '<root xmlns:o="http://olacle.com/db"
> >> xmlns:p="http://postgresql.org/db"><o:db><a><b></b></a></o:db><p:db></p
> >> :db></root>')); Produces:
> >> "{"<o:db>
> >>
> >>   <a>
> >>
> >>     <b/>
> >>
> >>   </a>
> >>
> >> </o:db>",<p:db/>}"
> >> In above<b></b>  was reduced to<b/>  this is different infoset then
> >> input, and those notations are differently interpreted e.g. by XML
> >> Binding&  WebServices. The 1st one will may be mapped to empty string,
> >> and 2nd one to to null.
> >
> > Oh, joy :-(
>
> I thought these were basically supposed to be the same.
>
> The XML Information Set for example specifically excludes:
>
>     The difference between the two forms of an empty element: |<foo/> |
>     and |<foo></foo>|.||||
>
>
> See <http://www.w3.org/TR/2004/REC-xml-infoset-20040204/> Appendix D.
> Note that this implies that <foo></foo> does not have content of an
> empty string, but that it has no content.
>
>
> cheers
>
> andrew

Indeed, Infoset Spec, and XML Canonization Spec treats <foo></foo> same, as
<foo/> - my wrong, but XML canonization preservs whitespaces, if I remember
well, I think there is example.

In any case if I will store image in XML (I've seen this), preservation of
white spaces and new lines is important.

Regards,
Radek.

On 06/17/2011 10:55 AM, Radosław Smogura wrote:
> Andrew Dunstan<andrew@dunslane.net>  Friday 17 of June 2011 15:47:04
>> On 06/17/2011 05:41 AM, Florian Pflug wrote:
>>> On Jun17, 2011, at 11:09 , Radosław Smogura wrote:
>>>> 1.
>>>> SELECT (XPATH('/root/*', '<root xmlns:o="http://olacle.com/db"
>>>> xmlns:p="http://postgresql.org/db"><o:db><a><b></b></a></o:db><p:db></p
>>>> :db></root>')); Produces:
>>>> "{"<o:db>
>>>>
>>>>    <a>
>>>>
>>>>      <b/>
>>>>
>>>>    </a>
>>>>
>>>> </o:db>",<p:db/>}"
>>>> In above<b></b>   was reduced to<b/>   this is different infoset then
>>>> input, and those notations are differently interpreted e.g. by XML
>>>> Binding&   WebServices. The 1st one will may be mapped to empty string,
>>>> and 2nd one to to null.
>>> Oh, joy :-(
>> I thought these were basically supposed to be the same.
>>
>> The XML Information Set for example specifically excludes:
>>
>>      The difference between the two forms of an empty element: |<foo/>  |
>>      and |<foo></foo>|.||||
>>
>>
>> See<http://www.w3.org/TR/2004/REC-xml-infoset-20040204/>  Appendix D.
>> Note that this implies that<foo></foo>  does not have content of an
>> empty string, but that it has no content.
>>
>>
>> cheers
>>
>> andrew
> Indeed, Infoset Spec, and XML Canonization Spec treats<foo></foo>  same, as
> <foo/>  - my wrong, but XML canonization preservs whitespaces, if I remember
> well, I think there is example.
>
> In any case if I will store image in XML (I've seen this), preservation of
> white spaces and new lines is important.

If you store images you should encode them anyway, in base64 or hex.

More generally, data that needs that sort of preservation should
possibly be in CDATA nodes.

cheers

andrew

Florian Pflug <fgp@phlo.org> Friday 17 of June 2011 11:41:08
> On Jun17, 2011, at 11:09 , Radosław Smogura wrote:
> > 1.
> > SELECT (XPATH('/root/*', '<root xmlns:o="http://olacle.com/db"
> > xmlns:p="http://postgresql.org/db"><o:db><a><b></b></a></o:db><p:db></p:
> > db></root>')); Produces:
> > "{"<o:db>
> >
> >  <a>
> >
> >    <b/>
> >
> >  </a>
> >
> > </o:db>",<p:db/>}"
> > In above <b></b> was reduced to <b/> this is different infoset then
> > input, and those notations are differently interpreted e.g. by XML
> > Binding & WebServices. The 1st one will may be mapped to empty string,
> > and 2nd one to to null.
>
> Oh, joy :-(
>
> Does this happen only with my patch applied or also with unpatched HEAD?
>
> > 2.
> > SELECT (XPATH('/root/*', '<root xmlns:o="http://olacle.com/db"
> > xmlns:p="http://postgresql.org/db"><o:db></o:db><p:db></p:db></root>'));
> > "{<o:db/>,<p:db/>}"
> > In above I missing namespaces.
>
> Hm, that's a hard problem a think. Your problem (1) basically tells us that
> ideally we'd return the matching parts of an XML document unmodified. Now,
> (2) tells us that isn't to most sensible thing to do either.
>
> > I may take on assignment 1st (fix is simple)
>
> Whats your proposed fix for (1)?
>
> > , but for 2nd I have mixed fillings. I think 2nd should transfer
> > namespaces in some way to client.
>
> I don't see how XPATH() can do that without breaking it's API. The only
> thing we could do AFAICS is the define a second XPATH evaluation function
> which returns a list of namespace declarations (prefix and uri) for every
> node.
>
> best regards,
> Florian Pflug

No this is not about Your patch, but was inspired by it.
Regards,
Radek

On Jun17, 2011, at 17:09 , Andrew Dunstan wrote:
> If you store images you should encode them anyway, in base64 or hex.
> More generally, data that needs that sort of preservation should possibly be in CDATA nodes.

All very true.

Still, ideally we'd return the XML exactly as stored, though, even
for the results of XPATH queries. But I've no idea if this is easily
done with libxml or not.

best regards,
Florian Pflug

2011/6/17, Andrew Dunstan <andrew@dunslane.net>:

> On 06/17/2011 10:55 AM, Radosław Smogura wrote:
>
>> XML canonization preservs whitespaces, if I remember
>> well, I think there is example.
>>
>> In any case if I will store image in XML (I've seen this), preservation of
>> white spaces and new lines is important.
>
> If you store images you should encode them anyway, in base64 or hex.

Whitespace that is not at certain obviously irrelevant places (such as
right after "<", between attributes, outside of the whole document,
etc), and that is not defined to be irrelevant by some schema (if the
parser is schema-aware), is relevant. You cannot just muck around with
it and consider that correct.

> More generally, data that needs that sort of preservation should
> possibly be in CDATA nodes.

CDATA sections are just syntactic sugar (a form of escaping):

<URL:http://www.w3.org/TR/xml-infoset/#omitted>

"Appendix D: What is not in the Information Set
[..]
19. The boundaries of CDATA marked sections."

Therefore, there is not such thing as a "CDATA node" that would be
different from "just text" (Infoset-wise).

Note that that does not mean that binary data is never supposed to be
altered or that all binary data is to be accepted: e.g., whether
newlines are represented using "\n", "\r", or "\r\n" is irrelevant;
also, binary data that is not valid according to the used encoding
must of course not be accepted.

Nicolas

--
A. Because it breaks the logical sequence of discussion.
Q. Why is top posting bad?

Andrew Dunstan <andrew@dunslane.net> Friday 17 of June 2011 17:09:25
> On 06/17/2011 10:55 AM, Radosław Smogura wrote:
> > Andrew Dunstan<andrew@dunslane.net>  Friday 17 of June 2011 15:47:04
> >
> >> On 06/17/2011 05:41 AM, Florian Pflug wrote:
> >>> On Jun17, 2011, at 11:09 , Radosław Smogura wrote:
> >>>> 1.
> >>>> SELECT (XPATH('/root/*', '<root xmlns:o="http://olacle.com/db"
> >>>> xmlns:p="http://postgresql.org/db"><o:db><a><b></b></a></o:db><p:db></
> >>>> p
> >>>>
> >>>> :db></root>')); Produces:
> >>>> "{"<o:db>
> >>>>
> >>>>    <a>
> >>>>
> >>>>      <b/>
> >>>>
> >>>>    </a>
> >>>>
> >>>> </o:db>",<p:db/>}"
> >>>> In above<b></b>   was reduced to<b/>   this is different infoset then
> >>>> input, and those notations are differently interpreted e.g. by XML
> >>>> Binding&   WebServices. The 1st one will may be mapped to empty
> >>>> string, and 2nd one to to null.
> >>>
> >>> Oh, joy :-(
> >>
> >> I thought these were basically supposed to be the same.
> >>
> >> The XML Information Set for example specifically excludes:
> >>      The difference between the two forms of an empty element: |<foo/>
> >>      | and |<foo></foo>|.||||
> >>
> >> See<http://www.w3.org/TR/2004/REC-xml-infoset-20040204/>  Appendix D.
> >> Note that this implies that<foo></foo>  does not have content of an
> >> empty string, but that it has no content.
> >>
> >>
> >> cheers
> >>
> >> andrew
> >
> > Indeed, Infoset Spec, and XML Canonization Spec treats<foo></foo>  same,
> > as <foo/>  - my wrong, but XML canonization preservs whitespaces, if I
> > remember well, I think there is example.
> >
> > In any case if I will store image in XML (I've seen this), preservation
> > of white spaces and new lines is important.
>
> If you store images you should encode them anyway, in base64 or hex.
>
> More generally, data that needs that sort of preservation should
> possibly be in CDATA nodes.
>
> cheers
>
> andrew
I know this answer, because this solution is better. But, during one work I
created XSL-FO with whitespace preserve attribute, if I would like to get part
of such XSL-FO I could destroy output document.

But those use-cases doesn't change fact that XPATH output doesn't preserves
whitepsaces, newlines, and produces different node, then was in original. It
same as regexp form varchar will trim result without control.

I emphasize this because it may cause problems with XML Digest algorithms
which are quite popular and may cause some legal! problems when you try to use
Advance Signature in Europe Union, as well with other application.

With XML Binding it's quite popular to interpret <foo/> as null, <foo></foo>
as empty string. In particulary mantoined Infoset Spec doesn't matters here.

I think no-formatting is reasonable requirement for XPATH function.

Regards,
Radek.

On 06/17/2011 11:29 AM, Nicolas Barbier wrote:
> 2011/6/17, Andrew Dunstan<andrew@dunslane.net>:
>
>> On 06/17/2011 10:55 AM, Radosław Smogura wrote:
>>
>>> XML canonization preservs whitespaces, if I remember
>>> well, I think there is example.
>>>
>>> In any case if I will store image in XML (I've seen this), preservation of
>>> white spaces and new lines is important.
>> If you store images you should encode them anyway, in base64 or hex.
> Whitespace that is not at certain obviously irrelevant places (such as
> right after "<", between attributes, outside of the whole document,
> etc), and that is not defined to be irrelevant by some schema (if the
> parser is schema-aware), is relevant. You cannot just muck around with
> it and consider that correct.


Sure, but if you're storing arbitrary binary data such as images 
whitespace is the least of your problems. That's why I've always encoded 
them in base64.


>> More generally, data that needs that sort of preservation should
>> possibly be in CDATA nodes.
> CDATA sections are just syntactic sugar (a form of escaping):
>
> <URL:http://www.w3.org/TR/xml-infoset/#omitted>
>
>

Yeah. OTOH doesn't an empty CDATA section force a child element, where a 
pure empty element does not?

Anyway, we're getting a bit far from what Postgres needs to be doing.

cheers

andrew

2011/6/17, Andrew Dunstan <andrew@dunslane.net>:

> On 06/17/2011 11:29 AM, Nicolas Barbier wrote:
>
>> CDATA sections are just syntactic sugar (a form of escaping):
>
> Yeah. OTOH doesn't an empty CDATA section force a child element, where a
> pure empty element does not?

Wow, some Googling around shows that there is much confusion about
this. I thought that it was obvious that adding <![CDATA[]]> shouldn't
change the content at all, but quite a few people seem to disagree
:-/.

Nicolas

--
A. Because it breaks the logical sequence of discussion.
Q. Why is top posting bad?

Nicolas Barbier <nicolas.barbier@gmail.com> Friday 17 of June 2011 17:29:57
> 2011/6/17, Andrew Dunstan <andrew@dunslane.net>:
> > On 06/17/2011 10:55 AM, Radosław Smogura wrote:
> >> XML canonization preservs whitespaces, if I remember
> >> well, I think there is example.
> >>
> >> In any case if I will store image in XML (I've seen this), preservation
> >> of white spaces and new lines is important.
> >
> > If you store images you should encode them anyway, in base64 or hex.
>
> Whitespace that is not at certain obviously irrelevant places (such as
> right after "<", between attributes, outside of the whole document,
> etc), and that is not defined to be irrelevant by some schema (if the
> parser is schema-aware), is relevant. You cannot just muck around with
> it and consider that correct.
>
> > More generally, data that needs that sort of preservation should
> > possibly be in CDATA nodes.
>
> CDATA sections are just syntactic sugar (a form of escaping):
>
> <URL:http://www.w3.org/TR/xml-infoset/#omitted>
>
> "Appendix D: What is not in the Information Set
> [..]
> 19. The boundaries of CDATA marked sections."
>
> Therefore, there is not such thing as a "CDATA node" that would be
> different from "just text" (Infoset-wise).
>
> Note that that does not mean that binary data is never supposed to be
> altered or that all binary data is to be accepted: e.g., whether
> newlines are represented using "\n", "\r", or "\r\n" is irrelevant;
> also, binary data that is not valid according to the used encoding
> must of course not be accepted.
>
> Nicolas

I would like to send patch to remove formatting. How to deal with collapsing
blank nodes I don't know.

Regards,
Radek