Thread: Bytea string operator support
> Here's a revised patch. Changes: > > 1. Now outputs '\\' instead of '\134' when using encode(bytea, 'escape') > Note that I ended up leaving \0 as \000 so that there are no ambiguities > when decoding something like, for example, \0123. > > 2. Fixed bug in byteain which allowed input values which were not valid > octals (e.g. \789), to be parsed as if they were octals. > > Joe > Here's rev 2 of the bytea string support patch. Changes: 1. Added missing declaration for MatchBytea function 2. Added PQescapeBytea to fe-exec.c 3. Applies cleanly on cvs tip from this afternoon I'm hoping that someone can review/approve/apply this before beta starts, so I guess I'd vote (not that it counts for much) to delay beta a few days :-) -- Joe
Attachment
Your patch has been added to the PostgreSQL unapplied patches list at:
http://candle.pha.pa.us/cgi-bin/pgpatches
I will try to apply it within the next 48 hours.
> > Here's a revised patch. Changes:
> >
> > 1. Now outputs '\\' instead of '\134' when using encode(bytea, 'escape')
> > Note that I ended up leaving \0 as \000 so that there are no ambiguities
> > when decoding something like, for example, \0123.
> >
> > 2. Fixed bug in byteain which allowed input values which were not valid
> > octals (e.g. \789), to be parsed as if they were octals.
> >
> > Joe
> >
>
> Here's rev 2 of the bytea string support patch. Changes:
>
> 1. Added missing declaration for MatchBytea function
> 2. Added PQescapeBytea to fe-exec.c
> 3. Applies cleanly on cvs tip from this afternoon
>
> I'm hoping that someone can review/approve/apply this before beta starts, so
> I guess I'd vote (not that it counts for much) to delay beta a few days :-)
>
> -- Joe
>
>
[ Attachment, skipping... ]
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026
Joe Conway writes: > 2. Added PQescapeBytea to fe-exec.c I'm not sure if we want to encode the peculiarities of particular data types into the client libraries. I could agree with adding a function that supplies a general-purpose encoding such as base64 (not necessarily preferred) which clients can use to feed data to the server. In the future we might want to support the SQL-standard input formats for binary data types, which would require yet another round of functions to be added to libpq. After all, bytea is utterly non-standard, and if we make API extensions we should consider them in the long run. -- Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter
> Joe Conway writes: > > > 2. Added PQescapeBytea to fe-exec.c > > I'm not sure if we want to encode the peculiarities of particular data > types into the client libraries. I could agree with adding a function > that supplies a general-purpose encoding such as base64 (not necessarily > preferred) which clients can use to feed data to the server. In the > future we might want to support the SQL-standard input formats for binary > data types, which would require yet another round of functions to be added > to libpq. > > After all, bytea is utterly non-standard, and if we make API extensions we > should consider them in the long run. Bytea is pecular because you can't have the escaping done in the backend because of the binary nature of bytea. I think there was an attempt to merge this into the escaping of SQL queries but that can't work because of the handling of nulls and the string possibly being four times bigger than the original. I agree this is type pollution into libpq but I don't see another option except having the user code this stuff into their application. Maybe we need some type-handling library in the future were we can put all this stuff. Maybe we can add this to libpq now and create a separate library when we need one. Comments? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026
> >
> > > 2. Added PQescapeBytea to fe-exec.c
> >
> > I'm not sure if we want to encode the peculiarities of particular data
> > types into the client libraries. I could agree with adding a function
> > that supplies a general-purpose encoding such as base64 (not necessarily
> > preferred) which clients can use to feed data to the server. In the
> > future we might want to support the SQL-standard input formats for
binary
> > data types, which would require yet another round of functions to be
added
> > to libpq.
> >
> > After all, bytea is utterly non-standard, and if we make API extensions
we
> > should consider them in the long run.
>
> Bytea is pecular because you can't have the escaping done in the backend
> because of the binary nature of bytea. I think there was an attempt to
> merge this into the escaping of SQL queries but that can't work because
> of the handling of nulls and the string possibly being four times bigger
> than the original.
>
> I agree this is type pollution into libpq but I don't see another option
> except having the user code this stuff into their application. Maybe we
> need some type-handling library in the future were we can put all this
> stuff. Maybe we can add this to libpq now and create a separate library
> when we need one.
>
> Comments?
That was the main reason I wanted to see the bytea escape function in the
PostgreSQL client library. Without it, every user needs to code something
that performs this same functionality into their own app if they want to do
something like: insert into foo(f1) values ('binary stuff');
Someone has mentioned that the Perl DBI module for PostgreSQL can
already do this, but that doesn't help anyone writing in C or PHP
(as two examples).
Certainly if there is a SQL-standard, it would be better to support it. I
scanned through SQL99, but didn't spot anything on SQL-standard input
formats for binary. Peter, can you point me in the right direction?
-- Joe
Have we made a decision on this? I think we should apply it.
> > >
> > > > 2. Added PQescapeBytea to fe-exec.c
> > >
> > > I'm not sure if we want to encode the peculiarities of particular data
> > > types into the client libraries. I could agree with adding a function
> > > that supplies a general-purpose encoding such as base64 (not necessarily
> > > preferred) which clients can use to feed data to the server. In the
> > > future we might want to support the SQL-standard input formats for
> binary
> > > data types, which would require yet another round of functions to be
> added
> > > to libpq.
> > >
> > > After all, bytea is utterly non-standard, and if we make API extensions
> we
> > > should consider them in the long run.
> >
> > Bytea is pecular because you can't have the escaping done in the backend
> > because of the binary nature of bytea. I think there was an attempt to
> > merge this into the escaping of SQL queries but that can't work because
> > of the handling of nulls and the string possibly being four times bigger
> > than the original.
> >
> > I agree this is type pollution into libpq but I don't see another option
> > except having the user code this stuff into their application. Maybe we
> > need some type-handling library in the future were we can put all this
> > stuff. Maybe we can add this to libpq now and create a separate library
> > when we need one.
> >
> > Comments?
>
> That was the main reason I wanted to see the bytea escape function in the
> PostgreSQL client library. Without it, every user needs to code something
> that performs this same functionality into their own app if they want to do
> something like: insert into foo(f1) values ('binary stuff');
>
> Someone has mentioned that the Perl DBI module for PostgreSQL can
> already do this, but that doesn't help anyone writing in C or PHP
> (as two examples).
>
> Certainly if there is a SQL-standard, it would be better to support it. I
> scanned through SQL99, but didn't spot anything on SQL-standard input
> formats for binary. Peter, can you point me in the right direction?
>
> -- Joe
>
>
>
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
> (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>
--
Bruce Momjian | http://candle.pha.pa.us
pgman@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026
> > Have we made a decision on this? I think we should apply it. > Well I'd vote for that ;-) If Peter feels strongly about *not* including PQescapeBytea, I'll remove it and resubmit the patch, but I would like to see it in there myself. Just let me know want you want me to do. I haven't seen any comments one way or the other on the rest of the patch, but again, I'd really like to see it applied. The backend functions are all per SQL99 (section 4.3 Binary strings, and 8.5 <like predicate>). -- Joe
Bruce Momjian writes: > I agree this is type pollution into libpq but I don't see another option > except having the user code this stuff into their application. Maybe we > need some type-handling library in the future were we can put all this > stuff. Maybe we can add this to libpq now and create a separate library > when we need one. We probably do. After all, the strings you get are not really the most optimal form, especially for numbers, dates, and booleans. Looks at the JDBC spec, they have getXXX for each data type. -- Peter Eisentraut peter_e@gmx.net http://funkturm.homeip.net/~peter
> Bruce Momjian writes: > > > I agree this is type pollution into libpq but I don't see another option > > except having the user code this stuff into their application. Maybe we > > need some type-handling library in the future were we can put all this > > stuff. Maybe we can add this to libpq now and create a separate library > > when we need one. > > We probably do. After all, the strings you get are not really the most > optimal form, especially for numbers, dates, and booleans. Looks at the > JDBC spec, they have getXXX for each data type. Agreed. Let's add this now and start thinking about developing a separate lib for 7.3. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026
Patch applied. Thanks. > > Here's a revised patch. Changes: > > > > 1. Now outputs '\\' instead of '\134' when using encode(bytea, 'escape') > > Note that I ended up leaving \0 as \000 so that there are no ambiguities > > when decoding something like, for example, \0123. > > > > 2. Fixed bug in byteain which allowed input values which were not valid > > octals (e.g. \789), to be parsed as if they were octals. > > > > Joe > > > > Here's rev 2 of the bytea string support patch. Changes: > > 1. Added missing declaration for MatchBytea function > 2. Added PQescapeBytea to fe-exec.c > 3. Applies cleanly on cvs tip from this afternoon > > I'm hoping that someone can review/approve/apply this before beta starts, so > I guess I'd vote (not that it counts for much) to delay beta a few days :-) > > -- Joe > > [ Attachment, skipping... ] > > ---------------------------(end of broadcast)--------------------------- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/users-lounge/docs/faq.html -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000 + If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania 19026