Thread: pgsql: Add support for piping COPY to/from an external program.
Add support for piping COPY to/from an external program. This includes backend "COPY TO/FROM PROGRAM '...'" syntax, and corresponding psql \copy syntax. Like with reading/writing files, the backend version is superuser-only, and in the psql version, the program is run in the client. In the passing, the psql \copy STDIN/STDOUT syntax is subtly changed: if you the stdin/stdout is quoted, it's now interpreted as a filename. For example, "\copy foo from 'stdin'" now reads from a file called 'stdin', not from standard input. Before this, there was no way to specify a filename called stdin, stdout, pstdin or pstdout. This creates a new function in pgport, wait_result_to_str(), which can be used to convert the exit status of a process, as returned by wait(3), to a human-readable string. Etsuro Fujita, reviewed by Amit Kapila. Branch ------ master Details ------- http://git.postgresql.org/pg/commitdiff/3d009e45bde2a2681826ef549637ada76508b597 Modified Files -------------- contrib/file_fdw/file_fdw.c | 4 +- doc/src/sgml/keywords.sgml | 7 + doc/src/sgml/ref/copy.sgml | 50 +++++++- doc/src/sgml/ref/psql-ref.sgml | 33 ++++-- src/backend/commands/copy.c | 205 ++++++++++++++++++++++-------- src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/parser/gram.y | 48 ++++++-- src/backend/storage/file/fd.c | 98 ++++++++++++++- src/bin/psql/copy.c | 131 ++++++++++++++++---- src/bin/psql/stringutils.c | 5 +- src/bin/psql/stringutils.h | 2 + src/include/commands/copy.h | 2 +- src/include/nodes/parsenodes.h | 1 + src/include/parser/kwlist.h | 1 + src/include/port.h | 3 + src/include/storage/fd.h | 4 + src/interfaces/ecpg/preproc/ecpg.addons | 2 +- src/port/Makefile | 3 +- src/port/exec.c | 37 ++---- src/port/wait_error.c | 92 ++++++++++++++ 21 files changed, 581 insertions(+), 149 deletions(-)
On 27 February 2013 16:22, Heikki Linnakangas <heikki.linnakangas@iki.fi> wrote: > Add support for piping COPY to/from an external program. > > This includes backend "COPY TO/FROM PROGRAM '...'" syntax, and corresponding > psql \copy syntax. Like with reading/writing files, the backend version is > superuser-only, and in the psql version, the program is run in the client. > > In the passing, the psql \copy STDIN/STDOUT syntax is subtly changed: if you > the stdin/stdout is quoted, it's now interpreted as a filename. For example, > "\copy foo from 'stdin'" now reads from a file called 'stdin', not from > standard input. Before this, there was no way to specify a filename called > stdin, stdout, pstdin or pstdout. > > This creates a new function in pgport, wait_result_to_str(), which can > be used to convert the exit status of a process, as returned by wait(3), > to a human-readable string. > > Etsuro Fujita, reviewed by Amit Kapila. A minor point: + Executing a command with <literal>PROGRAM</literal> might be restricted + by operating system's access control mechanisms, such as the SELinux. This doesn't read well. Could this instead be amended to something more like: "Executing a command with PROGRAM might be restricted by the operating system's access control mechanisms, such as those enforced by SELinux." -- Thom
On 27.02.2013 18:35, Thom Brown wrote: > On 27 February 2013 16:22, Heikki Linnakangas<heikki.linnakangas@iki.fi> wrote: >> Add support for piping COPY to/from an external program. >> >> This includes backend "COPY TO/FROM PROGRAM '...'" syntax, and corresponding >> psql \copy syntax. Like with reading/writing files, the backend version is >> superuser-only, and in the psql version, the program is run in the client. >> >> In the passing, the psql \copy STDIN/STDOUT syntax is subtly changed: if you >> the stdin/stdout is quoted, it's now interpreted as a filename. For example, >> "\copy foo from 'stdin'" now reads from a file called 'stdin', not from >> standard input. Before this, there was no way to specify a filename called >> stdin, stdout, pstdin or pstdout. >> >> This creates a new function in pgport, wait_result_to_str(), which can >> be used to convert the exit status of a process, as returned by wait(3), >> to a human-readable string. >> >> Etsuro Fujita, reviewed by Amit Kapila. > > A minor point: > > + Executing a command with<literal>PROGRAM</literal> might be restricted > + by operating system's access control mechanisms, such as the SELinux. > > This doesn't read well. Could this instead be amended to something more like: > > "Executing a command with PROGRAM might be restricted by the operating > system's access control mechanisms, such as those enforced by > SELinux." Hmm, that doesn't feel like an improvement to me. I guess your version is more accurate, if you think that SELinux provides the access control mechanism, rather than that SELinux itself is an access control mechanism. But I don't think such accuracy is important here. (I note that there is a spurious "the" in my wording, though. Should be "..., such as SELinux") - Heikki
On 27 February 2013 19:12, Heikki Linnakangas <hlinnakangas@vmware.com> wrote: > On 27.02.2013 18:35, Thom Brown wrote: >> >> On 27 February 2013 16:22, Heikki Linnakangas<heikki.linnakangas@iki.fi> >> wrote: >>> >>> Add support for piping COPY to/from an external program. >>> >>> This includes backend "COPY TO/FROM PROGRAM '...'" syntax, and >>> corresponding >>> psql \copy syntax. Like with reading/writing files, the backend version >>> is >>> superuser-only, and in the psql version, the program is run in the >>> client. >>> >>> In the passing, the psql \copy STDIN/STDOUT syntax is subtly changed: if >>> you >>> the stdin/stdout is quoted, it's now interpreted as a filename. For >>> example, >>> "\copy foo from 'stdin'" now reads from a file called 'stdin', not from >>> standard input. Before this, there was no way to specify a filename >>> called >>> stdin, stdout, pstdin or pstdout. >>> >>> This creates a new function in pgport, wait_result_to_str(), which can >>> be used to convert the exit status of a process, as returned by wait(3), >>> to a human-readable string. >>> >>> Etsuro Fujita, reviewed by Amit Kapila. >> >> >> A minor point: >> >> + Executing a command with<literal>PROGRAM</literal> might be >> restricted >> + by operating system's access control mechanisms, such as the SELinux. >> >> This doesn't read well. Could this instead be amended to something more >> like: >> >> "Executing a command with PROGRAM might be restricted by the operating >> system's access control mechanisms, such as those enforced by >> SELinux." > > > Hmm, that doesn't feel like an improvement to me. I guess your version is > more accurate, if you think that SELinux provides the access control > mechanism, rather than that SELinux itself is an access control mechanism. > But I don't think such accuracy is important here. > > (I note that there is a spurious "the" in my wording, though. Should be > "..., such as SELinux") My version also added a "the" before "operating system's". So I guess it's just a case of moving "the" from one place to another. -- Thom
On 27.02.2013 21:46, Thom Brown wrote: > On 27 February 2013 19:12, Heikki Linnakangas<hlinnakangas@vmware.com> wrote: >> (I note that there is a spurious "the" in my wording, though. Should be >> "..., such as SELinux") > > My version also added a "the" before "operating system's". So I guess > it's just a case of moving "the" from one place to another. Sounds good, committed, thanks. - Heikki
Heikki Linnakangas wrote:
> This creates a new function in pgport, wait_result_to_str(), which can
> be used to convert the exit status of a process, as returned by wait(3),
> to a human-readable string.
You have this:
+#ifndef FRONTEND
+ result = pstrdup(str);
+#else
+ result = strdup(str);
+#endif
This kind of thing is no longer necessary. In frontend, libpgcommon
provides a pstrdup() implementation which is strdup() plus error check,
so you can reduce the above to simply
result = pstrdup(str);
and it should work everywhere.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services