Re: [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION - Mailing list pgsql-hackers

Tom Lane wrote:
> Those discussions don't have a lot of credibility if they didn't take
> place on the public mailing lists.
>   
You know how people complain about how new contributors are treated 
here?  Throwing out comments like this, that come off as belittling to 
other people's work, doesn't help.  All I was suggesting was that Dan 
wasn't developing this in complete isolation from the hackers community 
as Robert had feared, as will be obvious when we get to:

> pgsql-hackers had some preliminary discussions a couple months back
> on refactoring COPY to allow things like this --- see the thread
> starting here:
> http://archives.postgresql.org/pgsql-hackers/2009-09/msg00486.php
> While I don't think we arrived at any final decisions, I would like
> to know how this design fits in with what was discussed then.
>   
The patch provided here is a first step toward what you suggested in the 
related "Copy enhancements thread" a few days later, at 
http://archives.postgresql.org/pgsql-hackers/2009-09/msg00616.php  It's 
one way to implement a better decoupled "data transformation" layer on 
top of COPY.  When Dan showed me an earlier implementation of the basic 
idea embodied in this patch (developed independently and earlier 
actually), I gave it a thumbs-up as seeming to match the general 
direction the community discussion suggested, and encouraged him to work 
on getting the code to where it could be released.  He started with 
output rather than input, mainly because the dblink feature had a 
compelling use-case that justified spending time on development for the 
company.  Rather than keep going into input transformation, this 
development checkpoint made a good place to pause and solicit public 
feedback, particularly since the input side has additional hurdles to 
clear before it can work.

As far as other past discussion here that might be relevant, this patch 
includes a direct change to gram.y to support the new syntax.  You've 
already suggested before that it might be time to update COPY the same 
way EXPLAIN and now VACUUM have been overhauled to provide a more 
flexible options interface:  
http://archives.postgresql.org/pgsql-hackers/2009-09/msg00616.php  This 
patch might be more fuel for that idea.

Emmanuel has given up the more error tolerant COPY patch that thread was 
associated with, and I haven't heard anything from Andrew about ragged 
CVS import either.  I think that ultimately those features are useful, 
but just exceed what the existing code could be hacked to handle 
cleanly.  If it's possible to lower the complexity bar to implementing 
them by abstracting the transformation into a set of functions, and have 
more flexible syntax for the built-in ones the database ships with, that 
may be useful groundwork for returning to implementing those features 
without bloating COPY's internals (and therefore performance) 
intolerably.  Dan even suggested in his first note that it might be 
possible to write the current STDOUT|file behavior in terms of the new 
function interface, which has the potential to make the COPY 
implementation cleaner rather than more cluttered (as long as the 
performance doesn't suffer).

-- 
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com