On 01/16/2017 06:07 PM, Jim Nasby wrote:
> On 1/16/17 7:32 PM, Adrian Klaver wrote:
>> All of this is very interesting and definitely worth exploring, just not
>> sure how much of it ties back to psycopg2 and this list. Not trying to
>> rain on anyone's parade, I am wondering if this might not be better
>> explored on a 'meta' list, something like the various Python projects
>> that deal with Excel do:
>
> Since this is a user mailing list that might make sense. Though, I'm
> getting the impression that there's some disconnect between what data
> science users are doing and this list. Tuple-based results vs
> vector-based (ie: columnar) results is an example of that.
Well we are talking about improving the flow of data in and out of a
database and to the extent that can be improved all users that depend on
that benefit. The issue then becomes the optimizations for the various
subsets of users. I suspect that will involve more then one solution and
quite possibly multiple solutions for any particular case. Seems to me a
place where that can be discussed on a broader scale would lead to a
cross discipline conversation that would be advantageous to all. Because
in a sense we are all data scientists.
>
> I do think there's 3 items that would best be handled at the "bottom" of
> the stack (namely, psycopg2), because they'll enable every higher level
> as well as make life easier for direct users of psycopg2:
>
> 1) Performance, both in low-latency (ie: filesystem socket) and
> high-latency environments.
> 2) Type conversion (in particular, getting rid of strings as the
> intermediate representation).
> 3) Optionally providing a columnar result set.
Seems a good starting point, though I have to admit my needs are fairly
simple and are already met by the existing implementation.
>
> #3 might be in direct opposition to the standard Python DB accessor
> stuff, so maybe that would need to be a separate module on top of
> psycopg2, but psycopg2 would certainly still need to support it. (IE:
> you certainly do NOT want psycopg2 to build a list of dicts only to then
> try and convert that to a columnar format).
--
Adrian Klaver
adrian.klaver@aklaver.com
