On 1/16/17 7:32 PM, Adrian Klaver wrote:
> All of this is very interesting and definitely worth exploring, just not
> sure how much of it ties back to psycopg2 and this list. Not trying to
> rain on anyone's parade, I am wondering if this might not be better
> explored on a 'meta' list, something like the various Python projects
> that deal with Excel do:
Since this is a user mailing list that might make sense. Though, I'm
getting the impression that there's some disconnect between what data
science users are doing and this list. Tuple-based results vs
vector-based (ie: columnar) results is an example of that.
I do think there's 3 items that would best be handled at the "bottom" of
the stack (namely, psycopg2), because they'll enable every higher level
as well as make life easier for direct users of psycopg2:
1) Performance, both in low-latency (ie: filesystem socket) and
high-latency environments.
2) Type conversion (in particular, getting rid of strings as the
intermediate representation).
3) Optionally providing a columnar result set.
#3 might be in direct opposition to the standard Python DB accessor
stuff, so maybe that would need to be a separate module on top of
psycopg2, but psycopg2 would certainly still need to support it. (IE:
you certainly do NOT want psycopg2 to build a list of dicts only to then
try and convert that to a columnar format).
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)
