Hi Adrian & Tom,
On Sat, 30 Sep 2023 15:57:32 -0700 Adrian Klaver wrote:
>On 9/30/23 14:54, pf@pfortin.com wrote:
>> On Sat, 30 Sep 2023 13:40:37 -0700 Adrian Klaver wrote:
>>
>>> On 9/30/23 11:32, pf@pfortin.com wrote:
>
>>
>> As I told Tom, the "test" DB has this issue; the production and test1 DBs
>> are fine; I should have thought to check those first... Sorry for the
>> noise.
>
>Still there was an issue with a database. Did you track down what was
>wrong with "test"?
Background: it's one of 18 databases on:
/dev/nvme1n1p1 3.6T 1.3T 2.2T 38% /mnt/work
Tried using a never before used table name; same error. I wondered if an
old SEQUENCE might be the cause; nope.
OK.. after:
* back up a table
* try import -- fails
* drop backed-up table
rinse and repeat until the test database is empty..
Still fails.
Backed up the empty db (attached).
I see the string "SET client_encoding = 'WIN1252';" in the dump -- some
files come from a remote colleague; but this has never been an issue
before...
Next, I was going to connect to test1 with SQL-workbench/J when I saw
this: "Cannot invoke "workbench.storage.RowData.getValue(int)" because
"<local4>" is null" on the SQL Source tab.
Emptied test1. Imported the table just fine. Dropped it. Dumped the db
(attached) which is 2 bytes smaller. "test1" appears 4 times, so this
dump should be 2 bytes larger, not smaller; it also says 'UTF8' instead
of 'WIN1252'. Lots of other diffs...
Hope there's something in there you guys can make sense of...
Regards,
Pierre
>> We were working with a python script (using INSERT) someone wrote a couple
>> years ago; with my version using COPY[1], imports have gone from 2+ hours
>> to under a minute each for 8M and 33M row files. If the above links help
>> even more that will be amazing!! THANK YOU!!
>
>The move to COPY will have accounted for the speed up. Using the
>psycopg2 COPY methods probably won't make a material difference, I just
>find them easier to use. Also in the new psycopg(3) there are more ways
>to use them per:
>
> https://www.psycopg.org/psycopg3/docs/basic/copy.html.
>
>
>
>
