Thread: psql freezes loading large file
I have a test ".sql" file of the form: insert into testtable values ('1'); insert into testtable values ('2'); insert into testtable values ('3'); .... 100 Million Right before I call the file with "\i" I do a begin transaction. At some point during the load the process stops. After some 5+ minutes of not seeing the inserts go by I pressed CTRL+C and got the errors: psql in free(): warning: recursive call xmalloc: out of virtual memory xmalloc: out of virtual memory Tried without calling begin transaction and got the same problem. Did psql ran out of memory? When I did it without begin I did a select count(*) from the table and only 472 records got merged. Can't imagine psql running out of memory after loading such small amount of records. This is postgresql 8.1.3 Running on FreeBSD 6.1
Francisco Reyes <lists@stringsutils.com> writes: > I have a test ".sql" file of the form: > insert into testtable values ('1'); > insert into testtable values ('2'); > insert into testtable values ('3'); > .... > 100 Million I don't have the patience to run this for 10^8 rows, but the test case I did run showed absolutely zero memory growth in either psql or the backend. Are you using any nonstandard psql options? regards, tom lane
Tom Lane writes: > I don't have the patience to run this for 10^8 rows, but the test case I got suspicious of my 'test' file so I took 1000 rows. That had problems and pointed out problems with the file. It seems I had mismatched single quotes.. my guess is that psql got confused and went into into an infinite loop somewhow. With the 1000 rows sample it reported the error. I am re-doing the input file.