how the fuck i unsubscribe to this mailing list , i get more than 100 emails a day
From: Tim Cross <theophilusx@gmail.com> Sent: Wednesday, June 20, 2018 2:59 AM To: Ravi Krishna Cc: PG mailing List Subject: Re: Load data from a csv file without using COPY
Ravi Krishna <srkrishna@yahoo.com> writes:
> In order to test a real life scenario (and use it for benchmarking) I want to load large number of data from csv files. > The requirement is that the load should happen like an application writing to the database ( that is, no COPY command). > Is there a tool which can do the job. Basically parse the csv file and insert it to the database row by row. >
Not clear what you mean by 'real world scenario', but you could possibly use PG's foreign table support and define a csv file as a foreign table and then have scripts which read from there and do whatever insert/update etc you need. However, this has a high level of 'fakery' going on and probably not testing what you really want.
There are lots of ways that applications write to the database - different drivers (e.g. jdbc, odbc, pg etc), different commit and transaction strategies and even different ways to handle things like an update or insert process. You can even use streams and copy from an application.
To get 'real world' equivalence, you really need to use the same interface as the application you are comparing. Most languages have support for processing CSV files, so you may be better off writing a small 'wrapper' app which uses the same drivers and assuming your database connectivity has been abstracted into some sort of module/library/class, use the same interface to write to the database that the application uses.
