Re: Batch Operations - Mailing list pgsql-hackers

From Christopher Browne
Subject Re: Batch Operations
Date
Msg-id m3bruy94tf.fsf@chvatal.cbbrowne.com
Whole thread Raw
In response to Batch Operations  ("Rahul_Iyer" <rahul_iyer@persistent.co.in>)
Responses Re: Batch Operations
List pgsql-hackers
In an attempt to throw the authorities off his trail, bruno@wolff.to (Bruno Wolff III) transmitted:
> On Fri, Aug 08, 2003 at 15:32:05 +0530,
>   Rahul_Iyer <rahul_iyer@persistent.co.in> wrote:
>> hi, im currently working on a project that requires batch
>> operations - eg. Batch insert/update etc. The database im using is
>> PostgreSQL. However, i cannot find any documentation for batch
>> opeartions on PostgreSQL. Does anyone know how to do this, or if it
>> is possible?
>
> Usually you just use psql to run a script with the commands in it.

Ah, but I rather think he's looking for some place to queue up sets of
operations inside the database, or inside some sort of "middleware"
application.

In SAP's R/3 application, automated batch processes are set up so that
the updates that they are doing are stored "serialized" so that you
can try and possibly retry them.

It's almost exactly analagous to the notion of taking the set of URLs
and FORM POSTs involved in navigating through a web application and
storing them somewhere so that you could rerun data entry.

To do things that way requires having quite a lot of application
infrastructure; the right answer probably is, if the input is a set of
SQL statements, to stick them in a file, and load the file.

If this is the sort of thing someone needed to do and redo, and they
had a lot of these files, it might make sense to make up a naming
convention for files in a directory, perhaps parallelling Maildir, and
have a daemon that looks there and processes whatever gets "spooled."

If you have a lot of reporting processes to throw at a "reports
server," this approach may make a lot of sense.  If the DB server is a
4 CPU box, then you surely don't want to flood it by running 28
reports at once.  Spooling the requests allows them to be processed
two or three or four at a time, and the other 25 will wait until the
predecessors are complete.
-- 
output = reverse("moc.enworbbc" "@" "enworbbc")
http://www3.sympatico.ca/cbbrowne/linuxxian.html
Signs of  a  Klingon  Programmer -  8.  "Debugging?   Klingons do  not
debug.  Our software  does not   coddle the  weak. Bugs  are  good for
building character in the user."


pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: 7.4beta1 build problem on unixware
Next
From: Larry Rosenman
Date:
Subject: Can't load a 7.3.4 dump into 7.4CVS