Hi,
I am currently writing an application which gathers some data from
machine data files and inserts it into a postgres database. The problem
is that some of these file can be quite large (eg. 2M) and these are not
text but binary files.
The insert procedure is as follows:
0. Open connection to database.
1. Look for a record call the MIR, perform an insert with data from
the MIR into the database table a.
2. Get the id of the the record which was created with this
insert.
3. Find one or more SBR records an insert each one into table c
with the reference num obtained from table a.
4. Find one or more HBR records an insert each one into table c
with the reference num obtained from table a.
5. Find a record PCR and update table a with data from the PCR
using the reference num from step 2.
6. Find a record MRR and insert data from it into table a with the
reference obtained in step 2.
7. Commit the transaction.
All seems to work fine until I found the bastard file from hell which is
22M is size. This one is exceptionaly large and whilst the MRR is found
qickly because it is the first record in the file, a very long time
(minutes) will pass until the first SBR is found upon which I get the
following when performing the insert:
openStdf: Found SBR
DB error: Error: pqReadData() -- backend closed the channel
unexpectedly.
This probably means the backend terminated abnormally before or
while processing the request.
line 348.
openStdf: Found SBR
DB error: Postgres error: PQsendQuery() -- There is no connection to the
backend.
line 348.
openStdf: Found SBR
DB error: Postgres error: PQsendQuery() -- There is no connection to the
backend.
line 348.
The process will eventually die leaving me with a 100Meg core file to
account for itself.
Do you think that the large interval between the first insert and the
second insert is causing the transaction to timeout with the loss of the
connection. If yes what can I do to avoid the time out.
Regards
Lincoln Spiteri
(STMicroelectronics, Malta)
