Postgres Pro
Downloads
Home   >   mailing lists

Thread: importing in sql

importing in sql

From
Abdul Karim
Date:
Hi, I am trying to import loads of data into postgres, I am  having
trouble with a field which is a sequence. I know how to import data
using a delimiter with the copy command. But I need to know how I
increase the sequence on each line of import.

My Table has the following format.

field1    serial primary key,
field2    char (50),
field3    char(50),
field4    int,
field5    date

The file has the following format

field2|field3|field4|field5|

Each field is separated by a | (pipe), How do I insert the sequence
number before field2? I have like 30 files in this format and I need to
import them in one table with each line having unique number.  Has
anyone come across similar situation? any help would be greatly
appreciated.

Please let me know if you need more info.

Thanks in advance.

--
Abdul Karim

The Hub Communications Company Ltd.
The Farmhouse
Syon Park
Middlesex   TW8 8JF

T: 020 8560 9222 (ext 243)
F: 020 8560 9333
Email: mailto:Karim@thehub.co.uk
URL: http://www.thehub.co.uk

Re: importing in sql

From
"Ross J. Reedstrom"
Date:
On Mon, Jul 10, 2000 at 05:03:08PM -0500, Abdul Karim wrote:
> Hi, I am trying to import loads of data into postgres, I am  having
> trouble with a field which is a sequence. I know how to import data
> using a delimiter with the copy command. But I need to know how I
> increase the sequence on each line of import.
> 
> My Table has the following format.
> 
> field1    serial primary key,
> field2    char (50),
> field3    char(50),
> field4    int,
> field5    date
> 
> The file has the following format
> 
> field2|field3|field4|field5|
> 
> Each field is separated by a | (pipe), How do I insert the sequence
> number before field2? I have like 30 files in this format and I need to
> import them in one table with each line having unique number.  Has
> anyone come across similar situation? any help would be greatly
> appreciated.
> 

you don't mention your operating system, but on Linux I'd do this
with awk:

awk '{print NR"|"$0}' data_file > data_file.out

You mention 30 files: to get them sequential, I'd do:

cat files1 file2 [...] | awk '{print NR"|"$0}' | split -C 1m - dataout

Use some glob pattern for the 'cat' command that puts the files in the
order you want (even if it's just listing them all individually)

Awk will then number them, and split will generate file with at most 1
Meg of lines in them. If you're not worried about doing an all in one
bulk load, skip the split.

After this is all loaded, be sure to set the sequence associated with
the serial filed you loading into:

SELECT setval('table_field_seq',max(field)) from table;


Ross
-- 
Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> 
NSBRI Research Scientist/Programmer
Computer and Information Technology Institute
Rice University, 6100 S. Main St.,  Houston, TX 77005

Re: importing in sql

From
jmr@computing.com
Date:
karim> The file has the following format
karim> 
karim> field2|field3|field4|field5|
karim> 
karim> Each field is separated by a | (pipe), How do I insert the sequence
karim> number before field2? I have like 30 files in this format and I need to
karim> import them in one table with each line having unique number.  Has
karim> anyone come across similar situation? any help would be greatly
karim> appreciated.

I'd create a new (temporary) table with only these 4 fields, and use
COPY to load it.  Then, insert the data into the real table from the
temp one.  Let postgres assign the serial numbers.
Something like this:    insert into <foo> (field2, field3, ...) select * from tmptable;
Then you can drop your temp table.

Jim Rowan
DCSI
jmr@computing.com