Thread: Importing undelimited files (Flat Files or Fixed-Length records)

I've got to load some large fixed-legnth ASCII records into PG and I was
wondering how this is done. The Copy command looks like it works only
with delimited files, and I would hate to have to convert these files to
INSERT-type SQL to run them through psql.. Is there a way one can
specify a table structure with raw field widths and then just pass it a
flat file?

On Thu, Jun 19, 2008 at 6:54 PM, Bill Thoen <bthoen@gisnet.com> wrote:
> I've got to load some large fixed-legnth ASCII records into PG and I was
> wondering how this is done. The Copy command looks like it works only with
> delimited files, and I would hate to have to convert these files to
> INSERT-type SQL to run them through psql.. Is there a way one can specify a
> table structure with raw field widths and then just pass it a flat file?

You'll need to use something external to PG, like sed, awk or Perl, to
convert to a delimited file, then load with COPY.

-Doug

Bill Thoen asked:

>
> I've got to load some large fixed-legnth ASCII records into PG and I was
> wondering how this is done. The Copy command looks like it works only
> with delimited files, and I would hate to have to convert these files to
> INSERT-type SQL to run them through psql.. Is there a way one can
> specify a table structure with raw field widths and then just pass it a
> flat file?

One possibility might be to create a table with one column of type text, load the data into that table with copy, and then, after creating the real table structure, populate it with the data by using substr and casts to get it into shape.

HTH,

Greg Williamson
Senior DBA
DigitalGlobe

Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information and must be protected in accordance with those provisions. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

(My corporate masters made me say this.)

On Thursday 19 June 2008 3:54 pm, Bill Thoen wrote:
> I've got to load some large fixed-legnth ASCII records into PG and I was
> wondering how this is done. The Copy command looks like it works only
> with delimited files, and I would hate to have to convert these files to
> INSERT-type SQL to run them through psql.. Is there a way one can
> specify a table structure with raw field widths and then just pass it a
> flat file?

Take a look at:
http://pgfoundry.org/projects/pgloader/

Latest version has support for fixed length records.
--
Adrian Klaver
aklaver@comcast.net

In article <485AE3AA.3030700@gisnet.com>, Bill Thoen <bthoen@gisnet.com> wrote:
% I've got to load some large fixed-legnth ASCII records into PG and I was
% wondering how this is done. The Copy command looks like it works only
% with delimited files, and I would hate to have to convert these files to
% INSERT-type SQL to run them through psql.. Is there a way one can
% specify a table structure with raw field widths and then just pass it a
% flat file?

pg_loader is supposed to handle this.

http://pgfoundry.org/projects/pgloader

--

Patrick TJ McPhee
North York  Canada
ptjm@interlog.com

Patrick TJ McPhee wrote:
> In article <485AE3AA.3030700@gisnet.com>, Bill Thoen <bthoen@gisnet.com> wrote:
> % I've got to load some large fixed-legnth ASCII records into PG and I was
> % wondering how this is done. The Copy command looks like it works only
> % with delimited files, and I would hate to have to convert these files to
> % INSERT-type SQL to run them through psql.. Is there a way one can
> % specify a table structure with raw field widths and then just pass it a
> % flat file?
>
> pg_loader is supposed to handle this.
>
> http://pgfoundry.org/projects/pgloader
>
>

Thanks, but as it turned out I also had to skip blank lines, so I wrote
a short sed script and piped the data through that into COPY. That
worked just fine.