Thread: importing data from Filemaker: weird newline characters

After importing a table from Filemaker 5.5 (via COPY from a csv file),
I've got newline characters from text fields in the original Filemaker
db which appear as ^K, or control-K on the console in psql or vi. I am
wondering how to deal with those. How do I find out what it is? Are
there standards as to how to represent newlines etc. in text fields in
SQL databases?

Thanks, Frank

On Mon, Jan 21, 2002 at 05:41:03PM +0100, Frank Joerdens wrote:
> After importing a table from Filemaker 5.5 (via COPY from a csv file),
> I've got newline characters from text fields in the original Filemaker
> db which appear as ^K, or control-K on the console in psql or vi. I am
> wondering how to deal with those. How do I find out what it is? Are
> there standards as to how to represent newlines etc. in text fields in
> SQL databases?

Low, many suns ago, I also imported some text from Filemaker: that's how
FM exports newlines. Postgresql can handle newlines in it's data, by
quoting them with '\'. So a sed run on the CSV file with something like:

sed 's/^K/\\^M' <old.csv >new.csv

Where the ^K and ^M are actual control codes (generated in bash with 
^I^K and ^I^M) should do it.

Ross

On Mon, Jan 21, 2002 at 05:21:17PM -0600, Ross J. Reedstrom wrote:
> On Mon, Jan 21, 2002 at 05:41:03PM +0100, Frank Joerdens wrote:
> > After importing a table from Filemaker 5.5 (via COPY from a csv file),
> > I've got newline characters from text fields in the original Filemaker
> > db which appear as ^K, or control-K on the console in psql or vi. I am
> > wondering how to deal with those. How do I find out what it is? Are
> > there standards as to how to represent newlines etc. in text fields in
> > SQL databases?
> 
> Low, many suns ago, I also imported some text from Filemaker: that's how
> FM exports newlines. Postgresql can handle newlines in it's data, by
> quoting them with '\'. So a sed run on the CSV file with something like:
> 
> sed 's/^K/\\^M' <old.csv >new.csv
> 
> Where the ^K and ^M are actual control codes (generated in bash with 
> ^I^K and ^I^M) should do it.

Hm . . . I think I know what you mean, in that I get the idea. I feel
kinda stupid asking again since I feel I should somehow know this: No
searching in my O'Reilly sed & awk book, no permutations of your example
on my command line, and no Google (I looked at the Advanced
Bash-Scripting Guide and the Text-Terminal-HOWTO) managed to resolve the
riddle of how to generate 'actual control codes'. Just typing

sed 's/\^K/\^M/' < adressen.csv > new_adressen.csv      ^   ^  ^
(this looks a little different to your line - I think it's what you
"meant" - your line gives a syntax error with sed because of the missing
/ after ^M)

obviously doesn't work. I'm saying obviously because you are already
saying, in effect, that that's not what you mean. It's not really all
that obvious to me because . . . well, I guess it's a real gap in my
understanding of how the shell deals these with special characters . . .
and I seem to be having real difficulties in finding a resource that
would help me close it. Can you try to explain again how I might
convince sed to perform the substitution? Or even where I could read up
on the matter? (If I can't get sed to do it - which would be rather neat
- I'd probably try to find a Hex editor next, and try to figure out what
^K is there.) 

Regards, Frank

On Wed, Jan 23, 2002 at 02:14:06AM +0100, Frank Joerdens wrote:
> On Mon, Jan 21, 2002 at 05:21:17PM -0600, Ross J. Reedstrom wrote:
[ . . . ]
> > Where the ^K and ^M are actual control codes (generated in bash with 
> > ^I^K and ^I^M) should do it.

On my terminal it's Control-V, which is bound to Readline's quoted-insert
function (found that eventually). 

sed 's/^K/^M/g' old.csv > new.csv

does the trick now.

Thanks!
Frank

On Wed, Jan 23, 2002 at 03:58:18PM +0100, Frank Joerdens wrote:
> On Wed, Jan 23, 2002 at 02:14:06AM +0100, Frank Joerdens wrote:
> > On Mon, Jan 21, 2002 at 05:21:17PM -0600, Ross J. Reedstrom wrote:
> [ . . . ]
> > > Where the ^K and ^M are actual control codes (generated in bash with 
> > > ^I^K and ^I^M) should do it.
> 
> On my terminal it's Control-V, which is bound to Readline's quoted-insert
> function (found that eventually). 
> 
> sed 's/^K/^M/g' old.csv > new.csv
> 

That's exactly what I meant. In fact, it _is_ control-V on my keyboard,
as well: don't know how control-I slipped in there. Sorry!

Ross