Re: Importing data from csv - Mailing list pgsql-sql

Well, being that there isn't a RFC for CSV...other than "defacto" definitions...I am pretty sure that is widely agreed
upon;)<br /><br /> Phillip Smith wrote: <blockquote cite="mid000501c6c7da$15a01b10$9b0014ac@ITPhil"
type="cite"><style>
<!--/* Font Definitions */@font-face{font-family:Wingdings;panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face{font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;}/* Style Definitions */p.MsoNormal, li.MsoNormal,
div.MsoNormal{margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;font-family:"TimesNew Roman";color:black;}
 
a:link, span.MsoHyperlink{color:blue;text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed{color:purple;text-decoration:underline;}
p{margin-right:0cm;margin-left:0cm;font-size:12.0pt;font-family:"Times New Roman";color:black;}
pre{margin:0cm;margin-bottom:.0001pt;font-size:10.0pt;font-family:"Courier New";color:black;}
span.emailstyle17{font-family:Arial;color:navy;}
span.EmailStyle20{font-family:Arial;color:navy;}
@page Section1{size:595.3pt 841.9pt;margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.Section1{page:Section1;}
--> </style><div class="Section1"><p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:
10pt;font-family: Arial; color: navy;">There you go – it was the quotes that did it, not the back-slashes. I knew
someoneelse would shed some better light! </span></font><font color="navy" face="Wingdings" size="2"><span
style="font-size:10pt; font-family: Wingdings; color: navy;">J</span></font><p class="MsoNormal"><font color="navy"
face="Arial"size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font><p
class="MsoNormal"><fontcolor="navy" face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color:
navy;">Cheers,</span></font><pclass="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt;
font-family:Arial; color: navy;">-p</span></font><p class="MsoNormal"><font color="navy" face="Arial" size="2"><span
style="font-size:10pt; font-family: Arial; color: navy;"> </span></font><p class="MsoNormal" style="margin-left:
36pt;"><fontcolor="black" face="Tahoma" size="2"><span lang="EN-US" style="font-size: 10pt; font-family: Tahoma; color:
windowtext;">-----OriginalMessage-----<br /><b><span style="font-weight: bold;">From:</span></b> <a
class="moz-txt-link-abbreviated"href="mailto:pgsql-sql-owner@postgresql.org">pgsql-sql-owner@postgresql.org</a> [<a
class="moz-txt-link-freetext"href="mailto:pgsql-sql-owner@postgresql.org">mailto:pgsql-sql-owner@postgresql.org</a>]
<b><spanstyle="font-weight: bold;">On Behalf Of </span></b>Scot P. Floess<br /><b><span style="font-weight:
bold;">Sent:</span></b>Friday, 25 August 2006 10:00<br /><b><span style="font-weight: bold;">To:</span></b> <a
class="moz-txt-link-abbreviated"href="mailto:floess@mindspring.com">floess@mindspring.com</a><br /><b><span
style="font-weight:bold;">Cc:</span></b> Phillip Smith; <a class="moz-txt-link-abbreviated"
href="mailto:pgsql-sql@postgresql.org">pgsql-sql@postgresql.org</a><br/><b><span style="font-weight:
bold;">Subject:</span></b>Re: [SQL] Importing data from csv</span></font><p class="MsoNormal" style="margin-left:
36pt;"><fontcolor="black" face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font><p
class="MsoNormal"style="margin-left: 36pt;"><font color="black" face="Times New Roman" size="3"><span style="font-size:
12pt;">Andif its contained with quotes...its considered a field<br /><br /> Scot P. Floess wrote: </span></font><p
class="MsoNormal"style="margin-left: 36pt;"><font color="black" face="Times New Roman" size="3"><span style="font-size:
12pt;">Anewline in CSV parlance denotes the end of a record....unless that newline is contained with quotes...<br /><br
/>Phillip Smith wrote: </span></font><p class="MsoNormal" style="margin-left: 36pt;"><font color="navy" face="Arial"
size="2"><spanstyle="font-size: 10pt; font-family: Arial; color: navy;">I recently did this by parsing the data through
aVB program that appended a “\” in front of any Char(10) and/or Char(13) characters which tells Postgres to accept the
nextcharacter as a literal part of the column value I believe – must do because it worked! I also quoted the whole
columnas part of the VB prog…</span></font><p class="MsoNormal" style="margin-left: 36pt;"><font color="navy"
face="Arial"size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font><p
class="MsoNormal"style="margin-left: 36pt;"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt;
font-family:Arial; color: navy;">Worked for me but I’m not sure the exact science behind it so someone else might be
ableto be of some more detailed help.</span></font><p class="MsoNormal" style="margin-left: 36pt;"><font color="navy"
face="Arial"size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;"> </span></font><p
class="MsoNormal"style="margin-left: 36pt;"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt;
font-family:Arial; color: navy;">Cheers,</span></font><p class="MsoNormal" style="margin-left: 36pt;"><font
color="navy"face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial; color: navy;">-p</span></font><p
class="MsoNormal"style="margin-left: 36pt;"><font color="navy" face="Arial" size="2"><span style="font-size: 10pt;
font-family:Arial; color: navy;"> </span></font><p class="MsoNormal" style="margin-left: 72pt;"><font color="black"
face="Tahoma"size="2"><span lang="EN-US" style="font-size: 10pt; font-family: Tahoma;">-----Original Message-----<br
/><b><spanstyle="font-weight: bold;">From:</span></b> <a
href="mailto:pgsql-sql-owner@postgresql.org">pgsql-sql-owner@postgresql.org</a>[<a
href="mailto:pgsql-sql-owner@postgresql.org">mailto:pgsql-sql-owner@postgresql.org</a>]<b><span style="font-weight:
bold;">OnBehalf Of </span></b>Sumeet<br /><b><span style="font-weight: bold;">Sent:</span></b> Friday, 25 August 2006
00:48<br/><b><span style="font-weight: bold;">To:</span></b> <a
href="mailto:pgsql-sql@postgresql.org">pgsql-sql@postgresql.org</a><br/><b><span style="font-weight:
bold;">Subject:</span></b>[SQL] Importing data from csv</span></font><p class="MsoNormal" style="margin-left:
72pt;"><fontcolor="black" face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font><p
class="MsoNormal"style="margin-left: 72pt;"><font color="black" face="Times New Roman" size="3"><span style="font-size:
12pt;">HiFolks,<br /><br /> sorry if this is a duplicate post, i've been tryin to find a solution of importing data
intopostgres from a csv file. The problem is, I have a database which consists of columns which contain newline
characters(mac and unix). now when i export these files to a csv format, there are some line breaks (mixed unix and
mac)in the data which breaks the copy procedure. <br /><br /> I also tried using the script posted in one of the
previousposts..<br clear="all" /><br /> #! /usr/bin/perl<br /> $inquotes = 0;<br /> while (<>){<br />      # Chop
thecrlf<br />      chop ($_);<br />      chop ($_);<br /><br />      # this first bit goes through and replaces <br />
    # all the commas that re not in  quotes with tildes<br />      for ($i=0 ; $i < length($_) ; $i++){<br />
        $char=substr($_,$i,1);<br />          if ($char eq '"' ){<br />              $inquotes = not($inquotes); <br />
        }else{<br />              if ( (!$inquotes) && ($char eq ",") ){<br />                 
substr($_,$i,1)="~";<br/>              }<br />          }<br />      }<br />      # this replaces any quotes<br />     
s/"//g;<br/>      print "$_\n";<br /> }<br /><br /><br /> cat data_file | perl scriptname.pl > outputfile.dat<br
/><br/> and when i run the copy command i get messages like data missing for xyz column.<br /> any possible
hints.......<br /><br /> --<br /> Thanks,<br /> Sumeet </span></font><p class="MsoNormal" style="margin-left:
36pt;"><fontcolor="black" face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font><p
style="margin-left:36pt;"><b><font color="black" face="Times New Roman" size="3"><span style="font-size: 12pt;
font-weight:bold;">*******************Confidentiality and Privilege Notice*******************</span></font></b><p
style="margin-left:36pt;"><font color="black" face="Times New Roman" size="3"><span style="font-size: 12pt;">The
materialcontained in this message is privileged and confidential to the addressee. If you are not the addressee
indicatedin this message or responsible for delivery of the message to such person, you may not copy or deliver this
messageto anyone, and you should destroy it and kindly notify the sender by reply email. </span></font><p
style="margin-left:36pt;"><font color="black" face="Times New Roman" size="3"><span style="font-size:
12pt;">Informationin this message that does not relate to the official business of Weatherbeeta must be treated as
neithergiven nor endorsed by Weatherbeeta. Weatherbeeta, its employees, contractors or associates shall not be liable
fordirect, indirect or consequential loss arising from transmission of this message or any attachments </span></font><p
class="MsoNormal"style="margin-left: 36pt;"><font color="black" face="Times New Roman" size="3"><span style="font-size:
12pt;"><br/><br /></span></font><pre style="margin-left: 36pt;"><font color="black" face="Courier New" size="2"><span
style="font-size:10pt;">-- </span></font></pre> <pre style="margin-left: 36pt;"><font color="black" face="Courier New"
size="2"><spanstyle="font-size: 10pt;">Scot P. Floess</span></font></pre> <pre style="margin-left: 36pt;"><font
color="black"face="Courier New" size="2"><span style="font-size: 10pt;">27 Lake Royale</span></font></pre> <pre
style="margin-left:36pt;"><font color="black" face="Courier New" size="2"><span style="font-size: 10pt;">Louisburg, NC 
27549</span></font></pre><pre style="margin-left: 36pt;"><font color="black" face="Courier New" size="2"><span
style="font-size:10pt;"> </span></font></pre> <pre style="margin-left: 36pt;"><font color="black" face="Courier New"
size="2"><spanstyle="font-size: 10pt;">252-478-8087 (Home)</span></font></pre> <pre style="margin-left: 36pt;"><font
color="black"face="Courier New" size="2"><span style="font-size: 10pt;">919-754-4592 (Work)</span></font></pre> <pre
style="margin-left:36pt;"><font color="black" face="Courier New" size="2"><span style="font-size:
10pt;"> </span></font></pre><pre style="margin-left: 36pt;"><font color="black" face="Courier New" size="2"><span
style="font-size:10pt;">Chief Architect JPlate  <a
href="http://sourceforge.net/projects/jplate">http://sourceforge.net/projects/jplate</a></span></font></pre><pre
style="margin-left:36pt;"><font color="black" face="Courier New" size="2"><span style="font-size: 10pt;">Chief
ArchitectJavaPIM <a
href="http://sourceforge.net/projects/javapim">http://sourceforge.net/projects/javapim</a></span></font></pre><pre
style="margin-left:36pt;"><font color="black" face="Courier New" size="2"><span style="font-size: 10pt;"> 
</span></font></pre><pclass="MsoNormal" style="margin-left: 36pt;"><font color="black" face="Times New Roman"
size="3"><spanstyle="font-size: 12pt;"><br /><br /></span></font><pre style="margin-left: 36pt;"><font color="black"
face="CourierNew" size="2"><span style="font-size: 10pt;">-- </span></font></pre> <pre style="margin-left: 36pt;"><font
color="black"face="Courier New" size="2"><span style="font-size: 10pt;">Scot P. Floess</span></font></pre> <pre
style="margin-left:36pt;"><font color="black" face="Courier New" size="2"><span style="font-size: 10pt;">27 Lake
Royale</span></font></pre><pre style="margin-left: 36pt;"><font color="black" face="Courier New" size="2"><span
style="font-size:10pt;">Louisburg, NC  27549</span></font></pre> <pre style="margin-left: 36pt;"><font color="black"
face="CourierNew" size="2"><span style="font-size: 10pt;"> </span></font></pre> <pre style="margin-left: 36pt;"><font
color="black"face="Courier New" size="2"><span style="font-size: 10pt;">252-478-8087 (Home)</span></font></pre> <pre
style="margin-left:36pt;"><font color="black" face="Courier New" size="2"><span style="font-size: 10pt;">919-754-4592
(Work)</span></font></pre><pre style="margin-left: 36pt;"><font color="black" face="Courier New" size="2"><span
style="font-size:10pt;"> </span></font></pre> <pre style="margin-left: 36pt;"><font color="black" face="Courier New"
size="2"><spanstyle="font-size: 10pt;">Chief Architect JPlate  <a
href="http://sourceforge.net/projects/jplate">http://sourceforge.net/projects/jplate</a></span></font></pre><pre
style="margin-left:36pt;"><font color="black" face="Courier New" size="2"><span style="font-size: 10pt;">Chief
ArchitectJavaPIM <a
href="http://sourceforge.net/projects/javapim">http://sourceforge.net/projects/javapim</a></span></font></pre></div><br
/><p><b>*******************Confidentialityand Privilege Notice*******************</b><p>The material contained in this
messageis privileged and confidential to the addressee. If you are not the addressee indicated in this message or
responsiblefor delivery of the message to such person, you may not copy or deliver this message to anyone, and you
shoulddestroy it and kindly notify the sender by reply email. <p>Information in this message that does not relate to
theofficial business of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta. Weatherbeeta, its
employees,contractors or associates shall not be liable for direct, indirect or consequential loss arising from
transmissionof this message or any attachments <br /></blockquote><br /><pre class="moz-signature" cols="72">-- 
 
Scot P. Floess
27 Lake Royale
Louisburg, NC  27549

252-478-8087 (Home)
919-754-4592 (Work)

Chief Architect JPlate  <a class="moz-txt-link-freetext"
href="http://sourceforge.net/projects/jplate">http://sourceforge.net/projects/jplate</a>
Chief Architect JavaPIM <a class="moz-txt-link-freetext"
href="http://sourceforge.net/projects/javapim">http://sourceforge.net/projects/javapim</a>
</pre>