Re: Importing data from csv - Mailing list pgsql-sql

<div class="Section1"><p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:
10.0pt;font-family:Arial;color:navy">There you go – it was the quotes that did it, not the back-slashes. I knew someone
elsewould shed some better light! </span></font><font color="navy" face="Wingdings" size="2"><span
style="font-size:10.0pt;font-family:Wingdings;color:navy">J</span></font><pclass="MsoNormal"><font color="navy"
face="Arial"size="2"><span style="font-size: 
10.0pt;font-family:Arial;color:navy"> </span></font><p class="MsoNormal"><font color="navy" face="Arial" size="2"><span
style="font-size:
10.0pt;font-family:Arial;color:navy">Cheers,</span></font><p class="MsoNormal"><font color="navy" face="Arial"
size="2"><spanstyle="font-size: 
10.0pt;font-family:Arial;color:navy">-p</span></font><p class="MsoNormal"><font color="navy" face="Arial"
size="2"><spanstyle="font-size: 
10.0pt;font-family:Arial;color:navy"> </span></font><p class="MsoNormal" style="margin-left:36.0pt"><font color="black"
face="Tahoma"size="2"><span lang="EN-US" style="font-size:10.0pt;font-family:Tahoma; 
color:windowtext">-----Original Message-----<br /><b><span style="font-weight:bold">From:</span></b>
pgsql-sql-owner@postgresql.org[mailto:pgsql-sql-owner@postgresql.org] <b><span style="font-weight:bold">On Behalf Of
</span></b>ScotP. Floess<br /><b><span style="font-weight:bold">Sent:</span></b> Friday, 25 August 2006 10:00<br
/><b><spanstyle="font-weight:bold">To:</span></b> floess@mindspring.com<br /><b><span
style="font-weight:bold">Cc:</span></b>Phillip Smith; pgsql-sql@postgresql.org<br /><b><span
style="font-weight:bold">Subject:</span></b>Re: [SQL] Importing data from csv</span></font><p class="MsoNormal"
style="margin-left:36.0pt"><fontcolor="black" face="Times New Roman" size="3"><span
style="font-size:12.0pt"> </span></font><pclass="MsoNormal" style="margin-left:36.0pt"><font color="black" face="Times
NewRoman" size="3"><span style="font-size:12.0pt">And if its contained with quotes...its considered a field<br /><br />
ScotP. Floess wrote: </span></font><p class="MsoNormal" style="margin-left:36.0pt"><font color="black" face="Times New
Roman"size="3"><span style="font-size:12.0pt">A newline in CSV parlance denotes the end of a record....unless that
newlineis contained with quotes...<br /><br /> Phillip Smith wrote: </span></font><p class="MsoNormal"
style="margin-left:36.0pt"><fontcolor="navy" face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial;color:navy">Irecently did this by parsing the data through a VB program that
appendeda “\” in front of any Char(10) and/or Char(13) characters which tells Postgres to accept the next character as
aliteral part of the column value I believe – must do because it worked! I also quoted the whole column as part of the
VBprog…</span></font><p class="MsoNormal" style="margin-left:36.0pt"><font color="navy" face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><pclass="MsoNormal"
style="margin-left:36.0pt"><fontcolor="navy" face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial;color:navy">Workedfor me but I’m not sure the exact science behind it so
someoneelse might be able to be of some more detailed help.</span></font><p class="MsoNormal"
style="margin-left:36.0pt"><fontcolor="navy" face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><pclass="MsoNormal"
style="margin-left:36.0pt"><fontcolor="navy" face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial;color:navy">Cheers,</span></font><pclass="MsoNormal"
style="margin-left:36.0pt"><fontcolor="navy" face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial;color:navy">-p</span></font><pclass="MsoNormal"
style="margin-left:36.0pt"><fontcolor="navy" face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><pclass="MsoNormal"
style="margin-left:72.0pt"><fontcolor="black" face="Tahoma" size="2"><span lang="EN-US"
style="font-size:10.0pt;font-family:Tahoma">-----OriginalMessage-----<br /><b><span
style="font-weight:bold">From:</span></b><a
href="mailto:pgsql-sql-owner@postgresql.org">pgsql-sql-owner@postgresql.org</a>[<a
href="mailto:pgsql-sql-owner@postgresql.org">mailto:pgsql-sql-owner@postgresql.org</a>]<b><span
style="font-weight:bold">OnBehalf Of </span></b>Sumeet<br /><b><span style="font-weight:bold">Sent:</span></b> Friday,
25August 2006 00:48<br /><b><span style="font-weight:bold">To:</span></b> <a
href="mailto:pgsql-sql@postgresql.org">pgsql-sql@postgresql.org</a><br/><b><span
style="font-weight:bold">Subject:</span></b>[SQL] Importing data from csv</span></font><p class="MsoNormal"
style="margin-left:72.0pt"><fontcolor="black" face="Times New Roman" size="3"><span
style="font-size:12.0pt"> </span></font><pclass="MsoNormal" style="margin-left:72.0pt"><font color="black" face="Times
NewRoman" size="3"><span style="font-size:12.0pt">Hi Folks,<br /><br /> sorry if this is a duplicate post, i've been
tryinto find a solution of importing data into postgres from a csv file. The problem is, I have a database which
consistsof columns which contain newline characters (mac and unix). now when i export these files to a csv format,
thereare some line breaks (mixed unix and mac) in the data which breaks the copy procedure. <br /><br /> I also tried
usingthe script posted in one of the previous posts..<br clear="all" /><br /> #! /usr/bin/perl<br /> $inquotes = 0;<br
/>while (<>){<br />      # Chop the crlf<br />      chop ($_);<br />      chop ($_);<br /><br />      # this
firstbit goes through and replaces <br />      # all the commas that re not in  quotes with tildes<br />      for ($i=0
;$i < length($_) ; $i++){<br />          $char=substr($_,$i,1);<br />          if ($char eq '"' ){<br />
            $inquotes = not($inquotes); <br />          }else{<br />              if ( (!$inquotes) && ($char
eq",") ){<br />                  substr($_,$i,1)="~";<br />              }<br />          }<br />      }<br />      #
thisreplaces any quotes<br />      s/"//g;<br />      print "$_\n";<br /> }<br /><br /><br /> cat data_file | perl
scriptname.pl> outputfile.dat<br /><br /> and when i run the copy command i get messages like data missing for xyz
column.<br/> any possible hints....... <br /><br /> --<br /> Thanks,<br /> Sumeet </span></font><p class="MsoNormal"
style="margin-left:36.0pt"><fontcolor="black" face="Times New Roman" size="3"><span
style="font-size:12.0pt"> </span></font><pstyle="margin-left:36.0pt"><b><font color="black" face="Times New Roman"
size="3"><spanstyle="font-size:12.0pt;font-weight:bold">*******************Confidentiality and Privilege
Notice*******************</span></font></b><pstyle="margin-left:36.0pt"><font color="black" face="Times New Roman"
size="3"><spanstyle="font-size:12.0pt">The material contained in this message is privileged and confidential to the
addressee.If you are not the addressee indicated in this message or responsible for delivery of the message to such
person,you may not copy or deliver this message to anyone, and you should destroy it and kindly notify the sender by
replyemail. </span></font><p style="margin-left:36.0pt"><font color="black" face="Times New Roman" size="3"><span
style="font-size:12.0pt">Informationin this message that does not relate to the official business of Weatherbeeta must
betreated as neither given nor endorsed by Weatherbeeta. Weatherbeeta, its employees, contractors or associates shall
notbe liable for direct, indirect or consequential loss arising from transmission of this message or any attachments
</span></font><pclass="MsoNormal" style="margin-left:36.0pt"><font color="black" face="Times New Roman" size="3"><span
style="font-size:12.0pt"><br/><br /></span></font><pre style="margin-left:36.0pt"><font color="black" face="Courier
New"size="2"><span style="font-size:10.0pt">-- </span></font></pre><pre style="margin-left:36.0pt"><font color="black"
face="CourierNew" size="2"><span style="font-size:10.0pt">Scot P. Floess</span></font></pre><pre
style="margin-left:36.0pt"><fontcolor="black" face="Courier New" size="2"><span style="font-size:10.0pt">27 Lake
Royale</span></font></pre><prestyle="margin-left:36.0pt"><font color="black" face="Courier New" size="2"><span
style="font-size:10.0pt">Louisburg,NC  27549</span></font></pre><pre style="margin-left:36.0pt"><font color="black"
face="CourierNew" size="2"><span style="font-size:10.0pt"> </span></font></pre><pre style="margin-left: 
36.0pt"><font color="black" face="Courier New" size="2"><span style="font-size:
10.0pt">252-478-8087 (Home)</span></font></pre><pre style="margin-left:36.0pt"><font color="black" face="Courier New"
size="2"><spanstyle="font-size:10.0pt">919-754-4592 (Work)</span></font></pre><pre style="margin-left:36.0pt"><font
color="black"face="Courier New" size="2"><span style="font-size:10.0pt"> </span></font></pre><pre style="margin-left: 
36.0pt"><font color="black" face="Courier New" size="2"><span style="font-size:
10.0pt">Chief Architect JPlate  <a
href="http://sourceforge.net/projects/jplate">http://sourceforge.net/projects/jplate</a></span></font></pre><pre
style="margin-left:36.0pt"><fontcolor="black" face="Courier New" size="2"><span style="font-size:10.0pt">Chief
ArchitectJavaPIM <a
href="http://sourceforge.net/projects/javapim">http://sourceforge.net/projects/javapim</a></span></font></pre><pre
style="margin-left:36.0pt"><fontcolor="black" face="Courier New" size="2"><span style="font-size:10.0pt"> 
</span></font></pre><pclass="MsoNormal" style="margin-left:36.0pt"><font color="black" face="Times New Roman"
size="3"><spanstyle="font-size:12.0pt"><br /><br /></span></font><pre style="margin-left:36.0pt"><font color="black"
face="CourierNew" size="2"><span style="font-size:10.0pt">-- </span></font></pre><pre style="margin-left:36.0pt"><font
color="black"face="Courier New" size="2"><span style="font-size:10.0pt">Scot P. Floess</span></font></pre><pre
style="margin-left:36.0pt"><fontcolor="black" face="Courier New" size="2"><span style="font-size:10.0pt">27 Lake
Royale</span></font></pre><prestyle="margin-left:36.0pt"><font color="black" face="Courier New" size="2"><span
style="font-size:10.0pt">Louisburg,NC  27549</span></font></pre><pre style="margin-left:36.0pt"><font color="black"
face="CourierNew" size="2"><span style="font-size:10.0pt"> </span></font></pre><pre style="margin-left: 
36.0pt"><font color="black" face="Courier New" size="2"><span style="font-size:
10.0pt">252-478-8087 (Home)</span></font></pre><pre style="margin-left:36.0pt"><font color="black" face="Courier New"
size="2"><spanstyle="font-size:10.0pt">919-754-4592 (Work)</span></font></pre><pre style="margin-left:36.0pt"><font
color="black"face="Courier New" size="2"><span style="font-size:10.0pt"> </span></font></pre><pre style="margin-left: 
36.0pt"><font color="black" face="Courier New" size="2"><span style="font-size:
10.0pt">Chief Architect JPlate  <a
href="http://sourceforge.net/projects/jplate">http://sourceforge.net/projects/jplate</a></span></font></pre><pre
style="margin-left:36.0pt"><fontcolor="black" face="Courier New" size="2"><span style="font-size:10.0pt">Chief
ArchitectJavaPIM <a
href="http://sourceforge.net/projects/javapim">http://sourceforge.net/projects/javapim</a></span></font></pre></div><br
/><p><b>*******************Confidentialityand Privilege Notice*******************</b><p> The material contained in this
messageis privileged and confidential to the addressee. If you are not the addressee indicated in this message or
responsiblefor delivery of the message to such person, you may not copy or deliver this message to anyone, and you
shoulddestroy it and kindly notify the sender by reply email. <p> Information in this message that does not relate to
theofficial business of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta. Weatherbeeta, its
employees,contractors or associates shall not be liable for direct, indirect or consequential loss arising from
transmissionof this message or any attachments <br />