COPY to table with array columns (Longish) - Mailing list pgsql-sql

<div class="Section1"><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial">Hi All,</span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial"> </span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial">Hope someone can help me – our main company system runs on Raining Data PICK/D3 (if anyone familiar
withit) which stores records in it’s “tables” as variable length items. Every item has a unique Primary Key (per table)
theneach item can have a variable number of fields. These fields are delimited by Char 254, then each field can have
sub-valuesdelimited by Char 253, then sub-sub-values delimited by Char 252.</span></font><p class="MsoNormal"><font
face="Arial"size="2"><span style="font-size:10.0pt; 
font-family:Arial"> </span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial">Anyway, we are trying to export everything to Postgres for reporting and querying etc (not to
actuallyrun the system…. Yet) and hasn’t been a problem so far – everything like stock and purchase orders, sales
ordersetc can pretty easily be turned in to a flat file with standard number of columns and consistent data. We
truncateevery table each night then import that latest TSV export from D3 using a COPY command.</span></font><p
class="MsoNormal"><fontface="Arial" size="2"><span style="font-size:10.0pt; 
font-family:Arial"> </span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial">The problem arises with tables like our SYS table which store generic system data, so one record
couldhave 3 fields, but the next could have 300. The only way I can work out how to export multi-valued data like this
toPostgres is to use an array column. So the table has 2 columns – the pkey and a data array.</span></font><p
class="MsoNormal"><fontface="Arial" size="2"><span style="font-size:10.0pt; 
font-family:Arial"> </span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial">How do I get this imported to the truncated table each night? At the moment I think my best option
isto modify the export for the SYS table to call PSQL and use standard SQL INSERT statements to directly insert it
insteadof exporting to a flat file, then import to Postgres.</span></font><p class="MsoNormal"><font face="Arial"
size="2"><spanstyle="font-size:10.0pt; 
font-family:Arial"> </span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial">Thanks all,</span></font><p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size:10.0pt;
font-family:Arial">-p</span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial"> </span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial">For those who are interested, or if it might help, here’s a rough comparison of the database
structureof D3:</span></font><p class="MsoNormal"><b><font face="Courier New" size="2"><span style="font-size: 
10.0pt;font-family:"Courier New";font-weight:bold">Windows           =     PICK/D3     =    
Postgres</span></font></b><pclass="MsoNormal"><font face="Courier New" size="2"><span style="font-size:10.0pt; 
font-family:"Courier New"">Drive             =     Account     =     Database</span></font><p class="MsoNormal"><font
face="CourierNew" size="2"><span style="font-size:10.0pt; 
font-family:"Courier New"">Directory         =     File        =     Table</span></font><p class="MsoNormal"><font
face="CourierNew" size="2"><span style="font-size:10.0pt; 
font-family:"Courier New"">File              =     Item        =     Row</span></font><p class="MsoNormal"><font
face="CourierNew" size="2"><span style="font-size:10.0pt; 
font-family:"Courier New"">Line in text file =     Attribute   =     Field</span></font><p class="MsoNormal"><font
face="CourierNew" size="2"><span style="font-size:10.0pt; 
font-family:"Courier New"">(none)            =     Value       =     Array Element (?)</span></font><p
class="MsoNormal"><fontface="Courier New" size="2"><span style="font-size:10.0pt; 
font-family:"Courier New"">(none)            =     Sub Value   =     (none?)</span></font><p class="MsoNormal"><font
face="Arial"size="2"><span style="font-size:10.0pt; 
font-family:Arial"> </span></font><p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial">PhillipSmith</span></font><p class="MsoNormal"><font face="Arial"
size="2"><spanstyle="font-size:10.0pt; 
font-family:Arial">IT Coordinator</span></font><p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size:10.0pt;
font-family:Arial">Weatherbeeta P/L</span></font><p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial">8 Moncrief Rd</span></font><p class="MsoNormal"><font face="Arial"
size="2"><spanstyle="font-size:10.0pt; 
font-family:Arial">Nunawading, VIC, 3131</span></font><p class="MsoNormal"><font face="Arial" size="2"><span
style="font-size:10.0pt;font-family:Arial">AUSTRALIA</span></font><p class="MsoNormal"><font face="Times New Roman"
size="3"><spanstyle="font-size: 
12.0pt"> </span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;
font-family:Arial">E. phillips@NO-SPAM.weatherbeeta.com.au</span></font><p class="MsoNormal"><font face="Times New
Roman"size="3"><span style="font-size: 
12.0pt"> </span></font></div><br /><p><b>*******************Confidentiality and Privilege
Notice*******************</b><p>The material contained in this message is privileged and confidential to the addressee.
Ifyou are not the addressee indicated in this message or responsible for delivery of the message to such person, you
maynot copy or deliver this message to anyone, and you should destroy it and kindly notify the sender by reply email.
<p>Information in this message that does not relate to the official business of Weatherbeeta must be treated as neither
givennor endorsed by Weatherbeeta. Weatherbeeta, its employees, contractors or associates shall not be liable for
direct,indirect or consequential loss arising from transmission of this message or any attachments <br />