Re: UTF8 encoding and non-text data types - Mailing list pgsql-sql

<br /><blockquote cite="" class="cite" type="cite">On Jan 13, 2008 8:51 PM, Steve Midgley <<a
href="mailto:public@misuse.org">public@misuse.org</a>>wrote:<br /><dl><dd>At 02:22 PM 1/13/2008, <a
href="mailto:pgsql-sql-owner@postgresql.org">pgsql-sql-owner@postgresql.org</a> wrote:<br /><dd>>Date: Sat, 12 Jan
200814:21:00 -0800<br /><dd>>From: "Medi Montaseri" <<a href="mailto:montaseri@gmail.com">
montaseri@gmail.com</a>><br/><dd>>To: <a href="mailto:pgsql-sql@postgresql.org">pgsql-sql@postgresql.org</a><br
/><dd>>Subject:UTF8 encoding and non-text data types<br /><dd>>Message-ID:<br /><dd> ><<a
href="mailto:8078a1730801121421l1c9b90c1lc65d36cf6e752a6f@mail.gmail.com">
8078a1730801121421l1c9b90c1lc65d36cf6e752a6f@mail.gmail.com</a>><br/><dd>><br /><dd>>I understand PG supports
UTF-8encoding and I have sucessfully<br /><dd>>inserted<br /><dd>>Unicode text into columns. I was wondering
aboutother data types such <br /><dd>>as<br /><dd>>numbers, decimal, dates<br /><dd>><br /><dd>>That is,
sayI have a table t1 with<br /><dd>>create table t1 { name text, cost decimal }<br /><dd>>I can insert UTF8 text
datatypeinto this table with no problem<br /><dd>>But if my application attempts to insert numbers encloded in UTF8,
<br/><dd>>then I<br /><dd>>get wrong datatype error<br /><dd>><br /><dd>>Is the solution for the
applicationlayer (not database) to convert<br /><dd>>the<br /><dd>>non-text UTF8 numbers to ASCII and then insert
itinto database ?<br /><dd>> <br /><dd>>Thanks<br /><dd>>Medi<br /><br /><dd>Hi Medi,<br /><br /><dd>I have
onlylimited experience in this area, but it sounds like you<br /><dd>sending your numbers as strings? In your
example:<br/><br /><dd>>create table t1 { name text, cost decimal }; <br /><br /><dd>insert into t1 (name, cost)
values('name1', '1');<br /><br /><dd>I can't think of how else you're sending numeric values as UTF8? I know<br
/><dd>thatPg will accept numbers as strings and convert internally (that has <br /><dd>worked for me in some object
relationalenvironments where I don't<br /><dd>choose to cope with data types), but I think it would be better if you<br
/><dd>simplydidn't send your numeric data in quotations, whether as UTF8 or <br /><dd>ASCII. If you don't have control
overthis layer (that quotes your<br /><dd>values), then I'd say converting to ASCII would solve the problem. But<br
/><dd>betterto convert to numeric and not ship quoted strings at all.<br /><br /><dd>I may be totally off-base and
missingsomething fundamental and I'm<br /><dd>very open to correction (by anyone), but that's what I can see here.<br
/><br/><dd>Best regards,<br /><font color="#888888"><br /></font><dd>Steve<br /></dl>At 11:01 AM 1/14/2008, Medi
Montaseriwrote:<br /> Thanks Steve,<br /><br /> Actually I do not insert text data into my numeric field.<br /> As I
mentionedgiven <br /> create table t1 { name text, cost decimal }<br /> then I would like to insert numeric data into
column"cost" because then I can later benefit from numerical operators like SUM, AVG, etc <br /><br /> More
specifically,I am using HTML, Perl and PG. So from the HTML point of view a textfield is just some strings. So my user
wouldenter 12345 but expressed in UTF8. Perl would get this and use DBI to insert it into PG <br /><br /> What I am
experiencingnow is that DB errors that I am trying to insert an incorrect data into column "cost" which is numeric and
thedata is coming in from HTML in UTF8<br /><br /> Mybe I have to convert it to ASCII numbers in Perl before inserting 
theminto PG <br /><br /> Thanks<br /> Medi<br /></blockquote><br /> Hi Medi,<br /><br /> I agree that you should
convertyour values in Perl before handing to DBI. I'm not familiar with DBI but presumably if you're sending it UTF8
valuesit's attempting to quote them or do something with them, that a numeric field in Pg can't handle. Can you
trap/monitorthe exact sql statement that is generated by DBI and sent to Pg? That would help a lot in knowing what it
isdoing, but I suspect if you just convert your numbers from the HTML/UTF8 source values into actual Perl numeric
valuesand then ship to DBI you'll be better off. And you'll get some input validation for free.<br /><br /> I hope this
helps,<br/><br /> Steve<br />