[GENERAL] Postgres, apps, special characters and UTF-8 encoding - Mailing list pgsql-general

From Ken Tanzer
Subject [GENERAL] Postgres, apps, special characters and UTF-8 encoding
Date
Msg-id CAD3a31VMvCjPOSgHyEobWr6YoOo3RY=9Oq7hAn2eqBy8pfHLtg@mail.gmail.com
Whole thread Raw
Responses Re: [GENERAL] Postgres, apps, special characters and UTF-8 encoding  ("David G. Johnston" <david.g.johnston@gmail.com>)
Re: [GENERAL] Postgres, apps, special characters and UTF-8 encoding  (Adrian Klaver <adrian.klaver@aklaver.com>)
Re: [GENERAL] Postgres, apps, special characters and UTF-8 encoding  (rob stone <floriparob@gmail.com>)
Re: [GENERAL] Postgres, apps, special characters and UTF-8 encoding  (vinny <vinny@xs4all.nl>)
Re: [GENERAL] Postgres, apps, special characters and UTF-8 encoding  (Albe Laurenz <laurenz.albe@wien.gv.at>)
List pgsql-general
Hi.  I've got a recurring problem with character encoding for a Postgres-based web PHP app, and am hoping someone can clue me in or at least point me in the right direction.  I'll confess upfront my understanding of encoding issues is extremely limited.  Here goes.

The app uses a Postgres database, UTF-8 encoded.  Through their browsers, users can add and edit records often including text.  Most of the time this works fine.  Though sometimes this will fail with Postgres complaining, for example, "Could query with ... , The error text was: ERROR: invalid byte sequence for encoding "UTF8": 0xe9 0x20 0x67"

So this generally happens when people copy and paste things out of their word documents and such.

As I understand it, those are likely encoded in something non-UTF-8, like WIN-1251 or something.  And that one way or another, the encoding needs to be translated before it can be placed into the database.  I'm not clear how this is supposed to happen though.  Automatically by the browser?  Done in the app?  Some other way?  And if in the app, how is one supposed to know what the incoming encoding is?

Thanks in advance for any help or pointers.

Ken


--
AGENCY Software  
A Free Software data system
By and for non-profits
(253) 245-3801

learn more about AGENCY or
follow the discussion.

pgsql-general by date:

Previous
From: Adrian Klaver
Date:
Subject: Re: [GENERAL] PGSQL 9.6.2 unable to find readline
Next
From: "David G. Johnston"
Date:
Subject: Re: [GENERAL] Postgres, apps, special characters and UTF-8 encoding