Thread: Converting from LATIN1 to UNICODE encoding?
I have a database that was created using the LATIN1 encoding. Is it necessary for me to convert the database to some other encoding (e.g. UNICODE) before I can store non-LATIN1 characters or does PostgreSQL catch these and encode them somehow? I have actually been attempting to convert a database by doing a pg_dump (from the LATIN1 database) followed by a pg_restore (to one created with the UNICODE encoding). Seemed to work with a sparsely populated 8.0.3 database, but I am running into all sorts of problems with 7.3.10 (e.g. corrupted database followed by corrupted pg_clog). What about JDBC, how does it know the encoding of the data I throw at it? I guess I should ask this last question on the JDBC list. Thanks in advance for any comments. Scott
On Wed, Sep 21, 2005 at 06:16:31PM +1000, Scott Eade wrote: > I have a database that was created using the LATIN1 encoding. > > Is it necessary for me to convert the database to some other encoding > (e.g. UNICODE) before I can store non-LATIN1 characters or does > PostgreSQL catch these and encode them somehow? > > I have actually been attempting to convert a database by doing a pg_dump > (from the LATIN1 database) followed by a pg_restore (to one created with > the UNICODE encoding). Seemed to work with a sparsely populated 8.0.3 > database, but I am running into all sorts of problems with 7.3.10 (e.g. > corrupted database followed by corrupted pg_clog). Hmm. Do tell, what database did you use as template? Was it correctly frozen? If not, has it suffered from Xid wraparound? Try dumping the template database. If it succeeds, then it's OK. If it complains in the same way, with a missing pg_clog file, there lies the problem. -- Alvaro Herrera http://www.planetpostgresql.org "En las profundidades de nuestro inconsciente hay una obsesiva necesidad de un universo lógico y coherente. Pero el universo real se halla siempre un paso más allá de la lógica" (Irulan)
Am Mittwoch, 21. September 2005 10:16 schrieb Scott Eade: > Is it necessary for me to convert the database to some other encoding > (e.g. UNICODE) before I can store non-LATIN1 characters or does > PostgreSQL catch these and encode them somehow? Yes and no. > I have actually been attempting to convert a database by doing a pg_dump > (from the LATIN1 database) followed by a pg_restore (to one created with > the UNICODE encoding). That is the right method. > Seemed to work with a sparsely populated 8.0.3 > database, but I am running into all sorts of problems with 7.3.10 (e.g. > corrupted database followed by corrupted pg_clog). That is the symptom of a different, much bigger problem. Encoding problems certainly never corrupt the clog. > What about JDBC, how does it know the encoding of the data I throw at > it? That depends on where your throws come from. As you surely know, Java mostly uses Unicode internally, so the data sent by the JDBC driver to the database is in Unicode, but as to what encoding the data that you input into your Java program has, you need to sort that out with the Java library functions that you use to read that data.