Re: [PATCHES] Charset WIN1252 - Mailing list pgsql-hackers

OK, good summary.  Shame we missed this for 8.0.  I am sure we will do
better next time..  Wow, you had been working on this since November
2nd.  Shame it didn't make it.  Sorry again.

Also, I was wondering how widely used 1252 encoding is.

Actually, would you send me an email privately that I can put in the 8.1
queue with the most recent versions?  Thanks.

---------------------------------------------------------------------------

Roland Volkmann wrote:
> Hello Bruce,
> 
> thank you for reply. Please let me put my remark here on top this time, 
> because otherwise complete message might get more confusing.
> 
> 1. the mail you quoted below is my reply to Alvaro Herrera. on 30.11.04, 
> which was also in your confirmation on 01.12.04. He missed the new files 
> as patches in my original mail, addressed to you and Peter Eisentraut 
> also on 30.11.04. There had only been the changed files, not the new 
> ones. In the mail to Alvaro Herrera with copy to pqsql-patches I sent a 
> correction to patch of  "src/backend/utils/mb/encnames.c" , and as 
> attachment again the missed *new* files.
> 
> 2. the patches (changes) of already existing files were in the mail, 
> addressed to you and Peter Eisentraut only. Based on this mail you sent 
> the first confirmation on 30.11.04, and there "include/mb/pg_wchar.h" is 
> listed.
> 
> 3. the complete package of modified and new files were in a ZIP attached 
> to a mail on 02.11.04:
> 
> > Hello developers,
> >
> > please find attached a ZIP with sources, headers, and tables for 
> > WIN1252-Support.
> >
> > Even if UTF-8 handling will be completed in the engine, there will 
> > still be some applications which need WIN1252 on client side. I added 
> > the new charset to pg_enc and pg_enc2name[] after WIN1250 and before 
> > the entries which are for client encoding only.
> 
> 
> Because this is the first time I put an input to a development group, I 
> didn't know, that "patches" were awaited. So when Peter Eisentraut wrote 
> on 29.11.04
> 
> >He should send a patch, not a source tarball, so it's possible to see 
> >what actually changed.
> >
> I sent the mail related on point 2 above.
> 
> 
> Okay, I think things are more clear now. After backing out these patches 
> from upcoming RC1, I hope you have all neccessary files available to 
> apply at a later time. Meanwhile I will continue to use my patched DLLs 
> for Win1252 support  ;-)
> 
> 
> With best regards,
> 
> Roland.
> 
> 
> ---------------------------------------------------------------------------
> 
> 
> Bruce Momjian schrieb am 05.12.2004 01:08:
> 
> >Below is the patch I applied.
> >
> >  
> >
> >>I'm the one who originally posted the patch. But the
> >>"include/mb/pg_wchar.h"  was as complete source in the ZIP attached to
> >>my mail on 02.11.04  as well as patch in my mail on 30.11.04 with
> >>correct contents.
> >>
> >>And also in the confirmation of Bruce Momjian from 30.11.04 "Your patch
> >>has been added to the PostgreSQL unapplied patches list ..." there was
> >>the correct file contents listed.
> >>    
> >>
> >
> >You will notice there is no mention of "include/mb/pg_wchar.h" anywhere
> >in the message, and it was not in the zip file, at least in the most
> >recent version I applied.
> >
> >FYI, this patch went through several revisions as people commented on it
> >so it is very possible some files get lost during the revisions.
> >
> >---------------------------------------------------------------------------
> >
> >pgman wrote:
> >  
> >
> >>Your patch has been added to the PostgreSQL unapplied patches list at:
> >>
> >>    http://momjian.postgresql.org/cgi-bin/pgpatches
> >>
> >>It will be applied as soon as one of the PostgreSQL committers reviews
> >>and approves it.
> >>
> >>---------------------------------------------------------------------------
> >>
> >>
> >>Roland Volkmann wrote:
> >>    
> >>
> >>>Hallo Alvaron
> >>>
> >>>
> >>>Alvaro Herrera schrieb am 30.11.2004 17:38:
> >>>
> >>>      
> >>>
> >>>>On Tue, Nov 30, 2004 at 02:05:31AM +0100, Roland Volkmann wrote:
> >>>>
> >>>> 
> >>>>
> >>>>        
> >>>>
> >>>>>this is the first time, I use CVS (TortoisCVS), so I hope the result is 
> >>>>>what you were missing:
> >>>>>   
> >>>>>
> >>>>>          
> >>>>>
> >>>>You have to include the new files separately.  And please send context
> >>>>diffs, not unified diffs.
> >>>> 
> >>>>
> >>>>        
> >>>>
> >>>sorry, but I can't find any option for changing patch style in 
> >>>TortoiseCVS. And the patches of other developers look similar to me.
> >>>
> >>>The new files were attached as ZIP-file to my mail on 02.11.2004. Please 
> >>>find these files again attached to this mail:
> >>>
> >>>Index: 
> >>>src/backend/utils/mb/conversion_procs/utf8_and_win1252/utf8_and_win1252.c
> >>>Index: src/backend/utils/mb/conversion_procs/utf8_and_win1252/Makefile
> >>>Index: src/backend/utils/mb/Unicode/utf8_to_win1252.map
> >>>Index: src/backend/utils/mb/Unicode/win1252_to_utf8.map
> >>>===================================================================
> >>>++ new files
> >>> 
> >>>
> >>>      
> >>>
> >>>>Also there's a typo here:
> >>>>
> >>>> 
> >>>>
> >>>>        
> >>>>
> >>>thank you, you're right. And here is the corrected patch which shoult 
> >>>replace the earlier faulty one:
> >>>
> >>>
> >>>Index: src/backend/utils/mb/encnames.c
> >>>===================================================================
> >>>RCS file: 
> >>>Y:\CVS_Repositories\postgres.org/postgresql-8.0.0beta5/src/backend/utils/mb/encnames.c,v
> >>>retrieving revision 1.1
> >>>diff -u -r1.1 encnames.c
> >>>--- src/backend/utils/mb/encnames.c    30 Nov 2004 23:42:16 -0000    1.1
> >>>+++ src/backend/utils/mb/encnames.c    30 Nov 2004 23:56:27 -0000
> >>>@@ -194,6 +194,9 @@
> >>>         "win1251", PG_WIN1251
> >>>     },                            /* alias for Windows-1251 */
> >>>     {
> >>>+        "win1252", PG_WIN1252
> >>>+    },                            /* alias for Windows-1252 */
> >>>+    {
> >>>         "win1256", PG_WIN1256
> >>>     },                            /* alias for Windows-1256 */
> >>>     {
> >>>@@ -221,6 +224,9 @@
> >>>         "windows1251", PG_WIN1251
> >>>     },                            /* Windows-1251; Microsoft */
> >>>     {
> >>>+        "windows1252", PG_WIN1252
> >>>+    },                            /* Windows-1251; Microsoft */
> >>>+    {
> >>>         "windows1256", PG_WIN1256
> >>>     },                            /* Windows-1256; Microsoft */
> >>>     {
> >>>@@ -342,6 +348,9 @@
> >>>     },
> >>>     {
> >>>         "WIN1250", PG_WIN1250
> >>>+    },
> >>>+    {
> >>>+        "WIN1252", PG_WIN1252
> >>>     },
> >>>     {
> >>>         "SJIS", PG_SJIS
> >>>
> >>>===================================================================
> >>>
> >>>
> >>>With best regards,
> >>>
> >>>Roland
> >>>
> >>>      
> >>>
> >>>#-------------------------------------------------------------------------
> >>>#
> >>># $PostgreSQL: pgsql/src/backend/utils/mb/conversion_procs/utf8_and_win1252/Makefile,v 1.3 2004/10/31 22:40:41
pgsqlExp $
 
> >>>#
> >>>#-------------------------------------------------------------------------
> >>>subdir = src/backend/utils/mb/conversion_procs/utf8_and_win1252
> >>>top_builddir = ../../../../../..
> >>>include $(top_builddir)/src/Makefile.global
> >>>
> >>>NAME        := utf8_and_win1252
> >>>
> >>>include $(srcdir)/../proc.mk
> >>>      
> >>>
> >>>/*-------------------------------------------------------------------------
> >>> *
> >>> *      WIN1252 and UTF-8
> >>> *
> >>> * Portions Copyright (c) 1996-2004, PostgreSQL Global Development Group
> >>> * Portions Copyright (c) 1994, Regents of the University of California
> >>> *
> >>> * IDENTIFICATION
> >>> *      $PostgreSQL: pgsql/src/backend/utils/mb/conversion_procs/utf8_and_win1252/utf8_and_win1252.c,v 1.8
2004/10/3104:12:59 momjian Exp $
 
> >>> *
> >>> *-------------------------------------------------------------------------
> >>> */
> >>>
> >>>#include "postgres.h"
> >>>#include "fmgr.h"
> >>>#include "mb/pg_wchar.h"
> >>>#include "../../Unicode/utf8_to_win1252.map"
> >>>#include "../../Unicode/win1252_to_utf8.map"
> >>>
> >>>PG_FUNCTION_INFO_V1(utf_to_win1252);
> >>>PG_FUNCTION_INFO_V1(win1252_to_utf);
> >>>
> >>>extern Datum utf_to_win1252(PG_FUNCTION_ARGS);
> >>>extern Datum win1252_to_utf(PG_FUNCTION_ARGS);
> >>>
> >>>/* ----------
> >>> * conv_proc(
> >>> *        INTEGER,    -- source encoding id
> >>> *        INTEGER,    -- destination encoding id
> >>> *        CSTRING,    -- source string (null terminated C string)
> >>> *        CSTRING,    -- destination string (null terminated C string)
> >>> *        INTEGER        -- source string length
> >>> * ) returns VOID;
> >>> * ----------
> >>> */
> >>>
> >>>Datum
> >>>utf_to_win1252(PG_FUNCTION_ARGS)
> >>>{
> >>>    unsigned char *src = PG_GETARG_CSTRING(2);
> >>>    unsigned char *dest = PG_GETARG_CSTRING(3);
> >>>    int            len = PG_GETARG_INT32(4);
> >>>
> >>>    Assert(PG_GETARG_INT32(0) == PG_UTF8);
> >>>    Assert(PG_GETARG_INT32(1) == PG_WIN1252);
> >>>    Assert(len >= 0);
> >>>
> >>>    UtfToLocal(src, dest, ULmapWIN1252,
> >>>               sizeof(ULmapWIN1252) / sizeof(pg_utf_to_local), len);
> >>>
> >>>    PG_RETURN_VOID();
> >>>}
> >>>
> >>>Datum
> >>>win1252_to_utf(PG_FUNCTION_ARGS)
> >>>{
> >>>    unsigned char *src = PG_GETARG_CSTRING(2);
> >>>    unsigned char *dest = PG_GETARG_CSTRING(3);
> >>>    int            len = PG_GETARG_INT32(4);
> >>>
> >>>    Assert(PG_GETARG_INT32(0) == PG_WIN1252);
> >>>    Assert(PG_GETARG_INT32(1) == PG_UTF8);
> >>>    Assert(len >= 0);
> >>>
> >>>    LocalToUtf(src, dest, LUmapWIN1252,
> >>>        sizeof(LUmapWIN1252) / sizeof(pg_local_to_utf), PG_WIN1252, len);
> >>>
> >>>    PG_RETURN_VOID();
> >>>}
> >>>      
> >>>
> >>>static pg_utf_to_local ULmapWIN1252[ 124 ] = {
> >>>  {0x0000, 0x0081},
> >>>  {0xc2a0, 0x00a0},
> >>>  {0xc2a1, 0x00a1},
> >>>  {0xc2a2, 0x00a2},
> >>>  {0xc2a3, 0x00a3},
> >>>  {0xc2a4, 0x00a4},
> >>>  {0xc2a5, 0x00a5},
> >>>  {0xc2a6, 0x00a6},
> >>>  {0xc2a7, 0x00a7},
> >>>  {0xc2a8, 0x00a8},
> >>>  {0xc2a9, 0x00a9},
> >>>  {0xc2aa, 0x00aa},
> >>>  {0xc2ab, 0x00ab},
> >>>  {0xc2ac, 0x00ac},
> >>>  {0xc2ad, 0x00ad},
> >>>  {0xc2ae, 0x00ae},
> >>>  {0xc2af, 0x00af},
> >>>  {0xc2b0, 0x00b0},
> >>>  {0xc2b1, 0x00b1},
> >>>  {0xc2b2, 0x00b2},
> >>>  {0xc2b3, 0x00b3},
> >>>  {0xc2b4, 0x00b4},
> >>>  {0xc2b5, 0x00b5},
> >>>  {0xc2b6, 0x00b6},
> >>>  {0xc2b7, 0x00b7},
> >>>  {0xc2b8, 0x00b8},
> >>>  {0xc2b9, 0x00b9},
> >>>  {0xc2ba, 0x00ba},
> >>>  {0xc2bb, 0x00bb},
> >>>  {0xc2bc, 0x00bc},
> >>>  {0xc2bd, 0x00bd},
> >>>  {0xc2be, 0x00be},
> >>>  {0xc2bf, 0x00bf},
> >>>  {0xc380, 0x00c0},
> >>>  {0xc381, 0x00c1},
> >>>  {0xc382, 0x00c2},
> >>>  {0xc383, 0x00c3},
> >>>  {0xc384, 0x00c4},
> >>>  {0xc385, 0x00c5},
> >>>  {0xc386, 0x00c6},
> >>>  {0xc387, 0x00c7},
> >>>  {0xc388, 0x00c8},
> >>>  {0xc389, 0x00c9},
> >>>  {0xc38a, 0x00ca},
> >>>  {0xc38b, 0x00cb},
> >>>  {0xc38c, 0x00cc},
> >>>  {0xc38d, 0x00cd},
> >>>  {0xc38e, 0x00ce},
> >>>  {0xc38f, 0x00cf},
> >>>  {0xc390, 0x00d0},
> >>>  {0xc391, 0x00d1},
> >>>  {0xc392, 0x00d2},
> >>>  {0xc393, 0x00d3},
> >>>  {0xc394, 0x00d4},
> >>>  {0xc395, 0x00d5},
> >>>  {0xc396, 0x00d6},
> >>>  {0xc397, 0x00d7},
> >>>  {0xc398, 0x00d8},
> >>>  {0xc399, 0x00d9},
> >>>  {0xc39a, 0x00da},
> >>>  {0xc39b, 0x00db},
> >>>  {0xc39c, 0x00dc},
> >>>  {0xc39d, 0x00dd},
> >>>  {0xc39e, 0x00de},
> >>>  {0xc39f, 0x00df},
> >>>  {0xc3a0, 0x00e0},
> >>>  {0xc3a1, 0x00e1},
> >>>  {0xc3a2, 0x00e2},
> >>>  {0xc3a3, 0x00e3},
> >>>  {0xc3a4, 0x00e4},
> >>>  {0xc3a5, 0x00e5},
> >>>  {0xc3a6, 0x00e6},
> >>>  {0xc3a7, 0x00e7},
> >>>  {0xc3a8, 0x00e8},
> >>>  {0xc3a9, 0x00e9},
> >>>  {0xc3aa, 0x00ea},
> >>>  {0xc3ab, 0x00eb},
> >>>  {0xc3ac, 0x00ec},
> >>>  {0xc3ad, 0x00ed},
> >>>  {0xc3ae, 0x00ee},
> >>>  {0xc3af, 0x00ef},
> >>>  {0xc3b0, 0x00f0},
> >>>  {0xc3b1, 0x00f1},
> >>>  {0xc3b2, 0x00f2},
> >>>  {0xc3b3, 0x00f3},
> >>>  {0xc3b4, 0x00f4},
> >>>  {0xc3b5, 0x00f5},
> >>>  {0xc3b6, 0x00f6},
> >>>  {0xc3b7, 0x00f7},
> >>>  {0xc3b8, 0x00f8},
> >>>  {0xc3b9, 0x00f9},
> >>>  {0xc3ba, 0x00fa},
> >>>  {0xc3bb, 0x00fb},
> >>>  {0xc3bc, 0x00fc},
> >>>  {0xc3bd, 0x00fd},
> >>>  {0xc3be, 0x00fe},
> >>>  {0xc3bf, 0x00ff},
> >>>  {0xc592, 0x008c},
> >>>  {0xc593, 0x009c},
> >>>  {0xc5a0, 0x008a},
> >>>  {0xc5a1, 0x009a},
> >>>  {0xc5b8, 0x009f},
> >>>  {0xc5bd, 0x008e},
> >>>  {0xc5be, 0x009e},
> >>>  {0xc692, 0x0083},
> >>>  {0xcb86, 0x0088},
> >>>  {0xcb9c, 0x0098},
> >>>  {0xe28093, 0x0096},
> >>>  {0xe28094, 0x0097},
> >>>  {0xe28098, 0x0091},
> >>>  {0xe28099, 0x0092},
> >>>  {0xe2809a, 0x0082},
> >>>  {0xe2809c, 0x0093},
> >>>  {0xe2809d, 0x0094},
> >>>  {0xe2809e, 0x0084},
> >>>  {0xe280a0, 0x0086},
> >>>  {0xe280a1, 0x0087},
> >>>  {0xe280a2, 0x0095},
> >>>  {0xe280a6, 0x0085},
> >>>  {0xe280b0, 0x0089},
> >>>  {0xe280b9, 0x008b},
> >>>  {0xe280ba, 0x009b},
> >>>  {0xe282ac, 0x0080},
> >>>  {0xe284a2, 0x0099}
> >>>};
> >>>      
> >>>
> >>>static pg_local_to_utf LUmapWIN1250[ 128 ] = {
> >>>  {0x0080, 0xe282ac},
> >>>  {0x0081, 0x0000},
> >>>  {0x0082, 0xe2809a},
> >>>  {0x0083, 0xc692},
> >>>  {0x0084, 0xe2809e},
> >>>  {0x0085, 0xe280a6},
> >>>  {0x0086, 0xe280a0},
> >>>  {0x0087, 0xe280a1},
> >>>  {0x0088, 0xcb86},
> >>>  {0x0089, 0xe280b0},
> >>>  {0x008a, 0xc5a0},
> >>>  {0x008b, 0xe280b9},
> >>>  {0x008c, 0xc592},
> >>>  {0x008d, 0x0000},
> >>>  {0x008e, 0xc5bd},
> >>>  {0x008f, 0x0000},
> >>>  {0x0090, 0x0000},
> >>>  {0x0091, 0xe28098},
> >>>  {0x0092, 0xe28099},
> >>>  {0x0093, 0xe2809c},
> >>>  {0x0094, 0xe2809d},
> >>>  {0x0095, 0xe280a2},
> >>>  {0x0096, 0xe28093},
> >>>  {0x0097, 0xe28094},
> >>>  {0x0098, 0xcb9c},
> >>>  {0x0099, 0xe284a2},
> >>>  {0x009a, 0xc5a1},
> >>>  {0x009b, 0xe280ba},
> >>>  {0x009c, 0xc593},
> >>>  {0x009d, 0x0000},
> >>>  {0x009e, 0xc5be},
> >>>  {0x009f, 0xc5b8},
> >>>  {0x00a0, 0xc2a0},
> >>>  {0x00a1, 0xc2a1},
> >>>  {0x00a2, 0xc2a2},
> >>>  {0x00a3, 0xc2a3},
> >>>  {0x00a4, 0xc2a4},
> >>>  {0x00a5, 0xc2a5},
> >>>  {0x00a6, 0xc2a6},
> >>>  {0x00a7, 0xc2a7},
> >>>  {0x00a8, 0xc2a8},
> >>>  {0x00a9, 0xc2a9},
> >>>  {0x00aa, 0xc2aa},
> >>>  {0x00ab, 0xc2ab},
> >>>  {0x00ac, 0xc2ac},
> >>>  {0x00ad, 0xc2ad},
> >>>  {0x00ae, 0xc2ae},
> >>>  {0x00af, 0xc2af},
> >>>  {0x00b0, 0xc2b0},
> >>>  {0x00b1, 0xc2b1},
> >>>  {0x00b2, 0xc2b2},
> >>>  {0x00b3, 0xc2b3},
> >>>  {0x00b4, 0xc2b4},
> >>>  {0x00b5, 0xc2b5},
> >>>  {0x00b6, 0xc2b6},
> >>>  {0x00b7, 0xc2b7},
> >>>  {0x00b8, 0xc2b8},
> >>>  {0x00b9, 0xc2b9},
> >>>  {0x00ba, 0xc2ba},
> >>>  {0x00bb, 0xc2bb},
> >>>  {0x00bc, 0xc2bc},
> >>>  {0x00bd, 0xc2bd},
> >>>  {0x00be, 0xc2be},
> >>>  {0x00bf, 0xc2bf},
> >>>  {0x00c0, 0xc380},
> >>>  {0x00c1, 0xc381},
> >>>  {0x00c2, 0xc382},
> >>>  {0x00c3, 0xc383},
> >>>  {0x00c4, 0xc384},
> >>>  {0x00c5, 0xc385},
> >>>  {0x00c6, 0xc386},
> >>>  {0x00c7, 0xc387},
> >>>  {0x00c8, 0xc388},
> >>>  {0x00c9, 0xc389},
> >>>  {0x00ca, 0xc38a},
> >>>  {0x00cb, 0xc38b},
> >>>  {0x00cc, 0xc38c},
> >>>  {0x00cd, 0xc38d},
> >>>  {0x00ce, 0xc38e},
> >>>  {0x00cf, 0xc38f},
> >>>  {0x00d0, 0xc390},
> >>>  {0x00d1, 0xc391},
> >>>  {0x00d2, 0xc392},
> >>>  {0x00d3, 0xc393},
> >>>  {0x00d4, 0xc394},
> >>>  {0x00d5, 0xc395},
> >>>  {0x00d6, 0xc396},
> >>>  {0x00d7, 0xc397},
> >>>  {0x00d8, 0xc398},
> >>>  {0x00d9, 0xc399},
> >>>  {0x00da, 0xc39a},
> >>>  {0x00db, 0xc39b},
> >>>  {0x00dc, 0xc39c},
> >>>  {0x00dd, 0xc39d},
> >>>  {0x00de, 0xc39e},
> >>>  {0x00df, 0xc39f},
> >>>  {0x00e0, 0xc3a0},
> >>>  {0x00e1, 0xc3a1},
> >>>  {0x00e2, 0xc3a2},
> >>>  {0x00e3, 0xc3a3},
> >>>  {0x00e4, 0xc3a4},
> >>>  {0x00e5, 0xc3a5},
> >>>  {0x00e6, 0xc3a6},
> >>>  {0x00e7, 0xc3a7},
> >>>  {0x00e8, 0xc3a8},
> >>>  {0x00e9, 0xc3a9},
> >>>  {0x00ea, 0xc3aa},
> >>>  {0x00eb, 0xc3ab},
> >>>  {0x00ec, 0xc3ac},
> >>>  {0x00ed, 0xc3ad},
> >>>  {0x00ee, 0xc3ae},
> >>>  {0x00ef, 0xc3af},
> >>>  {0x00f0, 0xc3b0},
> >>>  {0x00f1, 0xc3b1},
> >>>  {0x00f2, 0xc3b2},
> >>>  {0x00f3, 0xc3b3},
> >>>  {0x00f4, 0xc3b4},
> >>>  {0x00f5, 0xc3b5},
> >>>  {0x00f6, 0xc3b6},
> >>>  {0x00f7, 0xc3b7},
> >>>  {0x00f8, 0xc3b8},
> >>>  {0x00f9, 0xc3b9},
> >>>  {0x00fa, 0xc3ba},
> >>>  {0x00fb, 0xc3bb},
> >>>  {0x00fc, 0xc3bc},
> >>>  {0x00fd, 0xc3bd},
> >>>  {0x00fe, 0xc3be},
> >>>  {0x00ff, 0xc3bf}
> >>>};
> >>>      
> >>>
> >>>---------------------------(end of broadcast)---------------------------
> >>>TIP 3: if posting/reading through Usenet, please send an appropriate
> >>>      subscribe-nomail command to majordomo@postgresql.org so that your
> >>>      message can get through to the mailing list cleanly
> >>>      
> >>>
> >>-- 
> >>  Bruce Momjian                        |  http://candle.pha.pa.us
> >>  pgman@candle.pha.pa.us               |  (610) 359-1001
> >>  +  If your life is a hard drive,     |  13 Roberts Road
> >>  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073
> >>    
> >>
> >
> >  
> >
> 
> 

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073