Re: BUG #13440: unaccent does not remove all diacritics - Mailing list pgsql-bugs

From Thomas Munro
Subject Re: BUG #13440: unaccent does not remove all diacritics
Date
Msg-id CAEepm=2XAMTA8r3V682_aNZOz1kB3MdMvymDQmO7TA0qg99GAA@mail.gmail.com
Whole thread Raw
In response to Re: BUG #13440: unaccent does not remove all diacritics  (Thomas Munro <thomas.munro@enterprisedb.com>)
Responses Re: BUG #13440: unaccent does not remove all diacritics  (Tom Lane <tgl@sss.pgh.pa.us>)
List pgsql-bugs
On Tue, Jun 16, 2015 at 8:07 AM, Thomas Munro
<thomas.munro@enterprisedb.com> wrote:
> On Tue, Jun 16, 2015 at 12:55 AM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> Alvaro Herrera <alvherre@2ndquadrant.com> writes:
>>> My terminal shows these characters to be different.  One is
>>> http://graphemica.com/%C8%9B
>>>       latin small letter t with comma below (U+021B)
>>
>>> The other is
>>> http://graphemica.com/%C5%A3
>>>       latin small letter t with cedilla (U+0163)
>>
>> Ah-hah -- I did not look closely enough.  So the immediate answer for
>> Michael is to add another entry to his unaccent.rules file.
>>
>> Should we add the missing character to the standard unaccent.rules file?
>
> It looks like Romanian also has s with comma.  Perhaps we should have
> all these characters:
>
> $ curl -s http://unicode.org/Public/7.0.0/ucd/UnicodeData.txt | egrep
> ';LATIN (SMALL|CAPITAL) LETTER [A-Z] WITH ' | wc -l
>      702

Here is an unaccent.rules file that maps those 702 characters from
Unicode 7.0 with names like "LATIN (SMALL|CAPITAL) LETTER [A-Z] WITH
..." to their base letter, plus 14 extra cases to match the existing
unaccent.rules file.  If you sort and diff this and the existing file,
you can see that this file only adds new lines.  Also, here is the
script I used to build it from UnicodeData.txt.

--
Thomas Munro
http://www.enterprisedb.com

Attachment

pgsql-bugs by date:

Previous
From: Michael Paquier
Date:
Subject: Re: BUG #13443: master will remove dead rows when hot standby(use slot) disconnect
Next
From: Tom Lane
Date:
Subject: Re: BUG #13442: ISBN doesn't always roundtrip with text