SOLVED: Re: UTF-8 and stripping accents - Mailing list pgsql-general

From Christopher Murtagh
Subject SOLVED: Re: UTF-8 and stripping accents
Date
Msg-id 92fbb7920606151222w44b9c604pe6853d7b06e03b66@mail.gmail.com
Whole thread Raw
Responses Re: SOLVED: Re: UTF-8 and stripping accents
List pgsql-general
Hey, I solved my own problem! I'm posting here because while I was
looking for solutions, I found tons of folks tackling the same
problem, most didn't find the solution or had to do cumbersome
'translate()'s to get what they wanted.

The difference between my 7.4.6 and 8.1.4 DBs was that 7.4.6 had
UNICODE as it's encoding, whereas the 8.1.4 was UTF8. So, the 7.4.6
needs the decode and the 8.1.4 doesn't.

Also, I had to escape the '\' in the regex.

So, for the record, to strip out all accents from UTF8 encoded text:

CREATE OR REPLACE FUNCTION strip_accents(text) RETURNS text
AS '
 use Unicode::Normalize;
 use Encode;

 my $string = NFD($_[0]);
 $string =~ s/\\p{Mn}//ogsm;
 return NFC($string);
'
LANGUAGE plperlu;

For the 7.4.6 DB whose encoding was UNICODE, a slight difference:

CREATE OR REPLACE FUNCTION strip_accents(text) RETURNS text
AS '
 use Unicode::Normalize;
 use Encode;

 my $string = NFD(decode( utf8 => $_[0]));
 $string =~ s/\\p{Mn}//ogsm;
 return NFC($string);
'
LANGUAGE plperlu;

I hope this is of some use to other folks here. Thanks to Mike
Rylander for the initial code.

Cheers,

Chris

pgsql-general by date:

Previous
From: "Shoaib Mir"
Date:
Subject: Re: postgres password
Next
From: "Milen Kulev"
Date:
Subject: Partitioning and sub-partitioning problems