Thread: Unicode Variation Selector and Combining character

Unicode Variation Selector and Combining character

From

荒井元成

Date:

30 May 2022, 03:27:08

Hi,

I tried it on PostgreSQL 13. If you use the Unicode Variation Selector and Combining Character

, the base character and the Variation selector will be 2 in length. Since it will be one character on the display, we expect it to be one in length. Please provide a function corresponding to the unicode variasion selector. I hope It is supposed to be provided as an extension.

The functions that need to be supported are as follows:

|regexp_replace|regexp_split_to_array|regexp_split_to_table

Best regartds,

Re: Unicode Variation Selector and Combining character

From

Peter Eisentraut

Date:

01 June 2022, 08:27:00

On 30.05.22 02:27, 荒井元成 wrote:
> I tried it on PostgreSQL 13. If you use the Unicode Variation Selector 
> and Combining Character
> 
> , the base character and the Variation selector will be 2 in length. 
> Since it will be one character on the display, we expect it to be one in 
> length. Please provide a function corresponding to the unicode variasion 
> selector. I hope It is supposed to be provided as an extension.
> 
> The functions that need to be supported are as follows:
> 
> char_length|character_length|substring|trim|btrim|left
> 
> |length|lpad|ltrim|regexp_match|regexp_matches
> 
> |regexp_replace|regexp_split_to_array|regexp_split_to_table
> 
> |replace|reverse|right|rpad|rtrim|split_part|strpos|substr|starts_with

Please show a test case of what you mean.  For example,

select char_length(...) returns X but should return Y

Examples with Unicode escapes (U&'\NNNN...') would be the most robust.

RE: Unicode Variation Selector and Combining character

From

荒井元成

Date:

01 June 2022, 09:15:15

Thank you for your reply.

We use IPAmj Mincho Font in the specifications of the Government of Japan.

https://moji.or.jp/mojikiban/font/

Exsample)IVS

I will attach an image.

D209007=# select char_length(U&'\+0066FE' || U&'\+0E0103') ;

char_length

-------------

(1 行)

I expect length 1.

Exsample)Combining Character

I will attach an image.

D209007=# select char_length(U&'\+00304B' || U&'\+00309A') ;

char_length

-------------

(1 行)

I expect length 1.

thank you.

-----Original Message-----
From: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Sent: Wednesday, June 1, 2022 2:27 PM
To: 荒井元成 <n2029@ndensan.co.jp>; pgsql-hackers@lists.postgresql.org
Subject: Re: Unicode Variation Selector and Combining character

On 30.05.22 02:27, 荒井元成 wrote:

> I tried it on PostgreSQL 13. If you use the Unicode Variation Selector

> and Combining Character

> , the base character and the Variation selector will be 2 in length.

> Since it will be one character on the display, we expect it to be one

> in length. Please provide a function corresponding to the unicode

> variasion selector. I hope It is supposed to be provided as an extension.

> The functions that need to be supported are as follows:

> |regexp_replace|regexp_split_to_array|regexp_split_to_table

Please show a test case of what you mean. For example,

select char_length(...) returns X but should return Y

Examples with Unicode escapes (U&'\NNNN...') would be the most robust.

Attachment

Re: Unicode Variation Selector and Combining character

From

Thomas Munro

Date:

01 June 2022, 10:09:23

On Wed, Jun 1, 2022 at 6:15 PM 荒井元成 <n2029@ndensan.co.jp> wrote:
> D209007=# select char_length(U&'\+0066FE' || U&'\+0E0103') ;
> char_length
> -------------
>            2
> (1 行)
>
> I expect length 1.

No opinion here, but I did happen to see Noriyoshi Shinoda's slides
about this topic a little while ago, comparing different databases:

https://www.slideshare.net/noriyoshishinoda/postgresql-unconference-29-unicode-ivs

It's the same with Latin combining characters... we count the
individual codepoints of combining sequences:

postgres=# select 'e' || U&'\0301', length('e' || U&'\0301');
 ?column? | length
----------+--------
 é        |      2
(1 row)

Re: Unicode Variation Selector and Combining character

From

Thomas Munro

Date:

01 June 2022, 10:39:55

On Wed, Jun 1, 2022 at 7:09 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Wed, Jun 1, 2022 at 6:15 PM 荒井元成 <n2029@ndensan.co.jp> wrote:
> > D209007=# select char_length(U&'\+0066FE' || U&'\+0E0103') ;
> > char_length
> > -------------
> >            2
> > (1 行)
> >
> > I expect length 1.
>
> No opinion here, but I did happen to see Noriyoshi Shinoda's slides
> about this topic a little while ago, comparing different databases:
>
> https://www.slideshare.net/noriyoshishinoda/postgresql-unconference-29-unicode-ivs
>
> It's the same with Latin combining characters... we count the
> individual codepoints of combining sequences:
>
> postgres=# select 'e' || U&'\0301', length('e' || U&'\0301');
>  ?column? | length
> ----------+--------
>  é        |      2
> (1 row)

Looking around a bit, it might be interesting to check if the
icu_character_boundaries() function in Daniel Vérité's icu_ext treats
IVSs as single grapheme clusters.

Re: Unicode Variation Selector and Combining character

From

"Daniel Verite"

Date:

01 June 2022, 12:45:47

    Thomas Munro wrote:

> Looking around a bit, it might be interesting to check if the
> icu_character_boundaries() function in Daniel Vérité's icu_ext treats
> IVSs as single grapheme clusters.

It does.

with strings(s) as (
 values (U&'\+0066FE' || U&'\+0E0103'),
    (U&'\+00304B' || U&'\+00309A')
)
select s,
  octet_length(s),
  char_length(s),
  (select count(*) from icu_character_boundaries(s,'en')) as graphemes
from strings;


  s  | octet_length | char_length | graphemes
-----+--------------+-------------+-----------
 曾󠄃 |        7 |          2 |      1
 か゚  |         6 |           2 |       1



Best regards,
--
Daniel Vérité
https://postgresql.verite.pro/
Twitter: @DanielVerite

Re: Unicode Variation Selector and Combining character

From

Peter Eisentraut

Date:

01 June 2022, 13:25:49

On 01.06.22 08:15, 荒井元成 wrote:
> D209007=# select char_length(U&'\+0066FE' || U&'\+0E0103') ;
> 
> char_length
> 
> -------------
> 
>             2
> 
> (1 行)
> 
> I expect length 1.

The char_length function is defined to return the length in characters, 
so 2 is the correct answer.  What you appear to be looking for is length 
in glyphs or length in graphemes or display width, or something like 
that.  There is no built-in server side function for that.

It looks like psql is getting the display width wrong, but that's a 
separate issue.

RE: Unicode Variation Selector and Combining character

From

荒井元成

Date:

01 June 2022, 13:35:36

Thank you for your reply.

I will check if there is any function below char_length that is realized by icu_ext.

substring|trim|btrim|left
|lpad|ltrim|regexp_match|regexp_matches
|regexp_replace|regexp_split_to_array|regexp_split_to_table
|replace|reverse|right|rpad|rtrim|split_part|strpos|substr|starts_with


Best regards,

-----Original Message-----
From: Daniel Verite <daniel@manitou-mail.org>
Sent: Wednesday, June 1, 2022 6:46 PM
To: Thomas Munro <thomas.munro@gmail.com>
Cc: 荒井元成 <n2029@ndensan.co.jp>; Peter Eisentraut <peter.eisentraut@enterprisedb.com>; PostgreSQL Hackers
<pgsql-hackers@lists.postgresql.org>
Subject: Re: Unicode Variation Selector and Combining character

    Thomas Munro wrote:

> Looking around a bit, it might be interesting to check if the
> icu_character_boundaries() function in Daniel Vérité's icu_ext treats
> IVSs as single grapheme clusters.

It does.

with strings(s) as (
 values (U&'\+0066FE' || U&'\+0E0103'),
    (U&'\+00304B' || U&'\+00309A')
)
select s,
  octet_length(s),
  char_length(s),
  (select count(*) from icu_character_boundaries(s,'en')) as graphemes from strings;


  s  | octet_length | char_length | graphemes
-----+--------------+-------------+-----------
 曾󠄃 |        7 |          2 |      1
 か゚  |         6 |           2 |       1



Best regards,
--
Daniel Vérité
https://postgresql.verite.pro/
Twitter: @DanielVerite