Thread: Filtering dictionaries support and unaccent dictionary
Hi there,
we'd like to introduce filtering dictionaries support for text search
and new contrib module unaccent, which provides useful example of
filtering dictionary. It finally solves the known problem of
incorrect generation of headlines of text with accents.
Also, this module provides unaccent() functions, which is a simple
wrapper on unaccent dictionary.
Regards, Oleg
PS. I hope it's not late for July commitfest !
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
Oleg Bartunov wrote: Hi, > we'd like to introduce filtering dictionaries support for text search > and new contrib module unaccent, which provides useful example of > filtering dictionary. It finally solves the known problem of incorrect > generation of headlines of text with accents. I'm curious about the pg_regress change ... is it really necessary? AFAICS the changes to the core code are very small; I wonder if you should commit it separately i.e. without the contrib module, and add the that one in another commit. As for the contrib module, I think it could use a lot more function header comments! Also, it would be great if it could be used separately from tsearch, i.e. that it provided a function unaccent(text) returns text that unaccented arbitrary strings (I guess it would use the default tsconfig). -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
> I'm curious about the pg_regress change ... is it really necessary?
To test unaccent dictionary it's needed to input accented characters, not all
encodings allow that. UTF8 allows that, but it doesn't compatible with a lot of
locales. So, --no-locale should be propagated to CREATE DATABASE command as it's
done for encoding.
> AFAICS the changes to the core code are very small; I wonder if you
> should commit it separately i.e. without the contrib module, and add the
> that one in another commit.
Split patch to two parts:
filter_dictionary-0.1.gz - core changes, including pg_regress changes
unaccent-0.5.gz - contrib module
Also, I added some comments into code and did cosmetic changes in docs.
> As for the contrib module, I think it could use a lot more function
> header comments! Also, it would be great if it could be used separately
> from tsearch, i.e. that it provided a function unaccent(text) returns
> text that unaccented arbitrary strings (I guess it would use the default
> tsconfig).
Umm? Module provides unaccent(text) and unaccent(regdictionary, text) functions.
--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/
Attachment
On Tuesday 14 July 2009 22:12:28 Oleg Bartunov wrote: > we'd like to introduce filtering dictionaries support for text search > and new contrib module unaccent, which provides useful example of > filtering dictionary. It finally solves the known problem of > incorrect generation of headlines of text with accents. What is the source of the unaccent rules, and how complete is the rule set?
On Wed, 29 Jul 2009, Peter Eisentraut wrote:
> On Tuesday 14 July 2009 22:12:28 Oleg Bartunov wrote:
>> we'd like to introduce filtering dictionaries support for text search
>> and new contrib module unaccent, which provides useful example of
>> filtering dictionary. It finally solves the known problem of
>> incorrect generation of headlines of text with accents.
>
> What is the source of the unaccent rules, and how complete is the rule set?
unicode tables from unicode.org. It'be nice if someone check the completeness.
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
Teodor Sigaev wrote: > >As for the contrib module, I think it could use a lot more function > >header comments! Also, it would be great if it could be used separately > >from tsearch, i.e. that it provided a function unaccent(text) returns > >text that unaccented arbitrary strings (I guess it would use the default > >tsconfig). > Umm? Module provides unaccent(text) and unaccent(regdictionary, text) functions. Sorry, I failed to notice. Looks good. Isn't that function leaking "res" pointer? Also, I'm curious why you're allocating 2*sizeof(TSLexeme) in unaccent_lexize ... -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On Sat, Aug 1, 2009 at 12:35 AM, Alvaro Herrera<alvherre@commandprompt.com> wrote: > Teodor Sigaev wrote: > >> >As for the contrib module, I think it could use a lot more function >> >header comments! Also, it would be great if it could be used separately >> >from tsearch, i.e. that it provided a function unaccent(text) returns >> >text that unaccented arbitrary strings (I guess it would use the default >> >tsconfig). >> Umm? Module provides unaccent(text) and unaccent(regdictionary, text) functions. > > Sorry, I failed to notice. Looks good. > > Isn't that function leaking "res" pointer? Also, I'm curious why you're > allocating 2*sizeof(TSLexeme) in unaccent_lexize ... So are we waiting for an updated version of this patch? ...Robert
> Isn't that function leaking "res" pointer? Also, I'm curious why you're
fixed
> allocating 2*sizeof(TSLexeme) in unaccent_lexize ...
That's is a dictionary's interface part: lexize returns an array of TSLexeme and
last structure should have lexeme field NULL.
filter_dictionary file is not changed, it's attached only for consistency.
--
Teodor Sigaev E-mail: teodor@sigaev.ru
WWW: http://www.sigaev.ru/
Attachment
2009/8/6 Teodor Sigaev <teodor@sigaev.ru>: >> Isn't that function leaking "res" pointer? Also, I'm curious why you're > > fixed > >> allocating 2*sizeof(TSLexeme) in unaccent_lexize ... > > That's is a dictionary's interface part: lexize returns an array of TSLexeme > and last structure should have lexeme field NULL. > > > filter_dictionary file is not changed, it's attached only for consistency. I am not sure whether this has been formally reviewed by anyone yet; do we think it's "Ready for Committer"? Thanks, ...Robert
On Thu, Aug 6, 2009 at 10:46 PM, Robert Haas<robertmhaas@gmail.com> wrote: > > I am not sure whether this has been formally reviewed by anyone yet; > do we think it's "Ready for Committer"? > i was trying to make some review of this but besides that it compiles fine and passes regression tests doesn't know how to test it -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157
On Fri, Aug 7, 2009 at 10:44 AM, Jaime Casanova<jcasanov@systemguards.com.ec> wrote: > On Thu, Aug 6, 2009 at 10:46 PM, Robert Haas<robertmhaas@gmail.com> wrote: >> >> I am not sure whether this has been formally reviewed by anyone yet; >> do we think it's "Ready for Committer"? >> > > i was trying to make some review of this but besides that it compiles > fine and passes regression tests doesn't know how to test it > try to build the docs to see how to properly test this and seems like you have to teach contrib.sgml and bookindex.sgml about dict-unaccent... and when i did that i got this: """ openjade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -c /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml openjade:dict-unaccent.sgml:48:1:E: non SGML character number 128 openjade:dict-unaccent.sgml:49:1:E: non SGML character number 129 openjade:dict-unaccent.sgml:50:1:E: non SGML character number 130 openjade:dict-unaccent.sgml:51:1:E: non SGML character number 131 openjade:dict-unaccent.sgml:52:1:E: non SGML character number 132 openjade:dict-unaccent.sgml:53:1:E: non SGML character number 133 openjade:dict-unaccent.sgml:54:1:E: non SGML character number 134 openjade:dict-unaccent.sgml:116:4:E: element "B" undefined make: *** [HTML.index] Error 1 make: *** Se borra el archivo `HTML.index' """ -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157
On Tuesday 11 August 2009 08:28:24 Jaime Casanova wrote: > try to build the docs to see how to properly test this and seems like > you have to teach contrib.sgml and bookindex.sgml about > dict-unaccent... and when i did that i got this: > > """ > openjade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -c > /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d > stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml > openjade:dict-unaccent.sgml:48:1:E: non SGML character number 128 > openjade:dict-unaccent.sgml:49:1:E: non SGML character number 129 > openjade:dict-unaccent.sgml:50:1:E: non SGML character number 130 > openjade:dict-unaccent.sgml:51:1:E: non SGML character number 131 > openjade:dict-unaccent.sgml:52:1:E: non SGML character number 132 > openjade:dict-unaccent.sgml:53:1:E: non SGML character number 133 > openjade:dict-unaccent.sgml:54:1:E: non SGML character number 134 > openjade:dict-unaccent.sgml:116:4:E: element "B" undefined > make: *** [HTML.index] Error 1 > make: *** Se borra el archivo `HTML.index' > """ You should escape the special characters as well as the <b> that appears as part of the example output using character entitities (& etc.).
On Tue, Aug 11, 2009 at 4:31 AM, Peter Eisentraut<peter_e@gmx.net> wrote: > On Tuesday 11 August 2009 08:28:24 Jaime Casanova wrote: >> try to build the docs to see how to properly test this and seems like >> you have to teach contrib.sgml and bookindex.sgml about >> dict-unaccent... and when i did that i got this: >> >> """ >> openjade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -c >> /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d >> stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml >> openjade:dict-unaccent.sgml:48:1:E: non SGML character number 128 >> openjade:dict-unaccent.sgml:49:1:E: non SGML character number 129 >> openjade:dict-unaccent.sgml:50:1:E: non SGML character number 130 >> openjade:dict-unaccent.sgml:51:1:E: non SGML character number 131 >> openjade:dict-unaccent.sgml:52:1:E: non SGML character number 132 >> openjade:dict-unaccent.sgml:53:1:E: non SGML character number 133 >> openjade:dict-unaccent.sgml:54:1:E: non SGML character number 134 >> openjade:dict-unaccent.sgml:116:4:E: element "B" undefined >> make: *** [HTML.index] Error 1 >> make: *** Se borra el archivo `HTML.index' >> """ > > You should escape the special characters as well as the <b> that appears as > part of the example output using character entitities (& etc.). Sounds like this patch needs a little bit of doc adjustment per the above and is then ready for committer? ...Robert
Peter,
how to write accented characters in sgml ? Is't not allowed to write them
as is ?
Oleg
On Tue, 11 Aug 2009, Peter Eisentraut wrote:
> On Tuesday 11 August 2009 08:28:24 Jaime Casanova wrote:
>> try to build the docs to see how to properly test this and seems like
>> you have to teach contrib.sgml and bookindex.sgml about
>> dict-unaccent... and when i did that i got this:
>>
>> """
>> openjade -wall -wno-unused-param -wno-empty -wfully-tagged -D . -c
>> /usr/share/sgml/docbook/stylesheet/dsssl/modular/catalog -d
>> stylesheet.dsl -t sgml -i output-html -V html-index postgres.sgml
>> openjade:dict-unaccent.sgml:48:1:E: non SGML character number 128
>> openjade:dict-unaccent.sgml:49:1:E: non SGML character number 129
>> openjade:dict-unaccent.sgml:50:1:E: non SGML character number 130
>> openjade:dict-unaccent.sgml:51:1:E: non SGML character number 131
>> openjade:dict-unaccent.sgml:52:1:E: non SGML character number 132
>> openjade:dict-unaccent.sgml:53:1:E: non SGML character number 133
>> openjade:dict-unaccent.sgml:54:1:E: non SGML character number 134
>> openjade:dict-unaccent.sgml:116:4:E: element "B" undefined
>> make: *** [HTML.index] Error 1
>> make: *** Se borra el archivo `HTML.index'
>> """
>
> You should escape the special characters as well as the <b> that appears as
> part of the example output using character entitities (& etc.).
>
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
Oleg Bartunov wrote: > Peter, > > how to write accented characters in sgml ? Is't not allowed to write > them as is ? á for á, etc. You can't use characters that aren't in Latin-1 I think. Writing them literally is not allowed. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On Thursday 13 August 2009 18:07:51 Alvaro Herrera wrote: > Oleg Bartunov wrote: > > Peter, > > > > how to write accented characters in sgml ? Is't not allowed to write > > them as is ? > > á for á, etc. You can't use characters that aren't in Latin-1 I > think. Writing them literally is not allowed. It's somehow possible, but it's not as straightforward as say with XML. And you might get into a Latin-1 vs UTF-8 mixup. At least that's what I noticed in my limited testing the other day.
Peter Eisentraut wrote:
> On Thursday 13 August 2009 18:07:51 Alvaro Herrera wrote:
> > Oleg Bartunov wrote:
> > > Peter,
> > >
> > > how to write accented characters in sgml ? Is't not allowed to write
> > > them as is ?
> >
> > á for ?, etc. You can't use characters that aren't in Latin-1 I
> > think. Writing them literally is not allowed.
>
> It's somehow possible, but it's not as straightforward as say with XML. And
> you might get into a Latin-1 vs UTF-8 mixup. At least that's what I noticed
> in my limited testing the other day.
The top of release.sgml has instructions on that because that is often
something we need to do for names in release notes:
non-ASCII characters convert to HTML4 entity (&) escapes
official: http://www.w3.org/TR/html4/sgml/entities.html one page:
http://www.zipcon.net/~swhite/docs/computers/browsers/entities_page.html other lists:
http://www.zipcon.net/~swhite/docs/computers/browsers/entities.html
http://www.zipcon.net/~swhite/docs/computers/browsers/entities_page.html
http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references
we cannot use UTF8 because SGML Docbook does not support it
http://www.pemberley.com/janeinfo/latin1.html#latexta
-- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB
http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Thanks, Bruce ! Oleg On Thu, 13 Aug 2009, Bruce Momjian wrote: > Peter Eisentraut wrote: >> On Thursday 13 August 2009 18:07:51 Alvaro Herrera wrote: >>> Oleg Bartunov wrote: >>>> Peter, >>>> >>>> how to write accented characters in sgml ? Is't not allowed to write >>>> them as is ? >>> >>> á for ?, etc. You can't use characters that aren't in Latin-1 I >>> think. Writing them literally is not allowed. >> >> It's somehow possible, but it's not as straightforward as say with XML. And >> you might get into a Latin-1 vs UTF-8 mixup. At least that's what I noticed >> in my limited testing the other day. > > The top of release.sgml has instructions on that because that is often > something we need to do for names in release notes: > > non-ASCII characters convert to HTML4 entity (&) escapes > > official: http://www.w3.org/TR/html4/sgml/entities.html > one page: http://www.zipcon.net/~swhite/docs/computers/browsers/entities_page.html > other lists: http://www.zipcon.net/~swhite/docs/computers/browsers/entities.html > http://www.zipcon.net/~swhite/docs/computers/browsers/entities_page.html > http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references > > we cannot use UTF8 because SGML Docbook > does not support it > http://www.pemberley.com/janeinfo/latin1.html#latexta > > Regards, Oleg _____________________________________________________________ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83