Thread: Full-text search default vs specified configuration

I've been looking at a problem someone encountered with ts_headline:
http://archives.postgresql.org/pgsql-general/2008-02/msg01035.php

It turns out the problem was mixing ts_headline(<no specified config>) 
with to_tsquery(<specified config>) where <specified config> wasn't the 
default.

Fair enough, and in retrospect it's obvious. However, I fear it's going 
to be a pretty common error. It's also one that's not easy to catch - 
you can test a configuration, but you can't see what configuration 
generated a particular tsvector / tsquery (afaict).

I realise there was a lot of discussion during 8.3 devt about what was 
wanted from a default config and I'm guessing there's nothing that can 
be done for 8.3.x

Would there be any support for two changes in 8.4 though?

1. Tag tsvector/tsquery's with the (oid of) their configuration?
This could then generate a warning/error if you are running a tsquery 
against the wrong tsvector / combining two incompatible tsvectors etc.

2. Either warn or require CASCADE on changes to a 
configuration/dictionary that could impact existing indexes etc.
I've done it once myself where a stopword dictionary was changed from 
accept=true to accept=false. That change is OK (as long as you don't 
mind rogue tokens in your tsvectors) but others are probably not.

--   Richard Huxton  Archonet Ltd

Richard Huxton <dev@archonet.com> writes:
> Would there be any support for two changes in 8.4 though?

> 1. Tag tsvector/tsquery's with the (oid of) their configuration?

> 2. Either warn or require CASCADE on changes to a 
> configuration/dictionary that could impact existing indexes etc.

IIRC, the current behavior is intentional --- Oleg and Teodor argued
that tsvector values are relatively independent of small changes in
configuration and we should *not* force people to, say, reindex their
tables every time they add or subtract a stopword.  If we had some
measure of whether a TS configuration change was "critical" or not,
it might make sense to restrict critical changes; but I fear that
would be kind of hard to determine.
        regards, tom lane

Tom Lane wrote:
> Richard Huxton <dev@archonet.com> writes:
>> Would there be any support for two changes in 8.4 though?
> 
>> 1. Tag tsvector/tsquery's with the (oid of) their configuration?
> 
>> 2. Either warn or require CASCADE on changes to a 
>> configuration/dictionary that could impact existing indexes etc.
> 
> IIRC, the current behavior is intentional --- Oleg and Teodor argued
> that tsvector values are relatively independent of small changes in
> configuration and we should *not* force people to, say, reindex their
> tables every time they add or subtract a stopword.  If we had some
> measure of whether a TS configuration change was "critical" or not,
> it might make sense to restrict critical changes; but I fear that
> would be kind of hard to determine.

Well, clearly in my example it didn't impact operation at all, but it's 
an accident waiting to happen (and more importantly, a hard one to track 
down). It's like running SQL-ASCII encoding, everything just ticks along 
only to cause problems a month later.

What about the warning: "This may affect existing indexes - please 
check". Would that cause anyone problems?

What worries me is that it might take 10 messages on general/sql list to 
figure out the problem. This was reported as "words with many hits 
causes problems".

Maybe it's just a matter of getting the message out: "always specify the 
config or never specify the config".

--   Richard Huxton  Archonet Ltd

On Fri, 22 Feb 2008, Richard Huxton wrote:

> Tom Lane wrote:
>> Richard Huxton <dev@archonet.com> writes:
>>> Would there be any support for two changes in 8.4 though?
>> 
>>> 1. Tag tsvector/tsquery's with the (oid of) their configuration?
>> 
>>> 2. Either warn or require CASCADE on changes to a configuration/dictionary 
>>> that could impact existing indexes etc.
>> 
>> IIRC, the current behavior is intentional --- Oleg and Teodor argued
>> that tsvector values are relatively independent of small changes in
>> configuration and we should *not* force people to, say, reindex their
>> tables every time they add or subtract a stopword.  If we had some
>> measure of whether a TS configuration change was "critical" or not,
>> it might make sense to restrict critical changes; but I fear that
>> would be kind of hard to determine.
>
> Well, clearly in my example it didn't impact operation at all, but it's an 
> accident waiting to happen (and more importantly, a hard one to track down). 
> It's like running SQL-ASCII encoding, everything just ticks along only to 
> cause problems a month later.
>
> What about the warning: "This may affect existing indexes - please check". 
> Would that cause anyone problems?
>
> What worries me is that it might take 10 messages on general/sql list to 
> figure out the problem. This was reported as "words with many hits causes 
> problems".

He just didn't read documentation thoroughly.

>
> Maybe it's just a matter of getting the message out: "always specify the 
> config or never specify the config".

Probably, just stress this in documentation.

    Regards,        Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83