how does full text searching tokenize words ? can it be altered? - Mailing list pgsql-general

From Jonathan Vanasco
Subject how does full text searching tokenize words ? can it be altered?
Date
Msg-id 93848FE9-AECD-4044-AB1E-AD612FCDB09A@2xlp.com
Whole thread Raw
List pgsql-general
I'm getting a handful of 'can not index words longer than 2047 characters' on my `gin` indexes.

1. does this 2047 character count correspond to tokens / indexed words?
2. if so, is there a way to lower this number ?
3. is there a way to profile the index for the frequency of tokens ?


( apologies in advance if this looks familiar, i posted this as part of a larger question last month; everything but
thiswas answered by the list and I can't find answers to this online ) 




pgsql-general by date:

Previous
From: Don Brown
Date:
Subject: Windows Installation User account - Correct database for us
Next
From: Steve Atkins
Date:
Subject: Re: Windows Installation User account - Correct database for us