Thread: How to speed up tsearch2 indexing
All, I have a table with ~ 3 million records. I'm indexing a field holding names, no more than 200 bytes each. Indexing the resulting tsvector takes forever. It's been running now for more than 40 hours on a Linux with PG 8.01, a single Xeon & 4GB RAM. My work_mem postgresql.conf parameter is at 240960 and maintenance_work_mem at 96384, although the index task is using at most 12MB. Task is 99% cpu bound. Is there any way I may speed up the indexing? TIA, -- Werner Bohl <WernerBohl@infutor.com> IDS de Costa Rica S.A.
On Wed, 9 Mar 2005, Werner Bohl wrote: > All, > > I have a table with ~ 3 million records. I'm indexing a field holding > names, no more than 200 bytes each. Indexing the resulting tsvector > takes forever. It's been running now for more than 40 hours on a Linux > with PG 8.01, a single Xeon & 4GB RAM. My work_mem postgresql.conf > parameter is at 240960 and maintenance_work_mem at 96384, although the > index task is using at most 12MB. Task is 99% cpu bound. Is there any > way I may speed up the indexing? What's your tsearch2 configuration ? Do you use dictionaries ? I wrote a brief explanation of tsearch2 internals http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_internals Hope, it could help you. > > > TIA, > > > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
On Wed, 2005-03-09 at 20:41 +0300, Oleg Bartunov wrote: > What's your tsearch2 configuration ? Do you use dictionaries ? > I wrote a brief explanation of tsearch2 internals > http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_internals > Tsearch2 is using default english configuration. No dictionaries, just put some more stop words (10) in english.stop. > Hope, it could help you. > > > > > > > TIA, > > > > > > > > Regards, > Oleg > _____________________________________________________________ > Oleg Bartunov, sci.researcher, hostmaster of AstroNet, > Sternberg Astronomical Institute, Moscow University (Russia) > Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ > phone: +007(095)939-16-83, +007(095)939-23-83 > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org
On Wed, 9 Mar 2005, Werner Bohl wrote: > On Wed, 2005-03-09 at 20:41 +0300, Oleg Bartunov wrote: > >> What's your tsearch2 configuration ? Do you use dictionaries ? >> I wrote a brief explanation of tsearch2 internals >> http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_internals >> > Tsearch2 is using default english configuration. No dictionaries, just > put some more stop words (10) in english.stop. it's not good, because you, probably, have a lot of unique words. Do you have some statistics, see stat() function ? > >> Hope, it could help you. >> >>> >>> >>> TIA, >>> >>> >>> >> >> Regards, >> Oleg >> _____________________________________________________________ >> Oleg Bartunov, sci.researcher, hostmaster of AstroNet, >> Sternberg Astronomical Institute, Moscow University (Russia) >> Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ >> phone: +007(095)939-16-83, +007(095)939-23-83 >> >> ---------------------------(end of broadcast)--------------------------- >> TIP 6: Have you searched our list archives? >> >> http://archives.postgresql.org > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83