Thread: Re: PostgreSQL search engine (Perl)

Re: PostgreSQL search engine (Perl)

From
Anders Nielsen
Date:
I can recommend using the Lucene search-engine, it's in Java and it's got
nothing to do with PostgreSQL, but it performs excellently.

-----Original Message-----
From: Kaare Rasmussen [mailto:kar@kakidata.dk]
Sent: 31. maj 2002 12:57
To: pgsql-general@postgresql.org
Subject: [GENERAL] PostgreSQL search engine (Perl)


> Search the documentation does not work right now:

What I was trying to find was information about full text indexing.

It's supposed to be used in a search engine. But maybe someone knows a
search engine in Perl optimized for PostgreSQL?

One that works ? ;-)

 --
Kaare Rasmussen            --Linux, spil,--        Tlf:        3816 2582
Kaki Data                tshirts, merchandize      Fax:        3816 2501
Howitzvej 75               Åben 14.00-18.00        Web:      www.suse.dk
2000 Frederiksberg        Lørdag 11.00-17.00       Email: kar@kakidata.dk

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org

Re: PostgreSQL search engine (Perl)

From
Oleg Bartunov
Date:
On Fri, 31 May 2002, Anders Nielsen wrote:

> I can recommend using the Lucene search-engine, it's in Java and it's got
> nothing to do with PostgreSQL, but it performs excellently.

main drawback of not-db-affiliated search engines is problem to access
metadata stored in db. Modern portals require a lot of such metadata,
think about authorization layer, versioning etc.
Besides that Lucene and many others like namazu (www.namazu.org)
are good enough. OpenFTS os our attempt to overcome metadata barrier.


>
> -----Original Message-----
> From: Kaare Rasmussen [mailto:kar@kakidata.dk]
> Sent: 31. maj 2002 12:57
> To: pgsql-general@postgresql.org
> Subject: [GENERAL] PostgreSQL search engine (Perl)
>
>
> > Search the documentation does not work right now:
>
> What I was trying to find was information about full text indexing.
>
> It's supposed to be used in a search engine. But maybe someone knows a
> search engine in Perl optimized for PostgreSQL?
>
> One that works ? ;-)
>
>  --
> Kaare Rasmussen            --Linux, spil,--        Tlf:        3816 2582
> Kaki Data                tshirts, merchandize      Fax:        3816 2501
> Howitzvej 75               еben 14.00-18.00        Web:      www.suse.dk
> 2000 Frederiksberg        LЬrdag 11.00-17.00       Email: kar@kakidata.dk
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html
>

    Regards,
        Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83


Re: PostgreSQL search engine (Perl)

From
Anders Nielsen
Date:
The problem with in-database search-engines are that they're usually tagged
on as a add-on feature and doesn't recieve the concentrated programming
effort that is required to create a high-performance full-text indexing
solution.

I'd be interested in comparing performance stats with people who use
OpenFTS, as I have a 8.5 gig fulltext Lucene index (including the indexed
text) and I am more than happy with the response times I'm seeing.

regards,
Anders Nielsen

-----Original Message-----
From: Oleg Bartunov [mailto:oleg@sai.msu.su]
Sent: 31. maj 2002 13:05
To: Anders Nielsen
Cc: 'Kaare Rasmussen'; pgsql-general@postgresql.org
Subject: Re: [GENERAL] PostgreSQL search engine (Perl)


On Fri, 31 May 2002, Anders Nielsen wrote:

> I can recommend using the Lucene search-engine, it's in Java and it's got
> nothing to do with PostgreSQL, but it performs excellently.

main drawback of not-db-affiliated search engines is problem to access
metadata stored in db. Modern portals require a lot of such metadata,
think about authorization layer, versioning etc.
Besides that Lucene and many others like namazu (www.namazu.org)
are good enough. OpenFTS os our attempt to overcome metadata barrier.


>
> -----Original Message-----
> From: Kaare Rasmussen [mailto:kar@kakidata.dk]
> Sent: 31. maj 2002 12:57
> To: pgsql-general@postgresql.org
> Subject: [GENERAL] PostgreSQL search engine (Perl)
>
>
> > Search the documentation does not work right now:
>
> What I was trying to find was information about full text indexing.
>
> It's supposed to be used in a search engine. But maybe someone knows a
> search engine in Perl optimized for PostgreSQL?
>
> One that works ? ;-)
>
>  --
> Kaare Rasmussen            --Linux, spil,--        Tlf:        3816 2582
> Kaki Data                tshirts, merchandize      Fax:        3816 2501
> Howitzvej 75               еben 14.00-18.00        Web:      www.suse.dk
> 2000 Frederiksberg        LЬrdag 11.00-17.00       Email: kar@kakidata.dk
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/users-lounge/docs/faq.html
>

    Regards,
        Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

Re: PostgreSQL search engine (Perl)

From
"Marc G. Fournier"
Date:
What kind of hardware resources do you have it sitting on?  CPUs, memory,
disks?

On Fri, 31 May 2002, Anders Nielsen wrote:

> The problem with in-database search-engines are that they're usually tagged
> on as a add-on feature and doesn't recieve the concentrated programming
> effort that is required to create a high-performance full-text indexing
> solution.
>
> I'd be interested in comparing performance stats with people who use
> OpenFTS, as I have a 8.5 gig fulltext Lucene index (including the indexed
> text) and I am more than happy with the response times I'm seeing.
>
> regards,
> Anders Nielsen
>
> -----Original Message-----
> From: Oleg Bartunov [mailto:oleg@sai.msu.su]
> Sent: 31. maj 2002 13:05
> To: Anders Nielsen
> Cc: 'Kaare Rasmussen'; pgsql-general@postgresql.org
> Subject: Re: [GENERAL] PostgreSQL search engine (Perl)
>
>
> On Fri, 31 May 2002, Anders Nielsen wrote:
>
> > I can recommend using the Lucene search-engine, it's in Java and it's got
> > nothing to do with PostgreSQL, but it performs excellently.
>
> main drawback of not-db-affiliated search engines is problem to access
> metadata stored in db. Modern portals require a lot of such metadata,
> think about authorization layer, versioning etc.
> Besides that Lucene and many others like namazu (www.namazu.org)
> are good enough. OpenFTS os our attempt to overcome metadata barrier.
>
>
> >
> > -----Original Message-----
> > From: Kaare Rasmussen [mailto:kar@kakidata.dk]
> > Sent: 31. maj 2002 12:57
> > To: pgsql-general@postgresql.org
> > Subject: [GENERAL] PostgreSQL search engine (Perl)
> >
> >
> > > Search the documentation does not work right now:
> >
> > What I was trying to find was information about full text indexing.
> >
> > It's supposed to be used in a search engine. But maybe someone knows a
> > search engine in Perl optimized for PostgreSQL?
> >
> > One that works ? ;-)
> >
> >  --
> > Kaare Rasmussen            --Linux, spil,--        Tlf:        3816 2582
> > Kaki Data                tshirts, merchandize      Fax:        3816 2501
> > Howitzvej 75               �ben 14.00-18.00        Web:      www.suse.dk
> > 2000 Frederiksberg        L�rdag 11.00-17.00       Email: kar@kakidata.dk
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 5: Have you checked our extensive FAQ?
> >
> > http://www.postgresql.org/users-lounge/docs/faq.html
> >
>
>     Regards,
>         Oleg
> _____________________________________________________________
> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> Sternberg Astronomical Institute, Moscow University (Russia)
> Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
> phone: +007(095)939-16-83, +007(095)939-23-83
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>


Re: PostgreSQL search engine (Perl)

From
Oleg Bartunov
Date:
On Fri, 31 May 2002, Anders Nielsen wrote:

> The problem with in-database search-engines are that they're usually tagged
> on as a add-on feature and doesn't recieve the concentrated programming
> effort that is required to create a high-performance full-text indexing
> solution.

You're right.
RDBMS isn't good for high-performance full-text indexing, im my
opinion. I dont' know commercial databases which have built-in
fts support. I mean 'fair' built-in search engine with
transactions support and simultaneous indexing.
We did all the best we could at the moment and for small and
medium sites openfts is very good. There is room for improving
and we hope we'll keep our hands on it.

>
> I'd be interested in comparing performance stats with people who use
> OpenFTS, as I have a 8.5 gig fulltext Lucene index (including the indexed
> text) and I am more than happy with the response times I'm seeing.

I bet Lucene will be a winner.
There is no magick in Lucene performance. Inverted indices are fast
for searching but very-very-very slow for updating.
Probably, your application doesnt require so much from database.
It's another story. We've built many search engines like Lucene and we do
know all pro's and contra's, but there are applicatios where transactional
integrity and access to metadata are much more important than performance.

>
> regards,
> Anders Nielsen
>
> -----Original Message-----
> From: Oleg Bartunov [mailto:oleg@sai.msu.su]
> Sent: 31. maj 2002 13:05
> To: Anders Nielsen
> Cc: 'Kaare Rasmussen'; pgsql-general@postgresql.org
> Subject: Re: [GENERAL] PostgreSQL search engine (Perl)
>
>
> On Fri, 31 May 2002, Anders Nielsen wrote:
>
> > I can recommend using the Lucene search-engine, it's in Java and it's got
> > nothing to do with PostgreSQL, but it performs excellently.
>
> main drawback of not-db-affiliated search engines is problem to access
> metadata stored in db. Modern portals require a lot of such metadata,
> think about authorization layer, versioning etc.
> Besides that Lucene and many others like namazu (www.namazu.org)
> are good enough. OpenFTS os our attempt to overcome metadata barrier.
>
>
> >
> > -----Original Message-----
> > From: Kaare Rasmussen [mailto:kar@kakidata.dk]
> > Sent: 31. maj 2002 12:57
> > To: pgsql-general@postgresql.org
> > Subject: [GENERAL] PostgreSQL search engine (Perl)
> >
> >
> > > Search the documentation does not work right now:
> >
> > What I was trying to find was information about full text indexing.
> >
> > It's supposed to be used in a search engine. But maybe someone knows a
> > search engine in Perl optimized for PostgreSQL?
> >
> > One that works ? ;-)
> >
> >  --
> > Kaare Rasmussen            --Linux, spil,--        Tlf:        3816 2582
> > Kaki Data                tshirts, merchandize      Fax:        3816 2501
> > Howitzvej 75               еben 14.00-18.00        Web:      www.suse.dk
> > 2000 Frederiksberg        LЬrdag 11.00-17.00       Email: kar@kakidata.dk
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 5: Have you checked our extensive FAQ?
> >
> > http://www.postgresql.org/users-lounge/docs/faq.html
> >
>
>     Regards,
>         Oleg
> _____________________________________________________________
> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> Sternberg Astronomical Institute, Moscow University (Russia)
> Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
> phone: +007(095)939-16-83, +007(095)939-23-83
>

    Regards,
        Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83


Re: PostgreSQL search engine (Perl)

From
Anders Nielsen
Date:
Single CPU Athlon Palomino 1800
2 IDE HD RAID 1
1.5 gigabytes DDR RAM

-----Original Message-----
From: Marc G. Fournier
To: Anders Nielsen
Cc: 'Oleg Bartunov'; 'Kaare Rasmussen'; pgsql-general@postgresql.org
Sent: 31-05-2002 16:25
Subject: Re: [GENERAL] PostgreSQL search engine (Perl)

This message uses a character set that is not supported by the Internet
Service.  To view the original message content,  open the attached
message. If the text doesn't display correctly, save the attachment to
disk, and then open it using a viewer that can display the original
character set. <<message.txt>>