Thread: Web team meeting minutes
Hi! Here are the quick notes I made during the web-team meeting. Fills are always welcome. Moderation policy ----------------- o Show moderation policy on or linked from the news submission form. o On ultiple post per company: We're really after the poeple who announce every minor little thing. o Same issue on -announce. o Split between project and vendor announcements. And split events vs training. When we can get to the required fonts. o Summary: 1 product announcement per company per 30 days. Moderators may override if necessary. o We won't ask them to use HTML. We let them, but don't mention it. o Rejection message for news when rejected automated, in case people forget to send them the reason. o Gavin will check for the font to generate images - we really need that. Either we'll get the one we need, or find a similar free one that we can use. o Minimum content: Need a paragraph in each of the two boxes. Can be the same. o Summary must show why the item is relevant to PostgreSQL if it's not clear from the title. o On submission form: Inform people if the summary gets repeated or not. o Show limit size of boxes on submission form. Documentation and management ---------------------------- o Move to pgfoundry? o Set up trac for config files and wiki. Tickets as well. CommandPrompt will donate, or possibly EHPG. CMD host, gavin manage? Wwwmaster redundancy -------------------- o Wwwmaster = SPOF o Gavin can provide another mirror if needed. o Problem is the redundancy, not performance o Discussion on different replication and such things o Dave will set up a spare VM as standby. Verify backup schedules. Coldspare for manual failover in case of breakage for now. Mirror sites ------------ o All are ftp at the moment. o Add http download mirroring. Split links under download flags. o Update list of mirrors to include which area it's in, and show that in the list of mirrors. (Gavin) Gborg -> pgfoundry ------------------ o Larry will be working at this. Start by contact everybody and figure out what they need help with. //Magnus
I didn't participate in web-team meeting, so I'm adding:
use tsearch2 for full text searching over web-site.
Arguments: it's online index, it's has built-in support for
structured documents (4 currently, but it's enough for PostgreSQL site
and documentation), it's configurable and could be used in web-site engine.
I'd not move mailing list archive search right now, but after 8.2 release
with GIN support I see no problem to this.
I'm not available for web-development, but could help with configuring and
share my experience.
Oleg
On Fri, 14 Jul 2006, Magnus Hagander wrote:
> Hi!
>
> Here are the quick notes I made during the web-team meeting. Fills are
> always welcome.
>
>
>
> Moderation policy
> -----------------
> o Show moderation policy on or linked from the news submission
> form.
> o On ultiple post per company: We're really after the poeple who
> announce every minor little thing.
> o Same issue on -announce.
> o Split between project and vendor announcements. And split events
> vs training. When we can get to the required fonts.
> o Summary: 1 product announcement per company per 30 days.
> Moderators may override if necessary.
> o We won't ask them to use HTML. We let them, but don't mention
> it.
> o Rejection message for news when rejected automated, in case
> people forget to send them the reason.
> o Gavin will check for the font to generate images - we really
> need that. Either we'll get the one we need, or find a similar free one
> that we can use.
> o Minimum content: Need a paragraph in each of the two boxes. Can
> be the same.
> o Summary must show why the item is relevant to PostgreSQL if it's
> not clear from the title.
> o On submission form: Inform people if the summary gets repeated
> or not.
> o Show limit size of boxes on submission form.
>
>
>
>
> Documentation and management
> ----------------------------
> o Move to pgfoundry?
> o Set up trac for config files and wiki. Tickets as well.
> CommandPrompt will donate, or possibly EHPG. CMD host, gavin manage?
>
>
> Wwwmaster redundancy
> --------------------
> o Wwwmaster = SPOF
> o Gavin can provide another mirror if needed.
> o Problem is the redundancy, not performance
> o Discussion on different replication and such things
> o Dave will set up a spare VM as standby. Verify backup schedules.
> Coldspare for manual failover in case of breakage for now.
>
>
> Mirror sites
> ------------
> o All are ftp at the moment.
> o Add http download mirroring. Split links under download flags.
> o Update list of mirrors to include which area it's in, and show
> that in the list of mirrors. (Gavin)
>
>
> Gborg -> pgfoundry
> ------------------
> o Larry will be working at this. Start by contact everybody and
> figure out what they need help with.
>
>
> //Magnus
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
> -----Original Message----- > From: pgsql-www-owner@postgresql.org > [mailto:pgsql-www-owner@postgresql.org] On Behalf Of Oleg Bartunov > Sent: 14 July 2006 13:05 > To: Magnus Hagander > Cc: pgsql-www@postgresql.org > Subject: Re: [pgsql-www] Web team meeting minutes > > I didn't participate in web-team meeting, so I'm adding: > > use tsearch2 for full text searching over web-site. > > Arguments: it's online index, it's has built-in support for > structured documents (4 currently, but it's enough for PostgreSQL site > and documentation), it's configurable and could be used in > web-site engine. > I'd not move mailing list archive search right now, but after > 8.2 release > with GIN support I see no problem to this. > > I'm not available for web-development, but could help with > configuring and > share my experience. Hi Oleg, We can't add that to the minutes as it's not something we discussed. As a separate issue though, perhaps you could propose it, along with reasons why we should do it (ie. What benefit is there over the current solution), and what would actually be required to get it going if we decided to go that way? Regards, Dave.
On Fri, 14 Jul 2006, Dave Page wrote:
>
>
>> -----Original Message-----
>> From: pgsql-www-owner@postgresql.org
>> [mailto:pgsql-www-owner@postgresql.org] On Behalf Of Oleg Bartunov
>> Sent: 14 July 2006 13:05
>> To: Magnus Hagander
>> Cc: pgsql-www@postgresql.org
>> Subject: Re: [pgsql-www] Web team meeting minutes
>>
>> I didn't participate in web-team meeting, so I'm adding:
>>
>> use tsearch2 for full text searching over web-site.
>>
>> Arguments: it's online index, it's has built-in support for
>> structured documents (4 currently, but it's enough for PostgreSQL site
>> and documentation), it's configurable and could be used in
>> web-site engine.
>> I'd not move mailing list archive search right now, but after
>> 8.2 release
>> with GIN support I see no problem to this.
>>
>> I'm not available for web-development, but could help with
>> configuring and
>> share my experience.
>
> Hi Oleg,
>
> We can't add that to the minutes as it's not something we discussed.
Oh, I'm sorry
>
> As a separate issue though, perhaps you could propose it, along with
> reasons why we should do it (ie. What benefit is there over the current
> solution), and what would actually be required to get it going if we
> decided to go that way?
I just wanted to say, that current search is not designed for Web site indexing.
Search, for example, latest news title "Open Technology Group, Inc. announces
plPHP training" and you'll get nothing ! And will not be searched until new
index gets build. This is exactly why we've developed tsearch2 - online
indexing. If documents are in database, then requirement is just setup
tsearch2, if not - then you need sort of openfts.
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
> -----Original Message----- > From: Oleg Bartunov [mailto:oleg@sai.msu.su] > Sent: 14 July 2006 13:48 > To: Dave Page > Cc: Magnus Hagander; pgsql-www@postgresql.org > Subject: RE: [pgsql-www] Web team meeting minutes > > I just wanted to say, that current search is not designed for > Web site indexing. Err, from the site: ASPseek is an Internet search engine software developed by SWsoft and licensed as free software under GNU GPL. ASPseek consists of an indexing robot, a search daemon, and a CGI search frontend. It can index as many as a few million URLs and search for words and phrases, use wildcards, and do a Boolean search. Search results can be limited to time period given, site or Web space (set of sites) and sorted by relevance (PageRank is used) or date. > Search, for example, latest news title "Open Technology > Group, Inc. announces > plPHP training" and you'll get nothing ! And will not be > searched until new > index gets build. This is exactly why we've developed > tsearch2 - online > indexing. If documents are in database, then requirement is just setup > tsearch2, if not - then you need sort of openfts. Actually our port of Aspseek can do online indexing - John added an XML feed in which you can directly insert index data (he used to use it to accept catalogue feeds from online resellers iirc). The problem is that we don't have any way to stream the data off the website in that way, so we still end up crawling anyway. I do appreciate your point though, and if anyone can come up with a way to stream data from the website (perhaps just as part of the static build process) then it might be worth looking at. Archives would have the same problems I guess - whilst it would be easy enough index mail messages online, you have no way of knowing what the URL on archives.postgresql.org would be at that point, unless we fundamentally redesigned the entire archives site to run from the database. Regards, Dave.
Dave,
I see the main problem is not in search engine, but in the site engine !
It's just not database driven. So, I withdraw my words :)
Does web team consider changing web site engine ? I suggest not to use
home-made engines, since we have no power to support it, we do database
development, and we don't want to depend on specific person. There are
big open-source projects with stable, mature community and we could
just add fts capability we need, for example, to Drupal.
Oleg
On Fri, 14 Jul 2006, Dave Page wrote:
>
>
>> -----Original Message-----
>> From: Oleg Bartunov [mailto:oleg@sai.msu.su]
>> Sent: 14 July 2006 13:48
>> To: Dave Page
>> Cc: Magnus Hagander; pgsql-www@postgresql.org
>> Subject: RE: [pgsql-www] Web team meeting minutes
>>
>> I just wanted to say, that current search is not designed for
>> Web site indexing.
>
> Err, from the site:
>
> ASPseek is an Internet search engine software developed by SWsoft and
> licensed as free software under GNU GPL.
>
> ASPseek consists of an indexing robot, a search daemon, and a CGI search
> frontend. It can index as many as a few million URLs and search for
> words and phrases, use wildcards, and do a Boolean search. Search
> results can be limited to time period given, site or Web space (set of
> sites) and sorted by relevance (PageRank is used) or date.
>
>> Search, for example, latest news title "Open Technology
>> Group, Inc. announces
>> plPHP training" and you'll get nothing ! And will not be
>> searched until new
>> index gets build. This is exactly why we've developed
>> tsearch2 - online
>> indexing. If documents are in database, then requirement is just setup
>> tsearch2, if not - then you need sort of openfts.
>
> Actually our port of Aspseek can do online indexing - John added an XML
> feed in which you can directly insert index data (he used to use it to
> accept catalogue feeds from online resellers iirc). The problem is that
> we don't have any way to stream the data off the website in that way, so
> we still end up crawling anyway.
>
> I do appreciate your point though, and if anyone can come up with a way
> to stream data from the website (perhaps just as part of the static
> build process) then it might be worth looking at. Archives would have
> the same problems I guess - whilst it would be easy enough index mail
> messages online, you have no way of knowing what the URL on
> archives.postgresql.org would be at that point, unless we fundamentally
> redesigned the entire archives site to run from the database.
>
> Regards, Dave.
>
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83
> Dave, > > I see the main problem is not in search engine, but in the site > engine ! > It's just not database driven. So, I withdraw my words :) Does web > team consider changing web site engine ? I suggest not to use home- > made engines, since we have no power to support it, we do database > development, and we don't want to depend on specific person. There > are big open-source projects with stable, mature community and we > could just add fts capability we need, for example, to Drupal. We've had this discussion many times before, see the archives ;-) Basically, we have yet to find such an engine that actually does all the stuff that we want done without requiring *more* work than it is to maintain what we have now. One typical thing most fail on is the static mirroring. //Magnus
>
> Documentation and management
> ----------------------------
> o Move to pgfoundry?
> o Set up trac for config files and wiki. Tickets as well.
> CommandPrompt will donate, or possibly EHPG. CMD host, gavin manage?
I think we already have a trac setup for you guys, even subversion :)
The only think I haven't setup yet is the email on commit.
>
>
> Wwwmaster redundancy
> --------------------
> o Wwwmaster = SPOF
> o Gavin can provide another mirror if needed.
CMD could as well if it can be a vhost. We are currently out of boxen
(that will change in September)
Sincerely,
Joshua D. Drake
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
> It's just not database driven. So, I withdraw my words :) > Does web team consider changing web site engine ? I suggest not to use > home-made engines, since we have no power to support it, we do database > development, and we don't want to depend on specific person. There are > big open-source projects with stable, mature community and we could > just add fts capability we need, for example, to Drupal. Oleg this has been a huge undertaking to get the site where it is today. I doubt we will be changing soon, and definately not to Drupal of all things :) Drupal is nasty on all kinds of levels under the covers. If we switched to Drupal we would have to (at a minimum): 1. Insure that it runs with latest PHP (looks like the support 5 but no mention of 5.1) 2. Completely rework the database schema so it looks like someone who understands databases actually worked on the product 3. Rip out their search and replace it. And that is all before we get to work on getting content in there. Also keep in mind that Drupal's support for PostgreSQL is very limited. If I sound sour, I apologize... I have been working alot latetely on getting Drupal to be as cool as the website says. Sincerely, Joshua D. Drake > > Oleg > On Fri, 14 Jul 2006, Dave Page wrote: > >> >> >>> -----Original Message----- >>> From: Oleg Bartunov [mailto:oleg@sai.msu.su] >>> Sent: 14 July 2006 13:48 >>> To: Dave Page >>> Cc: Magnus Hagander; pgsql-www@postgresql.org >>> Subject: RE: [pgsql-www] Web team meeting minutes >>> >>> I just wanted to say, that current search is not designed for >>> Web site indexing. >> >> Err, from the site: >> >> ASPseek is an Internet search engine software developed by SWsoft and >> licensed as free software under GNU GPL. >> >> ASPseek consists of an indexing robot, a search daemon, and a CGI search >> frontend. It can index as many as a few million URLs and search for >> words and phrases, use wildcards, and do a Boolean search. Search >> results can be limited to time period given, site or Web space (set of >> sites) and sorted by relevance (PageRank is used) or date. >> >>> Search, for example, latest news title "Open Technology >>> Group, Inc. announces >>> plPHP training" and you'll get nothing ! And will not be >>> searched until new >>> index gets build. This is exactly why we've developed >>> tsearch2 - online >>> indexing. If documents are in database, then requirement is just setup >>> tsearch2, if not - then you need sort of openfts. >> >> Actually our port of Aspseek can do online indexing - John added an XML >> feed in which you can directly insert index data (he used to use it to >> accept catalogue feeds from online resellers iirc). The problem is that >> we don't have any way to stream the data off the website in that way, so >> we still end up crawling anyway. >> >> I do appreciate your point though, and if anyone can come up with a way >> to stream data from the website (perhaps just as part of the static >> build process) then it might be worth looking at. Archives would have >> the same problems I guess - whilst it would be easy enough index mail >> messages online, you have no way of knowing what the URL on >> archives.postgresql.org would be at that point, unless we fundamentally >> redesigned the entire archives site to run from the database. >> >> Regards, Dave. >> > > Regards, > Oleg > _____________________________________________________________ > Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), > Sternberg Astronomical Institute, Moscow University, Russia > Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ > phone: +007(495)939-16-83, +007(495)939-23-83 > > ---------------------------(end of broadcast)--------------------------- > TIP 5: don't forget to increase your free space map settings > -- === The PostgreSQL Company: Command Prompt, Inc. === Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240 Providing the most comprehensive PostgreSQL solutions since 1997 http://www.commandprompt.com/
> -----Original Message----- > From: pgsql-www-owner@postgresql.org > [mailto:pgsql-www-owner@postgresql.org] On Behalf Of Joshua D. Drake > Sent: 14 July 2006 15:45 > To: Magnus Hagander > Cc: pgsql-www@postgresql.org > Subject: Re: [pgsql-www] Web team meeting minutes > > > > > > Documentation and management > > ---------------------------- > > o Move to pgfoundry? > > o Set up trac for config files and wiki. Tickets as well. > > CommandPrompt will donate, or possibly EHPG. CMD host, > gavin manage? > > I think we already have a trac setup for you guys, even subversion :) > The only think I haven't setup yet is the email on commit. Didn't think you had setup Trac in the end. Devrim was going to speak to you about it anyway - if you're happy to host it, that'd be great. Gavin has volunteered to admin it, so please liase with him where necessary to get things going. > > > > Wwwmaster redundancy > > -------------------- > > o Wwwmaster = SPOF > > o Gavin can provide another mirror if needed. > > CMD could as well if it can be a vhost. We are currently out of boxen > (that will change in September) FreeBSD vhost? If linux, then I already have the ball rolling here. Regards, Dave.
> -----Original Message----- > From: Oleg Bartunov [mailto:oleg@sai.msu.su] > Sent: 14 July 2006 14:22 > To: Dave Page > Cc: Magnus Hagander; pgsql-www@postgresql.org > Subject: RE: [pgsql-www] Search (was: Web team meeting minutes) > > Dave, > > I see the main problem is not in search engine, but in the > site engine ! > It's just not database driven. So, I withdraw my words :) It's entirely database /driven/, it's just the text index that's fs based. We run a fork of the ASPSeek code which has a few improvements over the official code including the XML data feed I mentioned, and support for PostgreSQL (as opposed to MySQL or Oracle which the standard code support). > Does web team consider changing web site engine ? I suggest not to use > home-made engines, since we have no power to support it, we > do database > development, and we don't want to depend on specific person. There are > big open-source projects with stable, mature community and we could > just add fts capability we need, for example, to Drupal. Hmm, well, see JD's comments on Drupal. After many years of trying different search engines, ASPSeek is by far the best we've found yet which *doesn't* require lots of custom code, and can be relatively easily managed by any one of us. I'd love for us to use Tsearch to do it, but it seems to me we'd need far too much custom code that would definitely be harder to manage. So unless you're about to release pgGoogle 1.0... :-) Regards, Dave.
>> I think we already have a trac setup for you guys, even subversion :)
>> The only think I haven't setup yet is the email on commit.
>
> Didn't think you had setup Trac in the end. Devrim was going to speak to
> you about it anyway - if you're happy to host it, that'd be great. Gavin
> has volunteered to admin it, so please liase with him where necessary to
> get things going.
Devrim has full access to that box as well so this should be pretty easy.
>
>>> Wwwmaster redundancy
>>> --------------------
>>> o Wwwmaster = SPOF
>>> o Gavin can provide another mirror if needed.
>> CMD could as well if it can be a vhost. We are currently out of boxen
>> (that will change in September)
>
> FreeBSD vhost? If linux, then I already have the ball rolling here.
No Linux (we do no FreeBSD, we don't know it well enough).
Sincerely,
Joshua D. Drake
>
> Regards, Dave.
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>
--
=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/
On Fri, 14 Jul 2006, Dave Page wrote: > > >> -----Original Message----- >> From: Oleg Bartunov [mailto:oleg@sai.msu.su] >> Sent: 14 July 2006 14:22 >> To: Dave Page >> Cc: Magnus Hagander; pgsql-www@postgresql.org >> Subject: RE: [pgsql-www] Search (was: Web team meeting minutes) >> >> Dave, >> >> I see the main problem is not in search engine, but in the >> site engine ! >> It's just not database driven. So, I withdraw my words :) > > It's entirely database /driven/, it's just the text index that's fs > based. We run a fork of the ASPSeek code which has a few improvements > over the official code including the XML data feed I mentioned, and > support for PostgreSQL (as opposed to MySQL or Oracle which the standard > code support). if it's database driven what prevent updating index once db updates ? it's not easy task and ASPseek explicitly documents this. If you got, for example, 10 documents updated in a hour, you need to update a lot of rows, the best case you should update as many rows as the number of unique words in thes documents. That's why I don't believe you could ever run online index with ASPseek. Do you have a hook in the site engine to know if something gets changed ? Sort of webservice would be nice. I'd play with it to made a prototype of search engine. > >> Does web team consider changing web site engine ? I suggest not to use >> home-made engines, since we have no power to support it, we >> do database >> development, and we don't want to depend on specific person. There are >> big open-source projects with stable, mature community and we could >> just add fts capability we need, for example, to Drupal. > > Hmm, well, see JD's comments on Drupal. After many years of trying > different search engines, ASPSeek is by far the best we've found yet > which *doesn't* require lots of custom code, and can be relatively > easily managed by any one of us. I'd love for us to use Tsearch to do > it, but it seems to me we'd need far too much custom code that would > definitely be harder to manage. > I'm not insisting on Drupal, it was just an example. I want to say that better to have engine supported by community, than developing itself. What we need is a hook which could inform search engine all changes happened at site. I'd prefer to work with webservice and provide also webservice to be "loosly coupled". That would be nice project for student. I don't have time to develop search webservice myself, but could help. > So unless you're about to release pgGoogle 1.0... I once wrote simple perl crawler, based on OpenFTS and run it for several sites. It's not a lot of code, check this http://mira.sai.msu.su/~megera/pgsql/ http://mira.sai.msu.su/~megera/pgsql/varlena/ It's static, since there is no hooks available. Regards, Oleg _____________________________________________________________ Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru), Sternberg Astronomical Institute, Moscow University, Russia Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(495)939-16-83, +007(495)939-23-83
> -----Original Message----- > From: Oleg Bartunov [mailto:oleg@sai.msu.su] > Sent: 14 July 2006 18:36 > To: Dave Page > Cc: Magnus Hagander; pgsql-www@postgresql.org > Subject: RE: [pgsql-www] Search (was: Web team meeting minutes) > > Do you have a hook in the site > engine to know if > something gets changed ? Sort of webservice would be nice. > I'd play with > it to made a prototype of search engine. http://gborg.postgresql.org/cgi-bin/cvsweb.cgi/portal/tools/mirror.php?r ev=1.16;cvsroot=pgweb Is the crawler that builds the static site which is what users see for the most part. A hook could be added in HTTP_Mirror::_writeFile to feed files to an online index. Regards, Dave.