Re: [PERFORM] One large v. many small - Mailing list pgsql-general

On Thu, 30 Jan 2003, Gregory Wood wrote:

> While you make an excellent point (i.e. you can't always through hardware,
> especially excessive hardware at the problem), I would err on the side of
> doing things the right way. It usually ends up making the software easier to
> maintain and add to. A poor design to save a few thousand dollars on
> hardware now can cost many tens of thousands (or more) dollars on
> programming time down the road.
>


fun story - I was part of a dot com and we had an informix database and
the schema was pretty "good" - ref integrity and "logical layout".  What
happened
was our traffic started increasng dramatically.  We ended up having to
disable all the ref integrity simply because it gave us a 50% boost. It
was unfortuate, but you have to do it.  Sometimes you have to comprimise.
As for throwing hardware at it - it was already running on a $500k sun
box, an upgrade would have likely gone into the 7 digit range.
Not to mention you don't exactly get a quick turnaround on
hardware of that type.. a u10 perhaps, but not a big beefy box.
(Eventually  we did upgrade the db machine when we got another round of
funding)

so after a week of engineering and futzing we had things under control..
(db changes, massive app changes (agressive caching))

Yes it was horrid to throw out RI (which caused some minor issues
later) but when the business is riding on it.. you make it work any way
you can.  In a perfect world I would have done it another way, but when
the site is down (read: your business is not running, you are losing large
amounts of money) you need to put on your fire fighter suit, not your lab
coat.

I remember one time we were featured on CNBC and our traffic jumped by
1000% (yes, 1000) - our poor machines were hosed.  Now we did throw
hardware at this problem (more frontends) however aquiring hardware in
time of crisis is not terribly easy (took 3 days in this case).  So you
have to look at your other routes.

sometimes your design or some implementation details will be comprimised..
its a fact of business.  If the best design always won then why don't I
have an alpha for my machine machine? they are the fastest, best cpu
around.  (I'll admit, alpha failed a lot because of marketing
issues and cost) Business drives everything.  I'd rather continue getting
a paycheck than having a wonderfully designed db that doesn't perform
well and is causing us to lose money.

If you have the ability (ie, you know your site is going to end up doing
22M page views, or some other statistic like that) to see what things will
be like later and are not fighting a fire, design is wonderful. (Lets not
forget time.  I was just on a major project and they scheduled _3_ weeks
of design & coding.. we were quite upset about that one.. and they
arranged it so the launch date was set in stone.  man.. worked some
long nights..)

getting back to the original posters thing - why not just try a test to
see how things perform yourself?  Try some tests with 3000 tables, and try
a test with 1 table with a client_id (int) field.  Or as <insert name of
person who I forget> said, you could even make a boatload of views and
change your insertion logic..

anyway, sorry if I flamed anybody or if they took it personally.
just stating some experiences I've had.

------------------------------------------------------------------------------
Jeff Trout <jeff@jefftrout.com>                  http://www.jefftrout.com/
   Ronald McDonald, with the help of cheese soup,
       controls America from a secret volkswagon hidden in the past
-------------------------------------------------------------------------------