Thread: creating a cluster

Does anyone have any new ways to create clusters without using initdb 
or bootstrap mode? I need to be able to create one without those 2 
things. Any ideas?

thanks!

Alex

On Mon, Jun 21, 2004 at 09:16:35PM -0400, Alexander Cohen wrote:
> Does anyone have any new ways to create clusters without using initdb 
> or bootstrap mode? I need to be able to create one without those 2 
> things. Any ideas?

initdb'ing somewhere else and copying the resulting directory?

-- 
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Find a bug in a program, and fix it, and the program will work today.
Show the program how to find and fix a bug, and the program
will work forever" (Oliver Silfridge)

Alexander Cohen <alex@toomuchspace.com> writes:
> Does anyone have any new ways to create clusters without using initdb 
> or bootstrap mode? I need to be able to create one without those 2 
> things. Any ideas?

Perhaps you should explain *why* you think you need this?
        regards, tom lane

Alvaro Herrera wrote:
> On Mon, Jun 21, 2004 at 09:16:35PM -0400, Alexander Cohen wrote:
> 
>>Does anyone have any new ways to create clusters without using initdb 
>>or bootstrap mode? I need to be able to create one without those 2 
>>things. Any ideas?
> 
> 
> initdb'ing somewhere else and copying the resulting directory?

Btw, I've been doing this for a binary distribution on Windows (Cygwin) 
and Linux. Primarily because initdb-ing + doing a bunch of SQL commands 
to the db takes a long time on Cygwin. Seems fine so far.

-- 
dave

On Jun 23, 2004, at 10:18 AM, David Garamond wrote:

> Alvaro Herrera wrote:
>> On Mon, Jun 21, 2004 at 09:16:35PM -0400, Alexander Cohen wrote:
>>> Does anyone have any new ways to create clusters without using 
>>> initdb or bootstrap mode? I need to be able to create one without 
>>> those 2 things. Any ideas?
>> initdb'ing somewhere else and copying the resulting directory?
>
> Btw, I've been doing this for a binary distribution on Windows 
> (Cygwin) and Linux. Primarily because initdb-ing + doing a bunch of 
> SQL commands to the db takes a long time on Cygwin. Seems fine so far.

And how do you take care of users for your distribution. If you created 
the cluster on your computer, does it not have your user name as the 
main root user? That needs to be changed when copying over the cluster, 
how do i that?

Alex

David Garamond <lists@zara.6.isreserved.com> writes:
> Alvaro Herrera wrote:
>> On Mon, Jun 21, 2004 at 09:16:35PM -0400, Alexander Cohen wrote:
>>> Does anyone have any new ways to create clusters without using initdb 
>>> or bootstrap mode? I need to be able to create one without those 2 
>>> things. Any ideas?
>>
>> initdb'ing somewhere else and copying the resulting directory?

> Btw, I've been doing this for a binary distribution on Windows (Cygwin) 
> and Linux.

Yeah, that would work fine as long as the "somewhere else" is using an
identical Postgres build.  I found out in off-list conversation that
Alexander wants to build a hacked-up version of Postgres with all
bootstrap code removed (and, I suppose, a bunch of other changes too).
Seems to me that file-level compatibility would be difficult to
guarantee under such circumstances, so I told him he ought to put back
the bootstrap support ... it's not like it's large ...
        regards, tom lane

On Jun 23, 2004, at 11:36 AM, Tom Lane wrote:

> David Garamond <lists@zara.6.isreserved.com> writes:
>> Alvaro Herrera wrote:
>>> On Mon, Jun 21, 2004 at 09:16:35PM -0400, Alexander Cohen wrote:
>>>> Does anyone have any new ways to create clusters without using 
>>>> initdb
>>>> or bootstrap mode? I need to be able to create one without those 2
>>>> things. Any ideas?
>>>
>>> initdb'ing somewhere else and copying the resulting directory?
>
>> Btw, I've been doing this for a binary distribution on Windows 
>> (Cygwin)
>> and Linux.
>
> Yeah, that would work fine as long as the "somewhere else" is using an
> identical Postgres build.  I found out in off-list conversation that
> Alexander wants to build a hacked-up version of Postgres with all
> bootstrap code removed (and, I suppose, a bunch of other changes too).
> Seems to me that file-level compatibility would be difficult to
> guarantee under such circumstances, so I told him he ought to put back
> the bootstrap support ... it's not like it's large ...

For the meantime, i ended up compiling a normal version of postgres and 
using that with initdb, then switching it over to my "hacked-up" 
version. It works, and thats all i need for now!

Alex

Hello,

I seem to recall that HyperThreading and PostgreSQL != good stuff...
There was a whole bunch of stuff recently on this... google the archives.

Sincerely,

Joshua D. Drake


Jaime Casanova wrote:
> Hi all,
>
> Can anyone tell me if postgresql has problems with xeon processors?
> If so, there is any fix or project of fix it?
>
> Thanx in advance,
>
> Jaime Casanova
>
>
>
>
> ------------------------------------------------------------------------
> *Do You Yahoo!?*
> <http://espanol.yahoo.com/mail_tagline/*http://espanol.news.yahoo.com>
> Todo lo que quieres saber de Estados Unidos, América Latina y el resto
> del Mundo.
> Visíta Yahoo! Noticias
> <http://espanol.yahoo.com/mail_tagline/*http://espanol.news.yahoo.com>.


--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

On Fri, 2004-06-25 at 14:13, Jaime Casanova wrote:
> Hi all,
>  
> Can anyone tell me if postgresql has problems with xeon processors? 
> If so, there is any fix or project of fix it? 

To PostgreSQL, there's no difference between a dual CPU machine with no
hyperthreading, and a single CPU machine with hyperthreading.

HOWEVER, there have been some issues with certain Operating Systems
running slower with hyperthreading enabled than without it.  Late model
Linux kernels (2.6) seem to have gotten enough logic into the scheduler
to get good performance on a hyperthreaded system.

On Sat, Jun 26, 2004 at 07:31:59PM -0600, Scott Marlowe wrote:
> On Fri, 2004-06-25 at 14:13, Jaime Casanova wrote:
> > Hi all,
> >  
> > Can anyone tell me if postgresql has problems with xeon processors? 
> > If so, there is any fix or project of fix it? 
> 
> To PostgreSQL, there's no difference between a dual CPU machine with no
> hyperthreading, and a single CPU machine with hyperthreading.

At some point there was trouble with spinlocks on some of the newer
Xeons (maybe those with hyperthreading?).  I think there was a good deal
of discussion and resulting development because of that.  According to
my archives it seems to be around december 2003.  There's even a CVS
entry, listed by cvs2cl as


2003-12-27 17:58  tgl
       * src/: backend/storage/lmgr/s_lock.c, include/storage/s_lock.h:         Improve spinlock code for recent x86
processors:insert a PAUSE         instruction in the s_lock() wait loop, and use test before         test-and-set in
TAS()macro to avoid unnecessary bus traffic.         Patch from Manfred Spraul, reworked a bit by Tom.
 

Not sure if this made the 7.4.3 release ...

-- 
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"You knock on that door or the sun will be shining on places inside you
that the sun doesn't usually shine" (en Death: "The High Cost of Living")

Centuries ago, Nostradamus foresaw when systemguards@yahoo.com (Jaime Casanova) would write:
> Can anyone tell me if postgresql has problems with xeon processors?
>
> If so, there is any fix or project of fix it?

Well, there's a known issue that IA-32 systems with more than 4GB of
memory use an extension known as "PAE" to bank-switch between the
banks of memory.  

Any time you switch banks, there's a fair little bit of work to be
done.  That includes multitasking systems that need to context switch
a few thousand times per second.

The "fix" for this problem is to rewrite all of your applications so
that they become conscious of which bits of memory they're using so
they can tune their own behaviour.  This, of course, requires
discarding useful notions such as "virtual memory" that are _assumed_
by most modern operating systems.

The fix is to buy hardware that hasn't been hacked up so badly.
-- 
select 'cbbrowne' || '@' || 'ntlug.org';
http://www3.sympatico.ca/cbbrowne/lsf.html
"There  is something in  the lecture  course which  may not  have been
visible so far, which is reality ..."  -- Arthur Norman

Christopher Browne <cbbrowne@acm.org> writes:

> Centuries ago, Nostradamus foresaw when systemguards@yahoo.com (Jaime Casanova) would write:
>> Can anyone tell me if postgresql has problems with xeon processors?
>>
>> If so, there is any fix or project of fix it?
>
> Well, there's a known issue that IA-32 systems with more than 4GB of
> memory use an extension known as "PAE" to bank-switch between the
> banks of memory.  

AIUI, it's not really "bank switching" in the old 8-bit sense.
Rather, there is a big linear 36-bit physical address space, and the
processor's page tables have been extended so they can point to any
page in this space.  Any one process still sees a 4GB (32-bit) address
space since that's how big the registers are.

> Any time you switch banks, there's a fair little bit of work to be
> done.  That includes multitasking systems that need to context switch
> a few thousand times per second.

I don't think this is any more overhead than a "normal" context
switch--cache misses, TLB flush etc.

> The "fix" for this problem is to rewrite all of your applications so
> that they become conscious of which bits of memory they're using so
> they can tune their own behaviour.  This, of course, requires
> discarding useful notions such as "virtual memory" that are _assumed_
> by most modern operating systems.

This is only if you need to address more than 32-bits worth of address
in a single process, which is not always the case on server-class
systems (on scientific/numerical workloads, it's often a big win).  In
which case you are certainly right:

> The fix is to buy hardware that hasn't been hacked up so badly.

64-bit systems get cheaper all the time...  :)

-Doug

Christopher Browne wrote:

>The "fix" for this problem is to rewrite all of your applications so
>that they become conscious of which bits of memory they're using so
>they can tune their own behaviour.  This, of course, requires
>discarding useful notions such as "virtual memory" that are _assumed_
>by most modern operating systems.
>
>  
>
This is misleading: PAE means that a 32-bit cpu can have more that 4 GB 
physical memory. Each process can map at most 4 (in reality: ~2) GB memory.
Many databases manage their own, huge buffer pool and read/write the 
database tables with O_DIRECT. These apps must support buffer pools > 2 
GB, which requires some work. Linux and Solaris contain a special 
syscall that helps Oracle to manage it's buffer pool for such setups 
(remap_page_rage()).
OTHO postgres has a small user space buffer pool, the majority of the 
file buffers are handled by OS. Thus no changes are required inside 
postgres for PAE, all it needs is an OS that support PAE for the buffer 
pool.

Regarding hyperthreading: I'm aware of two changes:
- busy loops must contain PAUSE instructions. Postgres does that.
- virtual aliases should be avoided: If two processes access memory at 
the same virtual address, then this can cause cache collisions and then 
misses. I think this is handled by the C library by randomizing the 
return addresses of malloc() and Intel mitigated the issue by improving 
the cache.

--   Manfred