Thread: session servers in ram

I'm looking at running session servers in ram.  All the data is
throw-away data, so my plan is to have a copy of the empty db on the
hard drive ready to go, and have a script that just copies it into ram
and starts the db there.  We're currently IO write bound with
fsync=off using a 15k5 seagate SAS drive, so I'm hoping that moving
the db into /dev/shm will help quite a bit here.

Does anybody any real world experience here or any words of sage
advice before I go off and start testing this?

I assume you intend to this or some variation of it.
  mount -t tmpfs -o size=1G tmpfs /pgfast

tmpfs file systems, including /dev/shm, ***should not*** be counted on as being RAM only.  File systems of this type in Linux, and at least Solaris also, can be swap since the tmpfs type is derived from swap and swap = memory + active swap partitions.

I would think that a ram disk or ramfs may be more what you're after.  Add ramdisk_size=X, where X is the maximum size in kilobytes, to the kernel line in your grub.conf file.  Unlike tmpfs, the ramdisk_size X parameter cannot be more than the memory of your server.  When using a ramdisk the ext2 file system would be best with all the fun noatime and like mount parameters.  This is good for a fast, volatile, fixed-size ramdisk.

OTOH, ramfs can be just as fast but with all the fun system-hanging features like growing until all RAM is consumed.
  mount -t ramfs ramfs /pgfast

Pro ... it will always be as fast, or possibly a hair faster, than tmpfs.

Con ... it won't show up in a df output unlike tmpfs or ramdisk.  Use the mount command with no parameters to look for it and be sure to unmount it when you're done.

Pro/Con ... you can't specify the file system type like with ramdisk.

Pro... it will only take away from memory as space is used, i.e. if you have 500M of memory in use and mount the file system but do nothing else then only 500M of memory is in use.  If you then copy a 100M file to it then 600M of memory is in use.  Delete that 100M file and you're back to 500M of used memory.

Pro/Con ... unlike other file systems, it will grow with the need... unchecked.  It could attempt to consume all available memory pushing all other processes out to swap and this is a bad, bad thing.


I'm sure there are other pro's & con's to ramfs.

HTH.

Greg

Hello

this is maybe off topic. Do you know memcached? We use it without
postgresql six or seven months for short-live data with big success.

regards
Pavel Stehule

2009/9/22 Greg Spiegelberg <gspiegelberg@gmail.com>:
> On Mon, Sep 21, 2009 at 5:39 PM, Scott Marlowe <scott.marlowe@gmail.com>
> wrote:
>>
>> I'm looking at running session servers in ram.  All the data is
>> throw-away data, so my plan is to have a copy of the empty db on the
>> hard drive ready to go, and have a script that just copies it into ram
>> and starts the db there.  We're currently IO write bound with
>> fsync=off using a 15k5 seagate SAS drive, so I'm hoping that moving
>> the db into /dev/shm will help quite a bit here.
>>
>> Does anybody any real world experience here or any words of sage
>> advice before I go off and start testing this?
>>
>
> I assume you intend to this or some variation of it.
>   mount -t tmpfs -o size=1G tmpfs /pgfast
>
> tmpfs file systems, including /dev/shm, ***should not*** be counted on as
> being RAM only.  File systems of this type in Linux, and at least Solaris
> also, can be swap since the tmpfs type is derived from swap and swap =
> memory + active swap partitions.
>
> I would think that a ram disk or ramfs may be more what you're after.  Add
> ramdisk_size=X, where X is the maximum size in kilobytes, to the kernel line
> in your grub.conf file.  Unlike tmpfs, the ramdisk_size X parameter cannot
> be more than the memory of your server.  When using a ramdisk the ext2 file
> system would be best with all the fun noatime and like mount parameters.
> This is good for a fast, volatile, fixed-size ramdisk.
>
> OTOH, ramfs can be just as fast but with all the fun system-hanging features
> like growing until all RAM is consumed.
>   mount -t ramfs ramfs /pgfast
>
> Pro ... it will always be as fast, or possibly a hair faster, than tmpfs.
>
> Con ... it won't show up in a df output unlike tmpfs or ramdisk.  Use the
> mount command with no parameters to look for it and be sure to unmount it
> when you're done.
>
> Pro/Con ... you can't specify the file system type like with ramdisk.
>
> Pro... it will only take away from memory as space is used, i.e. if you have
> 500M of memory in use and mount the file system but do nothing else then
> only 500M of memory is in use.  If you then copy a 100M file to it then 600M
> of memory is in use.  Delete that 100M file and you're back to 500M of used
> memory.
>
> Pro/Con ... unlike other file systems, it will grow with the need...
> unchecked.  It could attempt to consume all available memory pushing all
> other processes out to swap and this is a bad, bad thing.
>
>
> I'm sure there are other pro's & con's to ramfs.
>
> HTH.
>
> Greg
>

* Scott Marlowe <scott.marlowe@gmail.com> [090921 19:39]:
> I'm looking at running session servers in ram.  All the data is
> throw-away data, so my plan is to have a copy of the empty db on the
> hard drive ready to go, and have a script that just copies it into ram
> and starts the db there.  We're currently IO write bound with
> fsync=off using a 15k5 seagate SAS drive, so I'm hoping that moving
> the db into /dev/shm will help quite a bit here.
>
> Does anybody any real world experience here or any words of sage
> advice before I go off and start testing this?

*If* fsync=off is really meaning that there are no sync commands
happening on your pg partitions (and nothing else, like syslog, is
causing syncs on them), and you're kernel is tuned to allow the maximum
dirty buffers/life, then I'm not sure that's going to gain you
anything...  If your pg processes are blocked writing, with no syncs,
then they are blocked because the kernel has no more buffers available
for buffering the writes...

Moving your backing store from a disk-based FS to disk-based swap is only
going to shift the route of being forced to hit the disk...

Of course, details matter, and results trump theory, so test it ;-)

a.

--
Aidan Van Dyk                                             Create like a god,
aidan@highrise.ca                                       command like a king,
http://www.highrise.ca/                                   work like a slave.

On Monday 21 September 2009, Scott Marlowe <scott.marlowe@gmail.com> wrote:
> I'm looking at running session servers in ram.
> Does anybody any real world experience here or any words of sage
> advice before I go off and start testing this?

Use memcached for session data.

--
"No animals were harmed in the recording of this episode. We tried but that
damn monkey was just too fast."

Alan Hodgson wrote:
> On Monday 21 September 2009, Scott Marlowe <scott.marlowe@gmail.com> wrote:
>> I'm looking at running session servers in ram.
>
> Use memcached for session data.

IMHO postgres is more appropriate for some types of session data.

One of the apps I work on involves session data that consists of
geospatial data which we store and index in postgres/postgis.

Scott Marlowe wrote:
> I'm looking at running session servers in ram.
>  We're currently IO write bound with
> fsync=off using a 15k5 seagate SAS drive, so I'm hoping that moving
> the db into /dev/shm will help quite a bit here.

"a 15k5 seagate SAS drive"

Is this implying that you have "a" == one session server?  I
bet that it'd be cheaper to throw a bunch of cheap boxes
in there and make a pool of session servers rather than one
fast one.   When a new session is created, your application
code can then pick the least loaded session server and put
the session-server-number in a cookie.

This approach works fine for me - but I suspect I have many
fewer, yet probably much larger sessions going through the
system.




> Does anybody any real world experience here or any words of sage
> advice before I go off and start testing this?
>

On Tue, Sep 22, 2009 at 12:01 PM, Ron Mayer
<rm_pg@cheapcomplexdevices.com> wrote:
> Alan Hodgson wrote:
>> On Monday 21 September 2009, Scott Marlowe <scott.marlowe@gmail.com> wrote:
>>> I'm looking at running session servers in ram.
>>
>> Use memcached for session data.
>
> IMHO postgres is more appropriate for some types of session data.
>
> One of the apps I work on involves session data that consists of
> geospatial data which we store and index in postgres/postgis.
>
> Scott Marlowe wrote:
>> I'm looking at running session servers in ram.
>>  We're currently IO write bound with
>> fsync=off using a 15k5 seagate SAS drive, so I'm hoping that moving
>> the db into /dev/shm will help quite a bit here.
>
> "a 15k5 seagate SAS drive"
>
> Is this implying that you have "a" == one session server?  I
> bet that it'd be cheaper to throw a bunch of cheap boxes
> in there and make a pool of session servers rather than one
> fast one.   When a new session is created, your application
> code can then pick the least loaded session server and put
> the session-server-number in a cookie.

We already have two using modulus load balancing, and each is handling
up to 100,000 sessons each, and an average session object of 10k to
20k.  I'm just looking at how to keep from throwing more cheap boxes
at it, or having to put more drives in them.  We're mostly IO bound on
these machines, even with 100 checkpoint segments and a 30 minute
checkpoint timeout and a low completion target to reduce checkpointing
even more.

Even with a move to a ramdisk, I'm guessing with our increasing load
we're gonna need to double our session servers eventually.

As for memcached (mentioned in another post), I'm not sure if it's the
right fit for this or not.  We already use it to cache app data and it
works well enough, so it's worth testing for this as well I guess.

Thanks for all the input from everybody, I'll let you know how it works out.