Re: Streaming base backups - Mailing list pgsql-hackers

From Magnus Hagander
Subject Re: Streaming base backups
Date
Msg-id AANLkTim+ZMEqZOdV-YgZAvNPd-xxaRhFrAeQByLc2Azb@mail.gmail.com
Whole thread Raw
In response to Re: Streaming base backups  (Cédric Villemain <cedric.villemain.debian@gmail.com>)
Responses Re: Streaming base backups  (Cédric Villemain <cedric.villemain.debian@gmail.com>)
Re: Streaming base backups  (Cédric Villemain <cedric.villemain.debian@gmail.com>)
Re: Streaming base backups  (Garick Hamlin <ghamlin@isc.upenn.edu>)
Re: Streaming base backups  (Fujii Masao <masao.fujii@gmail.com>)
List pgsql-hackers
On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
<cedric.villemain.debian@gmail.com> wrote:
> 2011/1/7 Magnus Hagander <magnus@hagander.net>:
>> On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
>> <cedric.villemain.debian@gmail.com> wrote:
>>> 2011/1/5 Magnus Hagander <magnus@hagander.net>:
>>>> On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
>>>>> Magnus Hagander <magnus@hagander.net> writes:
>>>>>> * Stefan mentiond it might be useful to put some
>>>>>> posix_fadvise(POSIX_FADV_DONTNEED)
>>>>>>   in the process that streams all the files out. Seems useful, as long as that
>>>>>>   doesn't kick them out of the cache *completely*, for other backends as well.
>>>>>>   Do we know if that is the case?
>>>>>
>>>>> Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
>>>>> not already in SHM?
>>>>
>>>> I think that's way more complex than we want to go here.
>>>>
>>>
>>> DONTNEED will remove the block from OS buffer everytime.
>>
>> Then we definitely don't want to use it - because some other backend
>> might well want the file. Better leave it up to the standard logic in
>> the kernel.
>
> Looking at the patch, it is (very) easy to add the support for that in
> basebackup.c
> That supposed allowing mincore(), so mmap(), and so probably switch
> the fopen() to an open() (or add an open() just for mmap
> requirement...)
>
> Let's go ?

Per above, I still don't think we *should* do this. We don't want to
kick things out of the cache underneath other backends, and since we
can't control that. Either way, it shouldn't happen in the beginning,
and if it does, should be backed with proper benchmarks.

I've committed the backend side of this, without that. Still working
on the client, and on cleaning up Heikki's patch for grammar/parser
support.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


pgsql-hackers by date:

Previous
From: tv@fuzzy.cz
Date:
Subject: Re: estimating # of distinct values
Next
From: Magnus Hagander
Date:
Subject: Re: system views for walsender activity