Re: Streaming base backups - Mailing list pgsql-hackers
From | Cédric Villemain |
---|---|
Subject | Re: Streaming base backups |
Date | |
Msg-id | AANLkTimgndx85Kg-Ksr45jyBf1XtUaHFb=MqC6uU0tQ=@mail.gmail.com Whole thread Raw |
In response to | Re: Streaming base backups (Garick Hamlin <ghamlin@isc.upenn.edu>) |
List | pgsql-hackers |
2011/1/11 Garick Hamlin <ghamlin@isc.upenn.edu>: > On Tue, Jan 11, 2011 at 11:39:20AM -0500, Cédric Villemain wrote: >> 2011/1/11 Garick Hamlin <ghamlin@isc.upenn.edu>: >> > On Mon, Jan 10, 2011 at 09:09:28AM -0500, Magnus Hagander wrote: >> >> On Sun, Jan 9, 2011 at 23:33, Cédric Villemain >> >> <cedric.villemain.debian@gmail.com> wrote: >> >> > 2011/1/7 Magnus Hagander <magnus@hagander.net>: >> >> >> On Fri, Jan 7, 2011 at 01:47, Cédric Villemain >> >> >> <cedric.villemain.debian@gmail.com> wrote: >> >> >>> 2011/1/5 Magnus Hagander <magnus@hagander.net>: >> >> >>>> On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote: >> >> >>>>> Magnus Hagander <magnus@hagander.net> writes: >> >> >>>>>> * Stefan mentiond it might be useful to put some >> >> >>>>>> posix_fadvise(POSIX_FADV_DONTNEED) >> >> >>>>>> in the process that streams all the files out. Seems useful, as long as that >> >> >>>>>> doesn't kick them out of the cache *completely*, for other backends as well. >> >> >>>>>> Do we know if that is the case? >> >> >>>>> >> >> >>>>> Maybe have a look at pgfincore to only tag DONTNEED for blocks that are >> >> >>>>> not already in SHM? >> >> >>>> >> >> >>>> I think that's way more complex than we want to go here. >> >> >>>> >> >> >>> >> >> >>> DONTNEED will remove the block from OS buffer everytime. >> >> >> >> >> >> Then we definitely don't want to use it - because some other backend >> >> >> might well want the file. Better leave it up to the standard logic in >> >> >> the kernel. >> >> > >> >> > Looking at the patch, it is (very) easy to add the support for that in >> >> > basebackup.c >> >> > That supposed allowing mincore(), so mmap(), and so probably switch >> >> > the fopen() to an open() (or add an open() just for mmap >> >> > requirement...) >> >> > >> >> > Let's go ? >> >> >> >> Per above, I still don't think we *should* do this. We don't want to >> >> kick things out of the cache underneath other backends, and since we >> >> can't control that. Either way, it shouldn't happen in the beginning, >> >> and if it does, should be backed with proper benchmarks. >> > >> > Another option that occurs to me is an option to use direct IO (or another >> > means as needed) to bypass the cache. So rather than kicking it out of >> > the cache, we attempt just not to pollute the cache by bypassing it for cold >> > pages and use either normal io for 'hot pages', or use a 'read()' to "heat" >> > the cache afterward. >> >> AFAIR, even Linus is rejecting the idea to use it seriously, except if >> I shuffle in my memory. > > Direct IO is generally a pain. > > POSIX_FADV_NOREUSE is an alternative (I think). Realistically I wasn't sure which > way(s) actually worked. My gut was that direct io would likely work right on Linux > and Solaris, at least. If POSIX_FADV_NOREUSE works than maybe that is the answer > instead, but I haven't tested either. yes it should be the best option, unfortunely it is a ghost flag, it doesn't do anythig. At some point there were a libprefetch library and a linux fincore() syscall in the air. Unfortunely actors of those items stop communication with open source afais. (I didn't get answers myself, neither linux ML get ones.) > > Garick > > >> >> > >> > Garick >> > >> >> >> >> I've committed the backend side of this, without that. Still working >> >> on the client, and on cleaning up Heikki's patch for grammar/parser >> >> support. >> >> >> >> -- >> >> Magnus Hagander >> >> Me: http://www.hagander.net/ >> >> Work: http://www.redpill-linpro.com/ >> >> >> >> -- >> >> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) >> >> To make changes to your subscription: >> >> http://www.postgresql.org/mailpref/pgsql-hackers >> > >> >> >> >> -- >> Cédric Villemain 2ndQuadrant >> http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support > -- Cédric Villemain 2ndQuadrant http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support
pgsql-hackers by date: