Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

From: Andres Freund
Subject: Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)
Date: ,
Msg-id: 201001191603.17013.andres@anarazel.de
(view: Whole thread, Raw)
In response to: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark)
List: pgsql-hackers

Tree view

Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
 Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
  Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
   Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
    Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
     Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
      Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
      Re: [PERFORM] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Smith, )
      Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Smith, )
     Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
     Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Tom Lane, )
      Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
       Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Tom Lane, )
        Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Robert Haas, )
         Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
          Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Robert Haas, )
           Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
           Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Tom Lane, )
            Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Robert Haas, )
             Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
              Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
              Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Robert Haas, )
               Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
                Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Smith, )
                 Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
                  Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Robert Haas, )
                   Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Smith, )
                    Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Tom Lane, )
                     Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Robert Haas, )
                      Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
                       Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
                        Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Alvaro Herrera, )
                         Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Robert Haas, )
                          Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
                          Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
                           Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
                          Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
                           Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Robert Haas, )
                            Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
                             Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
                              Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Tom Lane, )
                               Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
                                Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Tom Lane, )
                                 Re: Re: Faster CREATE DATABASE by delaying fsync  (Florian Weimer, )
                                  Re: Re: Faster CREATE DATABASE by delaying fsync  (Mark Mielke, )
                                   Re: Re: Faster CREATE DATABASE by delaying fsync  (Andres Freund, )
                                    Re: Re: Faster CREATE DATABASE by delaying fsync  (Mark Mielke, )
                              Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Robert Haas, )
                               Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
                               Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
       Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
       Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Robert Haas, )
     Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
    Re: [PERFORM] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
 Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Tom Lane, )
  Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
   Re: [PERFORM] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Smith, )
    Re: [PERFORM] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
     Re: [PERFORM] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Michael Clemmons, )
      Re: [PERFORM] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
       Re: [PERFORM] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Michael Clemmons, )
        Re: [PERFORM] Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
     Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
      Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
  Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
 Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Greg Stark, )
  Re: Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (Andres Freund, )
  Re: [PERFORM] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1ubuntu karmic slow createdb)  (Andres Freund, )
   Re: [PERFORM] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (, )
  Re: [PERFORM] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)  (, )

On Tuesday 19 January 2010 15:52:25 Greg Stark wrote:
> On Mon, Jan 18, 2010 at 4:35 PM, Greg Stark <> wrote:
> > Looking at this patch for the commitfest I have a few questions.
>
> So I've touched this patch up a bit:
>
> 1) moved the posix_fadvise call to a new fd.c function
> pg_fsync_start(fd,offset,nbytes) which initiates an fsync without
> waiting on it. Currently it's only implemented with
> posix_fadvise(DONT_NEED) but I want to look into using sync_file_range
> in the future -- it looks like this call might be good enough for our
> checkpoints.
>
> 2) advised each 64k chunk as we write it which should avoid poisoning
> the cache if you do a large create database on an active system.
>
> 3) added the promised but afaict missing fsync of the directory -- i
> think we should actually backpatch this.
Yes, that was a bit stupid from me - I added the fsync for directories which
get recursed into (by not checking if its a file) but not for the uppermost
level.
So all directories should get fsynced right now but the topmost one.

I will review the patch later when I finally will have some time off again...
~4h.

Thanks!

Andres


pgsql-hackers by date:

From: Stefan Kaltenbrunner
Date:
Subject: Re: MySQL-ism help patch for psql
From: Andrew Dunstan
Date:
Subject: Re: MySQL-ism help patch for psql