Re: Streaming base backups - Mailing list pgsql-hackers

From Dimitri Fontaine
Subject Re: Streaming base backups
Date
Msg-id m2sjx7gf5u.fsf@2ndQuadrant.fr
Whole thread Raw
In response to Re: Streaming base backups  (Magnus Hagander <magnus@hagander.net>)
Responses Re: Streaming base backups  (Heikki Linnakangas <heikki.linnakangas@enterprisedb.com>)
Re: Streaming base backups  (Magnus Hagander <magnus@hagander.net>)
Re: Streaming base backups  (Tatsuo Ishii <ishii@postgresql.org>)
List pgsql-hackers
Magnus Hagander <magnus@hagander.net> writes:
>> Compression in libpq would be a nice way to solve it, later.
>
> Yeah, I'm pretty much set on postponing that one.

+1, in case it was not clear for whoever's counting the votes :)

>> What about pg_streamrecv | gzip > …, which has the big advantage of
>
> That's part of what I meant with "easier and more useful".

Well…

> Right now though, pg_streamrecv will output one tar file for each
> tablespace, so you can't get it on stdout. But that can be changed of
> course. The easiest step 1 is to just use gzopen() from zlib on the
> files and use the same code as now :-)

Oh if integrating it is easier :)

>> Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
>> not already in SHM?
>
> I think that's way more complex than we want to go here.

Yeah.

>> Well, I would guess that if you're streaming the WAL files in parallel
>> while the base backup is taken, then you're able to have it all without
>> archiving setup, and the server could still recycling them.
>
> Yes, this was mostly for the use-case of "getting a single tarfile
> that you can actually use to restore from without needing the log
> archive at all".

It also allows for a simpler kick-start procedure for preparing a
standby, and allows to stop worrying too much about wal_keep_segments
and archive servers.

When do the standby launch its walreceiver? It would be extra-nice for
the base backup tool to optionally continue streaming WALs until the
standby starts doing it itself, so that wal_keep_segments is really
deprecated.  No idea how feasible that is, though.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support


pgsql-hackers by date:

Previous
From: Dimitri Fontaine
Date:
Subject: Re: making an unlogged table logged
Next
From: Robert Treat
Date:
Subject: Re: We need to log aborted autovacuums