Re: Something else about Redo Logs disappearing - Mailing list pgsql-general

Greetings,

* Peter (pmc@citylink.dinoex.sub.org) wrote:
> On Tue, Jun 09, 2020 at 03:42:48PM -0400, Stephen Frost wrote:
> ! > And then 90% of the things offered here become superfluous, because
> ! > they are already handled site-wide. And then you will have to
> ! > consider integration of both pieces - and that will most likely be
> ! > more work and more error-prone than just writing a few adapters in
> ! > shell.
> !
> ! pgbackrest's repo can be safely backed up using the simple file-based
> ! backup utilities that you're referring to here.  I suspect some of the
> ! other solution's backups also could be, but you'd probably want to make
> ! sure.
>
> What repo?? I seem to have missed that at first glance.

Yes, pgbackrest has a repo, like most other tools (though they call them
different things... pg_basebackup has one though it's not really
formal).

> Are You indeed suggesting that one should have their data within
> the database, where it is worked with, and then use Your tool
> to copy it to some "repo" disk playground whatever area, and then
> use their regular backup system to COPY IT AGAIN into their
> backup/archiving system? Are You kiddin'?

No, I'm not kidding and yes, that's what I'm suggesting.  You need a
consistent backup of your database that includes all the needed WAL to
perform a restore.

This is only one option though, there are others- you can also use
pgbackrest to push your backups to s3 (or any s3-compatible data storage
system, which includes some backup systems), and we'll be adding support
for Azure very shortly, and have plans to add GCS too in the future,
along with others probably.

> Is this becoming a madhouse, or are You going to refund them that?

I concur that this is becoming a madhouse, and is pushing past the limit
for what I'm willing to deal with when trying to assist someone.

> Let me tell You something: the people I used to work for, sometimes
> had a problem. They had some amount of data that was created during
> the day, and they had the night to write that data away to backup.
> That would usually mean, four or eight of the big tapes, streaming in
> parallel, fibers saturated, all night thru. And the problem usually was
> that they would need a longer night. At least the math had to be done
> properly.

Indeed, parallel backup is important, which is why pgbackrest supports
it, along with compression and encryption, all in-stream between the
database server and the repo, along with calculating a SHA to be stored
of every single file seen, allowing you to validate that the files
haven't changed since the backup was done when restoring.

> Maybe You never encountered these, but there are surroundings where
> there is no spare room for nonsense. Maybe that'S why these people
> preferred to use oracle.

I've both dealt with keeping tape drives fully loaded to avoid breaking
the tape by studdering it (and writing dedicated C code to deal with
exactly that), and dealt with backing up and restoring Oracle, including
with various "enterprise" backup technologies (with varying levels of
success...).  None of what is being brought up here is new, novel, or
even particularly interesting.

Thanks,

Stephen