Re: Storing files: 2.3TBytes, 17M file count - Mailing list pgsql-general

From Thomas Güttler
Subject Re: Storing files: 2.3TBytes, 17M file count
Date
Msg-id 8d88644c-27b8-3f60-4263-168fe1a3ab1d@thomas-guettler.de
Whole thread Raw
In response to Re: Storing files: 2.3TBytes, 17M file count  (Adrian Klaver <adrian.klaver@aklaver.com>)
Responses Re: Storing files: 2.3TBytes, 17M file count  (Adrian Klaver <adrian.klaver@aklaver.com>)
List pgsql-general

Am 28.11.2016 um 16:01 schrieb Adrian Klaver:
> On 11/28/2016 06:28 AM, Thomas Güttler wrote:
>> Hi,
>>
>> PostgreSQL is rock solid and one of the most reliable parts of our
>> toolchain.
>>
>>    Thank you
>>
>> Up to now, we don't store files in PostgreSQL.
>>
>> I was told, that you must not do this .... But this was 20 years ago.
>>
>>
>> I have 2.3TBytes of files. File count is 17M
>>
>> Up to now we use rsync (via rsnapshot) to backup our data.
>>
>> But it takes longer and longer for rsync to detect
>> the changes. Rsync checks many files. But daily only
>> very few files really change. More than 99.9% don't.
>
> Are you rsyncing over all the files at one time?

Yes, we rsyncing every night.

> Or do break it down into segments over the day?

No, up to now it is one rsync run.

> The closest I remember is Bacula:
>
> http://blog.bacula.org/documentation/documentation/
>
> It uses a hybrid solution where the files are stored on a file server and data about the files is stored in a
database.
> Postgres is one of the database backends it can work with.

I heard of Bacula, but I was not aware of the fact, that they can use postfres for the meta data.

>>
>> I have the hope, that it would be easier to backup only the files which
>> changed.
>
> Backup to where and how?
> Are you thinking of using replication?

No, replication is not the current issue. Plain old backup is my current issue.

Backup where and how? ... That's what this question is about :-)

--
Thomas Guettler http://www.thomas-guettler.de/


pgsql-general by date:

Previous
From: rob
Date:
Subject: Re: Invoice Table Design
Next
From: Thomas Güttler
Date:
Subject: We reached the limit of inotify. Was: Storing files: 2.3TBytes, 17M file count