Re: WIP/PoC for parallel backup - Mailing list pgsql-hackers

From Asif Rehman
Subject Re: WIP/PoC for parallel backup
Date
Msg-id CADM=JeiCJ9bDroduk=VAn4WBNCxrnahogMvyx7jfjhc=6PqPrQ@mail.gmail.com
Whole thread Raw
In response to Re: WIP/PoC for parallel backup  (Jeevan Chalke <jeevan.chalke@enterprisedb.com>)
Responses Re: WIP/PoC for parallel backup  (Robert Haas <robertmhaas@gmail.com>)
List pgsql-hackers


On Tue, Apr 7, 2020 at 10:03 PM Jeevan Chalke <jeevan.chalke@enterprisedb.com> wrote:


On Tue, Apr 7, 2020 at 10:14 PM Asif Rehman <asifr.rehman@gmail.com> wrote:
Hi,

Thanks, Kashif and Rajkumar. I have fixed the reported issues.

I have added the shared state as previously described. The new grammar changes
are as follows:

START_BACKUP [LABEL '<label>'] [FAST] [MAX_RATE %d]
    - This will generate a unique backupid using pg_strong_random(16) and hex-encoded
      it. which is then returned as the result set.
    - It will also create a shared state and add it to the hashtable. The hash table size is set
      to BACKUP_HASH_SIZE=10, but since hashtable can expand dynamically, I think it's
      sufficient initial size. max_wal_senders is not used, because it can be set to quite a 
      large values.

JOIN_BACKUP 'backup_id'
    - finds 'backup_id' in hashtable and attaches it to server process.


SEND_FILE '(' 'FILE' ')' [NOVERIFY_CHECKSUMS]
    - renamed SEND_FILES to SEND_FILE
    - removed START_WAL_LOCATION from this because 'startptr' is now accessible through
      shared state.

There is no change in other commands:
STOP_BACKUP [NOWAIT]
LIST_TABLESPACES [PROGRESS]
LIST_FILES [TABLESPACE]
LIST_WAL_FILES [START_WAL_LOCATION 'X/X'] [END_WAL_LOCATION 'X/X']

The current patches (v11) have been rebased to the latest master. The backup manifest is enabled
by default, so I have disabled it for parallel backup mode and have generated a warning so that
user is aware of it and not expect it in the backup.

So, are you working on to make it work? I don't think a parallel backup feature should be creating a backup with no manifest.

I will, however parallel backup is already quite a large patch. So I think we should first
agree on the current work before adding a backup manifest and progress-reporting support.


--
Asif Rehman
Highgo Software (Canada/China/Pakistan)
URL : www.highgo.ca

pgsql-hackers by date:

Previous
From: Etsuro Fujita
Date:
Subject: Re: [HACKERS] advanced partition matching algorithm forpartition-wise join
Next
From: Tom Lane
Date:
Subject: Re: proposal \gcsv