Re: WIP/PoC for parallel backup - Mailing list pgsql-hackers

From Jeevan Chalke
Subject Re: WIP/PoC for parallel backup
Date
Msg-id CAM2+6=ViN95PNoJc_8Q3tK=fFvQeZ5HQU1+T=RLkp9qAq6kCRA@mail.gmail.com
Whole thread Raw
In response to Re: WIP/PoC for parallel backup  (Asif Rehman <asifr.rehman@gmail.com>)
Responses Re: WIP/PoC for parallel backup  (Asif Rehman <asifr.rehman@gmail.com>)
List pgsql-hackers


On Tue, Apr 7, 2020 at 10:14 PM Asif Rehman <asifr.rehman@gmail.com> wrote:
Hi,

Thanks, Kashif and Rajkumar. I have fixed the reported issues.

I have added the shared state as previously described. The new grammar changes
are as follows:

START_BACKUP [LABEL '<label>'] [FAST] [MAX_RATE %d]
    - This will generate a unique backupid using pg_strong_random(16) and hex-encoded
      it. which is then returned as the result set.
    - It will also create a shared state and add it to the hashtable. The hash table size is set
      to BACKUP_HASH_SIZE=10, but since hashtable can expand dynamically, I think it's
      sufficient initial size. max_wal_senders is not used, because it can be set to quite a 
      large values.

JOIN_BACKUP 'backup_id'
    - finds 'backup_id' in hashtable and attaches it to server process.


SEND_FILE '(' 'FILE' ')' [NOVERIFY_CHECKSUMS]
    - renamed SEND_FILES to SEND_FILE
    - removed START_WAL_LOCATION from this because 'startptr' is now accessible through
      shared state.

There is no change in other commands:
STOP_BACKUP [NOWAIT]
LIST_TABLESPACES [PROGRESS]
LIST_FILES [TABLESPACE]
LIST_WAL_FILES [START_WAL_LOCATION 'X/X'] [END_WAL_LOCATION 'X/X']

The current patches (v11) have been rebased to the latest master. The backup manifest is enabled
by default, so I have disabled it for parallel backup mode and have generated a warning so that
user is aware of it and not expect it in the backup.

So, are you working on to make it work? I don't think a parallel backup feature should be creating a backup with no manifest.





--
--
Asif Rehman
Highgo Software (Canada/China/Pakistan)
URL : www.highgo.ca



--
Jeevan Chalke
Associate Database Architect & Team Lead, Product Development
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

pgsql-hackers by date:

Previous
From: Fujii Masao
Date:
Subject: Re: Don't try fetching future segment of a TLI.
Next
From: David Steele
Date:
Subject: Re: pgsql: Generate backup manifests for base backups, and validatethem.