Configuring S3 Connectivity

pg_probackup3 supports S3 interface for storing backups. Backup data is transferred to and from S3 without saving it in intermediate locations thus eliminating the need of having a large temporary storage.

An example configuration with a remote agent and a cloud storage (S3) is shown in Figure 3.1, “pg_probackup3 setup with a remote agent and S3”.

Figure 3.1. pg_probackup3 setup with a remote agent and S3

pg_probackup3 setup with a remote agent and S3


In this figure, the following logical components are shown:

Backup server

A server where the main process of pg_probackup3 runs and where local backups are stored.

Database server

A server with a database instance that needs to be backed up.

Remote agent

A secondary pg_probackup3 process running on the database server. Only applicable to the remote mode.

Cloud storage

A cloud storage for backups.

Set up Access to S3 Storage

If you are going to use pg_probackup3 with S3 interface, complete the following steps:

  • Create a bucket with a unique and meaningful name in the S3 storage for you future backups.

  • Create ACCESS_KEY and SECRET_ACCESS_KEY tokens to be used for secure connectivity instead of your username and password.

  • For communication between pg_probackup3 and S3 server, set values of environment variables corresponding to your S3 server. For example:

    export PG_PROBACKUP_S3_HOST=127.0.0.1
    export PG_PROBACKUP_S3_PORT=9000
    export PG_PROBACKUP_S3_REGION=ru-msk
    export PG_PROBACKUP_S3_BUCKET_NAME=test1
    export PG_PROBACKUP_S3_ACCESS_KEY=admin
    export PG_PROBACKUP_S3_SECRET_ACCESS_KEY=password
    export PG_PROBACKUP_S3_HTTPS=ON
    

    Alternatively, you can provide S3 server settings in the S3 configuration file (see the --s3-config-file option in the section S3 Options for details).

    It makes sense to specify S3 server settings if --s3=minio, as described in the section S3 Options.

    The following environment variables can be specified:

    PG_PROBACKUP_S3_HOST

    Address or list of addresses of the S3 server. A list of one or several semicolon-delimited addresses. Do not add a semicolon after the last address in the list. Each address can include the port number, separated by a colon. If the port number is not specified, the value of PG_PROBACKUP_S3_PORT is assumed. Do not add a colon if the port number is not specified.

    For example:

    export PG_PROBACKUP_S3_PORT=80
    export PG_PROBACKUP_S3_HOST="127.0.0.1:9000;10.4.13.56:443;172.17.0.1"
    

    In this example, for the 127.0.0.1 address, the port 9000 is explicitly specified, for 10.4.13.56, the port 443 is specified, while for the 172.17.0.1 address, port 80, specified through PG_PROBACKUP_S3_PORT, will be used.

    If any of the specified addresses gets unavailable while pg_probackup3 is in operation, requests to the S3 storage are distribited between the rest of the specified addresses. That is, when several addresses are specified, pg_probackup3 performs load balancing of S3 requests.

    PG_PROBACKUP_S3_PORT

    The port of the S3 server.

    PG_PROBACKUP_S3_REGION

    The region of the S3 server.

    PG_PROBACKUP_S3_BUCKET_NAME

    The name of the bucket on the S3 server.

    PG_PROBACKUP_S3_ACCESS_KEY, PG_PROBACKUP_S3_SECRET_ACCESS_KEY

    Secure tokens on the S3 server.

    PG_PROBACKUP_S3_HTTPS

    The protocol to be used. Possible values:

    • ON or HTTPS — use HTTPS

    • Other than ON or HTTPS — use HTTP

    PG_PROBACKUP_S3_BUFFER_SIZE

    The size of the read/write buffer for communicating with S3, in MiB. The default is 16.

    PG_PROBACKUP_S3_RETRIES

    The maximum number of attempts to execute an S3 request in case of failures. The default is 3.

    PG_PROBACKUP_S3_TIMEOUT

    The maximum amount of time to execute an HTTP request to the S3 server, in seconds. The default is 300.

    PG_PROBACKUP_S3_IGNORE_CERT_VER

    Don't verify the certificate host and peer. The default is ON.

    PG_PROBACKUP_S3_CA_CERTIFICATE

    Specify the path to file with trust Certificate Authority (CA) bundle.

    PG_PROBACKUP_S3_CA_PATH

    Specify the directory with trust CA certificates.

    PG_PROBACKUP_S3_CLIENT_CERT

    Setup SSL client certificate.

    PG_PROBACKUP_S3_CLIENT_KEY

    Setup private key file for TLS and SSL client certificate.