Thread: WAL-G shipping to the cloud

WAL-G shipping to the cloud

From

aslı cokay

Date:

19 March 2021, 00:18:30

Hi all,

I'd like to get postgres log files to the cloud but i want this process going live i want to automate test process and after that i want it go live.

What i have in my mind is setting archive_mode is on and archiving_command with a script. After that i want to execute pgbench to get more WAL files and trigger with pg_switch_wal() to change WAL files.

Is there any idea do you have about it or is there anyone shipped their log to the cloud to give me an idea?

Thanks.

Re: WAL-G shipping to the cloud

From

Adrian Klaver

Date:

19 March 2021, 00:21:24

On 3/18/21 2:18 PM, aslı cokay wrote:
> Hi all,
> 
> I'd like to get postgres log files to the cloud but i want this process 
> going live i want to automate test process and after that i want it go 
> live.
> 
> What i have in my mind is setting archive_mode is on and 
> archiving_command with a script. After that i want to execute pgbench to 
> get more WAL files and trigger with pg_switch_wal() to change WAL files.
> 
> Is there any idea do you have about it or is there anyone shipped their 
> log to the cloud to give me an idea?

https://github.com/wal-e/wal-e

> 
> Thanks.
> 


-- 
Adrian Klaver
adrian.klaver@aklaver.com

Re: WAL-G shipping to the cloud

From

Bruce Momjian

Date:

19 March 2021, 00:24:47

On Thu, Mar 18, 2021 at 05:18:30PM -0400, aslı cokay wrote:
> Hi all,
> 
> I'd like to get postgres log files to the cloud but i want this process going
> live i want to automate test process and after that i want it go live. 
> 
> What i have in my mind is setting archive_mode is on and archiving_command with
> a script. After that i want to execute pgbench to get more WAL files and
> trigger with pg_switch_wal() to change WAL files.
> 
> Is there any idea do you have about it or is there anyone shipped their log to
> the cloud to give me an idea? 

Well, there is Wal-E and Wal-G, which archive to the cloud. 
pg_backreset also supports cloud WAL storage.

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  If only the physical world exists, free will is an illusion.

Re: WAL-G shipping to the cloud

From

asli cokay

Date:

19 March 2021, 02:30:32

Thank you, Bruce. Actually I mean I am using WAL-G what I am curious about is that I want to test shipping process automate, and I am looking for the ideas. But thanks for your help.

Bruce Momjian <bruce@momjian.us>, 18 Mar 2021 Per, 17:24 tarihinde şunu yazdı:

On Thu, Mar 18, 2021 at 05:18:30PM -0400, aslı cokay wrote:
> Hi all,
>
> I'd like to get postgres log files to the cloud but i want this process going
> live i want to automate test process and after that i want it go live.
>
> What i have in my mind is setting archive_mode is on and archiving_command with
> a script. After that i want to execute pgbench to get more WAL files and
> trigger with pg_switch_wal() to change WAL files.
>
> Is there any idea do you have about it or is there anyone shipped their log to
> the cloud to give me an idea?

Well, there is Wal-E and Wal-G, which archive to the cloud.
pg_backreset also supports cloud WAL storage.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

If only the physical world exists, free will is an illusion.

Re: WAL-G shipping to the cloud

From

Adalberto Caccia

Date:

19 March 2021, 12:31:44

Hi,

I'm using Wal-G like this, in pg12:

archive_mode = always
archive_command = 'wal-g wal-push %p'
archive_timeout = 60
restore_command = 'wal-g wal-fetch %f %p'

The only change for pg10 is last line (restore_command) has to go into a separate file, recovery.conf.

Please note, however, I'm using this in Kubernetes, so the relevant docker image has already got all the relevant ENV config for Wall-G loaded beforehand.

But Wal-G is really straightforward to configure, however.

Adalberto

Il giorno ven 19 mar 2021 alle ore 00:30 asli cokay <aslicokay@gmail.com> ha scritto:

Thank you, Bruce. Actually I mean I am using WAL-G what I am curious about is that I want to test shipping process automate, and I am looking for the ideas. But thanks for your help.

Bruce Momjian <bruce@momjian.us>, 18 Mar 2021 Per, 17:24 tarihinde şunu yazdı:
On Thu, Mar 18, 2021 at 05:18:30PM -0400, aslı cokay wrote:
> Hi all,
>
> I'd like to get postgres log files to the cloud but i want this process going
> live i want to automate test process and after that i want it go live.
>
> What i have in my mind is setting archive_mode is on and archiving_command with
> a script. After that i want to execute pgbench to get more WAL files and
> trigger with pg_switch_wal() to change WAL files.
>
> Is there any idea do you have about it or is there anyone shipped their log to
> the cloud to give me an idea?

Well, there is Wal-E and Wal-G, which archive to the cloud.
pg_backreset also supports cloud WAL storage.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

If only the physical world exists, free will is an illusion.

Re: WAL-G shipping to the cloud

From

asli cokay

Date:

19 March 2021, 18:03:57

Thanks for replying. I am just curious about that why are you using archive_mode = always? In this situation WAL-G trying to load same file at the same place i think.

Thanks.

19 Mart 2021 Cuma tarihinde Adalberto Caccia <adacaccia@gmail.com> yazdı:

Hi,
I'm using Wal-G like this, in pg12:

archive_mode = always
archive_command = 'wal-g wal-push %p'
archive_timeout = 60
restore_command = 'wal-g wal-fetch %f %p'

The only change for pg10 is last line (restore_command) has to go into a separate file, recovery.conf.

Please note, however, I'm using this in Kubernetes, so the relevant docker image has already got all the relevant ENV config for Wall-G loaded beforehand.
But Wal-G is really straightforward to configure, however.

Adalberto

Il giorno ven 19 mar 2021 alle ore 00:30 asli cokay <aslicokay@gmail.com> ha scritto:
Thank you, Bruce. Actually I mean I am using WAL-G what I am curious about is that I want to test shipping process automate, and I am looking for the ideas. But thanks for your help.

Bruce Momjian <bruce@momjian.us>, 18 Mar 2021 Per, 17:24 tarihinde şunu yazdı:
On Thu, Mar 18, 2021 at 05:18:30PM -0400, aslı cokay wrote:
> Hi all,
>
> I'd like to get postgres log files to the cloud but i want this process going
> live i want to automate test process and after that i want it go live.
>
> What i have in my mind is setting archive_mode is on and archiving_command with
> a script. After that i want to execute pgbench to get more WAL files and
> trigger with pg_switch_wal() to change WAL files.
>
> Is there any idea do you have about it or is there anyone shipped their log to
> the cloud to give me an idea?

Well, there is Wal-E and Wal-G, which archive to the cloud.
pg_backreset also supports cloud WAL storage.

--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com

If only the physical world exists, free will is an illusion.