Thread: Re: Where Can I Find The Code Segment For WAL Control?

 

I am now trying to develop the PG to support real-time backup.My
architecture is somehow similar to the "Database Mirroring" technology of
SQL Server 2005.The server end of the system is consisted of two DB servers
one is the Principal server,the other is the Mirror server.whenever the
Principal flushes its log buffer to the local log file,it must send the
content of its buffer to the Mirror simultaneously.After the mirror receives
the buffer,it write to its own log file and send a response message to the
Principal,then the Mirror redo/undo its log.By this way, we can guarantee
the database instances of the two servers identical. 

But now I encounter a problem.I don't know where the functions to control
log buffer are.Which code segment may I refer to? 

I have just participated the PG project for a short time,and I will
appreciate your help very much! 

Look forward to you all! 


Charlie Wang 

Charlie,

I'm currently working on a similar solution (it's true I'm only in the
thinking phase). I don't have too much time to spend on it (~few hours
per week, on the train during commuting), so it's not progressing too
fast...

Nevertheless, I would like to do a first proof-of-concept version in a
contrib module with minimal coding involved. I don't have time right now
to explain all the details how I would like to do it, but in essence I
want to:
- create a few functions which expose the WAL files and the data base
files as streams;
- use the archive_command framework to control when the WAL files can
be recycled, in combination with a WAL subscription system to make sure
the standby won't be missing WAL files (with safeguards so the primary
system will not go out of WAL file system space if the standby can't
keep up with the WAL stream);
- create a standby manager program which only needs to know how to
access the primary server in order to create the standby (by connecting
to it through normal data base connections and using the above mentioned
functions to stream the files);

These are just the main ideas, the devil is in the details.

As per answering your question, a possible starting point would be to
take a look at:

src/backend/postmaster/pg_arch.c

This one deals with archiving WAL files, so it could give you some
insight in how you can deal with the WAL files.

Or just search the source tree for 'WAL' ... I guess there's quite a few
places in the code where WAL is involved.

Next week I'll post more detailed design (some charts I did to better
understand the idea, and some more text to add details), if anybody is
interested in co-working on it, otherwise I'll just keep working in my
own pace until it gets done...

Cheers,
Csaba.


On Fri, 2006-03-10 at 07:34, 王宝兵 wrote:
>
> I am now trying to develop the PG to support real-time backup.My
> architecture is somehow similar to the "Database Mirroring" technology of
> SQL Server 2005.The server end of the system is consisted of two DB servers
> one is the Principal server,the other is the Mirror server.whenever the
> Principal flushes its log buffer to the local log file,it must send the
> content of its buffer to the Mirror simultaneously.After the mirror receives
> the buffer,it write to its own log file and send a response message to the
> Principal,then the Mirror redo/undo its log.By this way, we can guarantee
> the database instances of the two servers identical.
>
> But now I encounter a problem.I don't know where the functions to control
> log buffer are.Which code segment may I refer to?
>
> I have just participated the PG project for a short time,and I will
> appreciate your help very much!
>
> Look forward to you all!
>
>
>
> Charlie Wang
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster

[Please use "reply to all" so the list is CC-d]

Charlie,

I guess what you're after is to make sure the WAL buffers are shipped to
the stand-by at the same time as they are committed to disk. In any
other case your desire to have the stand-by EXACTLY in sync with the
primary server will not gonna work.

But that would mean that the communication to the stand-by will become a
performance bottleneck, as all transactions are only finished after the
WAL records for them are synced to the disk. So if you want your
stand-by completely in sync with your primary, you will want that the
transactions finish only after their WAL records are pushed to the
stand-by too... and then if the communication to the stand-by fails, all
your transactions will wait after it, possibly causing the primary to
stop working properly. So now you have another point of failure, and
instead of making the setup safer, you make it unsafer. What I want to
say is that it is likely not feasible to keep the stand-by completely in
sync.

In practice it is enough to keep the standby NEARLY in sync with the
primary server. That means you will ship the WAL records asynchronously,
i.e. after they are written to the disk, and in a separate thread.

What I'm after is to have a thread which starts streaming the current
WAL file, and keeps streaming it as it grows. I'm not completely sure
how I'll implement that, but I guess it will need to do a loop and
transfer whatever records are available, and then sleep a few seconds if
it reaches the end. It must be prepared to stumble upon partially
written WAL records, and sleep on those too. On the stand-by end, the
current partial WAL will not be used unless the stand-by is fired up...

So I'm after a solution which makes sure the stand-by is as up to date
as possible, with a few seconds allowed gap in normal operation, and
possibly more if the communication channel has bandwidth problems and
the server is very busy. Usually if the server crashes, than there are
worse problems than the few seconds/minutes worth of lost transactions.
To name one, if the server crashes you will have for sure at least a few
minutes of downtime. At least for our application, downtime in a busy
period is actually worse than the lost data (that we can recover from
other logs)...

Cheers,
Csaba.

On Sun, 2006-03-12 at 02:50, 王宝兵 wrote:
>
> Csaba:
>
>
>
> Firstly I must thank you for your help.Some of our designs are identical
> except the following:
>
>
>
> - create a standby manager program which only needs to know how to
>
> Access the primary server in order to create the standby (by connecting
>
> To it through normal data base connections and using the above mentioned
>
> Functions to stream the files);
>
>
>
> In my opinion,if we create a standby manager program and run it as a daemon
> process,it will check the state of the WAL files of the Principal every few
> seconds.But there is a risk for data lost.For an instance,if the Principal
> has flushed its log buffer to the disk and the dirty data are also flushed
> immediately,but the standby manager program is running in its interval.Then
> the Principal fails.In this situation,the Principal has updated its database
> but the log segment hasn't been sent to the Mirror,because the time point
> for the standby manager program to check the WAL files hasn't come.And then
> these data are lost.
>
>
>
> I think this situation will happen very probably in a big cooperation and it
> s very serious.
>
>
>
> Perhaps I have misunderstood your opinion.If that,I apologize.
>
>
>
> Charlie Wang
>
>
>
>
>
>
>
>
>
>
>
>