Thread: Refectoring of receivelog.c

Refectoring of receivelog.c

From
Magnus Hagander
Date:
I was working on adding the tar streaming functionality we talked about at the developer meeting to pg_basebackup, and rapidly ran across the issue that Andres has been complaining about for a while. The code in receivelog.c just passes an insane number of parameters around. Adding or changing even a small thing ends up touching a huge number of places.

Here's an attempt to refactor the code to instead pass around a control structure. I think it's a definite win already now, and we can't just keep adding new functionality on top of the current one.

I'll proceed to work on the actual functionality I was working on to go on top of this separately, but would appreciate a review of this part independently. It's mostly mechanical, but there may definitely be mistakes - or thinkos in the whole idea...

--
Attachment

Re: Refectoring of receivelog.c

From
Craig Ringer
Date:
On 15 February 2016 at 04:48, Magnus Hagander <magnus@hagander.net> wrote:
I was working on adding the tar streaming functionality we talked about at the developer meeting to pg_basebackup, and rapidly ran across the issue that Andres has been complaining about for a while. The code in receivelog.c just passes an insane number of parameters around. Adding or changing even a small thing ends up touching a huge number of places.
 

Other than the lack of comments on the fields in StreamCtl to indicate their functions I think this looks good.

I didn't find any mistakes, but I do admit my eyes started glazing over after a bit.

I'd rather not have StreamCtl as a typedef of an anonymous struct if it's exposed in a header though. It should have a name that can be used in forward declarations etc.

After recently working with the XLogReader I can't emphasise enough how important *useful* comments on the fields in these sorts of structs are - the XLogReaderState has ReadRecPtr, readSegNo + readOff + readLen, currRecPtr AND latestPagePtr. Which actually do have comments, just not super helpful ones.

--
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

Re: Refectoring of receivelog.c

From
Magnus Hagander
Date:


On Mon, Feb 15, 2016 at 7:15 AM, Craig Ringer <craig@2ndquadrant.com> wrote:
On 15 February 2016 at 04:48, Magnus Hagander <magnus@hagander.net> wrote:
I was working on adding the tar streaming functionality we talked about at the developer meeting to pg_basebackup, and rapidly ran across the issue that Andres has been complaining about for a while. The code in receivelog.c just passes an insane number of parameters around. Adding or changing even a small thing ends up touching a huge number of places.
 

Other than the lack of comments on the fields in StreamCtl to indicate their functions I think this looks good.

I copied that lack of comments from the previous implementation :P

But yeah, I agree, it's probably a good idea to add them.

 
I didn't find any mistakes, but I do admit my eyes started glazing over after a bit.

I'd rather not have StreamCtl as a typedef of an anonymous struct if it's exposed in a header though. It should have a name that can be used in forward declarations etc.


It's not exactly uncommon with anonymous ones either in our code, but I see no problem adding that.

Thanks!


-- 

Re: Refectoring of receivelog.c

From
Daniel Gustafsson
Date:
> On 15 Feb 2016, at 14:46, Magnus Hagander <magnus@hagander.net> wrote:
>
> On Mon, Feb 15, 2016 at 7:15 AM, Craig Ringer <craig@2ndquadrant.com <mailto:craig@2ndquadrant.com>> wrote:
> On 15 February 2016 at 04:48, Magnus Hagander <magnus@hagander.net <mailto:magnus@hagander.net>> wrote:
> I was working on adding the tar streaming functionality we talked about at the developer meeting to pg_basebackup,
andrapidly ran across the issue that Andres has been complaining about for a while. The code in receivelog.c just
passesan insane number of parameters around. Adding or changing even a small thing ends up touching a huge number of
places.
>
> Other than the lack of comments on the fields in StreamCtl to indicate their functions I think this looks good.
>
> I copied that lack of comments from the previous implementation :P
>
> But yeah, I agree, it's probably a good idea to add them.
>
> I didn't find any mistakes, but I do admit my eyes started glazing over after a bit.
>
> I'd rather not have StreamCtl as a typedef of an anonymous struct if it's exposed in a header though. It should have
aname that can be used in forward declarations etc. 
>
> It's not exactly uncommon with anonymous ones either in our code, but I see no problem adding that.

Short review of this patch:

The patch applies, and builds, cleanly on top of master as of today.  No new
functionality is introduced and thus no new tests or doc patches etc are
applicable.

The main point of the patch is to improve readability and reduce the number of
parameters passed, goals which are reached.  However, I agree with Craig that
comments on the struct fields should be added to improve readability even
further.  The comment on ReceiveXlogStream() also now reads a bit strange since
it describes fields inside the StreamCtl without referencing StreamCtl at all.
For first time readers of the code it could perhaps be helpful with a brief
note that the discussed parameters are in StreamCtl to avoid potential
confusion.

Overall I think this patch is an improvement on the current code and consider
it ready for committer.

cheers ./daniel


Re: Refectoring of receivelog.c

From
Magnus Hagander
Date:


On Fri, Mar 11, 2016 at 9:40 AM, Daniel Gustafsson <daniel@yesql.se> wrote:
> On 15 Feb 2016, at 14:46, Magnus Hagander <magnus@hagander.net> wrote:
>
> On Mon, Feb 15, 2016 at 7:15 AM, Craig Ringer <craig@2ndquadrant.com <mailto:craig@2ndquadrant.com>> wrote:
> On 15 February 2016 at 04:48, Magnus Hagander <magnus@hagander.net <mailto:magnus@hagander.net>> wrote:
> I was working on adding the tar streaming functionality we talked about at the developer meeting to pg_basebackup, and rapidly ran across the issue that Andres has been complaining about for a while. The code in receivelog.c just passes an insane number of parameters around. Adding or changing even a small thing ends up touching a huge number of places.
>
> Other than the lack of comments on the fields in StreamCtl to indicate their functions I think this looks good.
>
> I copied that lack of comments from the previous implementation :P
>
> But yeah, I agree, it's probably a good idea to add them.
>
> I didn't find any mistakes, but I do admit my eyes started glazing over after a bit.
>
> I'd rather not have StreamCtl as a typedef of an anonymous struct if it's exposed in a header though. It should have a name that can be used in forward declarations etc.
>
> It's not exactly uncommon with anonymous ones either in our code, but I see no problem adding that.

Short review of this patch:

The patch applies, and builds, cleanly on top of master as of today.  No new
functionality is introduced and thus no new tests or doc patches etc are
applicable.

The main point of the patch is to improve readability and reduce the number of
parameters passed, goals which are reached.  However, I agree with Craig that
comments on the struct fields should be added to improve readability even
further.  The comment on ReceiveXlogStream() also now reads a bit strange since
it describes fields inside the StreamCtl without referencing StreamCtl at all.
For first time readers of the code it could perhaps be helpful with a brief
note that the discussed parameters are in StreamCtl to avoid potential
confusion.

Overall I think this patch is an improvement on the current code and consider
it ready for committer.


Pushed with updated comments and a named stsruct.

Thanks!

--

Re: Refectoring of receivelog.c

From
Tomas Vondra
Date:
On 03/11/2016 11:15 AM, Magnus Hagander wrote:
>
...
>
>
> Pushed with updated comments and a named stsruct.

Pretty sure this memset call in pg_basebackup.c is incorrect, as it 
passes parameters in the wrong order:
    MemSet(&stream, sizeof(stream), 0);

It seems benign, because we're setting all the fields explicitly, but 
gcc is nagging about it.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



Re: Refectoring of receivelog.c

From
Magnus Hagander
Date:


On Sun, Mar 13, 2016 at 12:15 AM, Tomas Vondra <tomas.vondra@2ndquadrant.com> wrote:
On 03/11/2016 11:15 AM, Magnus Hagander wrote:

...


Pushed with updated comments and a named stsruct.

Pretty sure this memset call in pg_basebackup.c is incorrect, as it passes parameters in the wrong order:

    MemSet(&stream, sizeof(stream), 0);

It seems benign, because we're setting all the fields explicitly, but gcc is nagging about it.

Indeed,that's backwards. Interestingly enough I thought I did a c&p between that and the one in pg_receivexlog.c, but clearly I did not. And my gcc did not nag about it.

Fixed, thanks for the pointer!


--