Thread: de-deduplicate code in DML execution hooks in postgres_fdw
Hi, While working on a fix related to non-direct DML [1], I noticed that postgresExecForeignInsert(), postgresExecForeignUpdate() and postgresExecForeignDelete() functions are almost identical except that postgresExecForeignInsert() doesn't require ctid. The fix that I was working is applicable to Delete and Update but can be useful for Insert as well. I had to add the same code to two places at least and might have missed fixing one of them. Why don't we have a single function which prepares the statement, extract parameters, executes the prepared statement and checks for the results, returned rows etc? It's been a while that these functions are there and haven't produced code which is a lot different for each of these cases. Here's a patch to extract that code into a separate function and use it in all the three hook implementations. [1] https://www.postgresql.org/message-id/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS+OxcQo=aBDn1COywmcg@mail.gmail.com -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company
Attachment
(2018/04/18 19:34), Ashutosh Bapat wrote: > Hi, > While working on a fix related to non-direct DML [1], I noticed that > postgresExecForeignInsert(), postgresExecForeignUpdate() and > postgresExecForeignDelete() functions are almost identical except that > postgresExecForeignInsert() doesn't require ctid. The fix that I was > working is applicable to Delete and Update but can be useful for > Insert as well. I had to add the same code to two places at least and > might have missed fixing one of them. Why don't we have a single > function which prepares the statement, extract parameters, executes > the prepared statement and checks for the results, returned rows etc? > It's been a while that these functions are there and haven't produced > code which is a lot different for each of these cases. Here's a patch > to extract that code into a separate function and use it in all the > three hook implementations. > > [1] https://www.postgresql.org/message-id/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS+OxcQo=aBDn1COywmcg@mail.gmail.com +1 for the general idea. (Actually, I also thought the same thing before.) But since this is definitely a matter of PG12, ISTM that it's wise to work on this after addressing the issue in [1]. My concern is: if we do this refactoring now, we might need two patches for fixing the issue in case of backpatching as the fix might need to change those executor functions. Best regards, Etsuro Fujita
On Thu, Jul 19, 2018 at 2:05 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote: > (2018/04/18 19:34), Ashutosh Bapat wrote: >> >> Hi, >> While working on a fix related to non-direct DML [1], I noticed that >> postgresExecForeignInsert(), postgresExecForeignUpdate() and >> postgresExecForeignDelete() functions are almost identical except that >> postgresExecForeignInsert() doesn't require ctid. The fix that I was >> working is applicable to Delete and Update but can be useful for >> Insert as well. I had to add the same code to two places at least and >> might have missed fixing one of them. Why don't we have a single >> function which prepares the statement, extract parameters, executes >> the prepared statement and checks for the results, returned rows etc? >> It's been a while that these functions are there and haven't produced >> code which is a lot different for each of these cases. Here's a patch >> to extract that code into a separate function and use it in all the >> three hook implementations. >> >> [1] >> https://www.postgresql.org/message-id/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS+OxcQo=aBDn1COywmcg@mail.gmail.com > > > +1 for the general idea. (Actually, I also thought the same thing before.) > But since this is definitely a matter of PG12, ISTM that it's wise to work > on this after addressing the issue in [1]. My concern is: if we do this > refactoring now, we might need two patches for fixing the issue in case of > backpatching as the fix might need to change those executor functions. The only thing in [1] that would conflict with this patch is the 0002 (and possibly 0001) patch in [2]. We haven't yet decided anything about whether those patches can be back-patched or not. I think it's going to take much longer time for the actual solution to arise. But we don't have to wait for it to commit this patch as well as 0001 and 0002 patches in [2] because a. the larger solution is not likely to be back-patchable b. it's going to take much longer time. We don't have any estimate about how much longer it's going to take. We don't want this small piece to get stuck for that larger piece worst get stuck indefinitely. So, I will vote for getting it committed ASAP, if we agree that it's good change. If the solution to [1] happens to require this refactoring we we can back-patch that since back-patching it won't hurt. [1] https://www.postgresql.org/message-id/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS+OxcQo=aBDn1COywmcg@mail.gmail.com [2] https://www.postgresql.org/message-id/CAFjFpRfK69ptCTNChBBk%2BLYMXFzJ92SW6NmG4HLn_1y7xFk%3Dkw%40mail.gmail.com -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company
(2018/07/19 17:52), Ashutosh Bapat wrote: > On Thu, Jul 19, 2018 at 2:05 PM, Etsuro Fujita > <fujita.etsuro@lab.ntt.co.jp> wrote: >> +1 for the general idea. (Actually, I also thought the same thing before.) >> But since this is definitely a matter of PG12, ISTM that it's wise to work >> on this after addressing the issue in [1]. My concern is: if we do this >> refactoring now, we might need two patches for fixing the issue in case of >> backpatching as the fix might need to change those executor functions. > > The only thing in [1] that would conflict with this patch is the 0002 > (and possibly 0001) patch in [2]. We haven't yet decided anything > about whether those patches can be back-patched or not. I think it's > going to take much longer time for the actual solution to arise. But > we don't have to wait for it to commit this patch as well as 0001 and > 0002 patches in [2] I've just started catching up the discussions in [1], so I don't think I understand those fully, but it appears that we haven't yet reached a consensus on what to do for that issue. > because a. the larger solution is not likely to be > back-patchable b. it's going to take much longer time. We don't have > any estimate about how much longer it's going to take. I don't understand the solution yet, so I'll study about that. Best regards, Etsuro Fujita
On Thu, Jul 19, 2018 at 05:35:11PM +0900, Etsuro Fujita wrote: > +1 for the general idea. (Actually, I also thought the same thing before.) > But since this is definitely a matter of PG12, ISTM that it's wise to work > on this after addressing the issue in [1]. My concern is: if we do this > refactoring now, we might need two patches for fixing the issue in case of > backpatching as the fix might need to change those executor functions. FWIW, I would think that if some cleanup of the code is obvious, we should make it without waiting for the other issues to settle down because there is no way to know when those are done, and this patch could be forgotten. This indeed makes back-patching a bit harder but it also reduces the code chunk for HEAD with the extra fixes. Looking at the proposed patch, moving the new routine closer to execute_dml_stmt and renaming it execute_dml_single_row would be nicer. -- Michael
Attachment
(2018/07/20 13:49), Michael Paquier wrote: > On Thu, Jul 19, 2018 at 05:35:11PM +0900, Etsuro Fujita wrote: >> +1 for the general idea. (Actually, I also thought the same thing before.) >> But since this is definitely a matter of PG12, ISTM that it's wise to work >> on this after addressing the issue in [1]. My concern is: if we do this >> refactoring now, we might need two patches for fixing the issue in case of >> backpatching as the fix might need to change those executor functions. > > FWIW, I would think that if some cleanup of the code is obvious, we > should make it without waiting for the other issues to settle down > because there is no way to know when those are done, I would agree to that if we were late in the development cycle for PG12. > and this patch > could be forgotten. I won't forget this patch. :) > Looking at the proposed patch, moving the new routine closer to > execute_dml_stmt and renaming it execute_dml_single_row would be nicer. Or execute_parameterized_dml_stmt? Best regards, Etsuro Fujita
On Fri, Jul 20, 2018 at 2:27 PM, Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote: > (2018/07/20 13:49), Michael Paquier wrote: >> >> On Thu, Jul 19, 2018 at 05:35:11PM +0900, Etsuro Fujita wrote: >>> >>> +1 for the general idea. (Actually, I also thought the same thing >>> before.) >>> But since this is definitely a matter of PG12, ISTM that it's wise to >>> work >>> on this after addressing the issue in [1]. My concern is: if we do this >>> refactoring now, we might need two patches for fixing the issue in case >>> of >>> backpatching as the fix might need to change those executor functions. >> >> >> FWIW, I would think that if some cleanup of the code is obvious, we >> should make it without waiting for the other issues to settle down >> because there is no way to know when those are done, > > > I would agree to that if we were late in the development cycle for PG12. > >> and this patch >> could be forgotten. > > > I won't forget this patch. :) > >> Looking at the proposed patch, moving the new routine closer to >> execute_dml_stmt and renaming it execute_dml_single_row would be nicer. > > > Or execute_parameterized_dml_stmt? I have placed it closer to its caller, I think that's better than moving it closer to execute_dml_stmt, unless there is any other strong advantage. I used perform instead of execute since the later is usually associated with local operation. I added "foreign" in the name of the function to indicate that it's executed on foreign server. I am happy with "remote" as well. I don't think "one" and "single" make any difference. I don't like "parameterized" since that gets too tied to the method we are using rather than what's actually being done. In short I don't find any of the suggestions to be significantly better or worse than the name I have chosen. Said that, I am not wedded to any of those. A committer is free to choose anything s/he likes. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company
On Mon, Jul 23, 2018 at 02:17:38PM +0530, Ashutosh Bapat wrote: > I used perform instead of execute since the later is usually > associated with local operation. I added "foreign" in the name of the > function to indicate that it's executed on foreign server. I am happy > with "remote" as well. I don't think "one" and "single" make any > difference. I don't like "parameterized" since that gets too tied to > the method we are using rather than what's actually being done. In > short I don't find any of the suggestions to be significantly better > or worse than the name I have chosen. Said that, I am not wedded to > any of those. A committer is free to choose anything s/he likes. Fujita-san, you are registered as a reviewer of this patch. Are you planning to look at it soon? It looks useful to me to rework this code anyway to help with future maintenance, so in the worst case I could always look at it. For now I have moved it to the next commit fest. -- Michael
Attachment
(2018/10/01 19:42), Michael Paquier wrote: > On Mon, Jul 23, 2018 at 02:17:38PM +0530, Ashutosh Bapat wrote: >> I used perform instead of execute since the later is usually >> associated with local operation. I added "foreign" in the name of the >> function to indicate that it's executed on foreign server. I am happy >> with "remote" as well. I don't think "one" and "single" make any >> difference. I don't like "parameterized" since that gets too tied to >> the method we are using rather than what's actually being done. In >> short I don't find any of the suggestions to be significantly better >> or worse than the name I have chosen. Said that, I am not wedded to >> any of those. A committer is free to choose anything s/he likes. > > Fujita-san, you are registered as a reviewer of this patch. Are you > planning to look at it soon? Yeah, I'm planning to work on this immediately after fixing the issue [1], because it still seems to me wise to work on it after addressing that issue. (I'll post an updated version of the patch for that tomorrow.) > It looks useful to me to rework this code > anyway to help with future maintenance, Agreed. > so in the worst case I could > always look at it. For now I have moved it to the next commit fest. Thanks for taking care of this! Best regards, Etsuro Fujita
(2018/10/01 21:54), Etsuro Fujita wrote: > (2018/10/01 19:42), Michael Paquier wrote: >> On Mon, Jul 23, 2018 at 02:17:38PM +0530, Ashutosh Bapat wrote: >>> I used perform instead of execute since the later is usually >>> associated with local operation. I added "foreign" in the name of the >>> function to indicate that it's executed on foreign server. I am happy >>> with "remote" as well. I don't think "one" and "single" make any >>> difference. I don't like "parameterized" since that gets too tied to >>> the method we are using rather than what's actually being done. In >>> short I don't find any of the suggestions to be significantly better >>> or worse than the name I have chosen. Said that, I am not wedded to >>> any of those. A committer is free to choose anything s/he likes. >> >> Fujita-san, you are registered as a reviewer of this patch. Are you >> planning to look at it soon? > > Yeah, I'm planning to work on this immediately after fixing the issue > [1], because it still seems to me wise to work on it after addressing > that issue. (I'll post an updated version of the patch for that tomorrow.) Sorry, I forgot to add the pointer for [1]: https://www.postgresql.org/message-id/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%40mail.gmail.com Best regards, Etsuro Fujita
On Tue, Oct 02, 2018 at 01:50:46PM +0900, Etsuro Fujita wrote: > Sorry, I forgot to add the pointer for [1]: > > https://www.postgresql.org/message-id/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%40mail.gmail.com OK, thanks for your update! -- Michael
Attachment
> On Mon, Oct 1, 2018 at 2:55 PM Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp> wrote: > > (2018/10/01 19:42), Michael Paquier wrote: > > On Mon, Jul 23, 2018 at 02:17:38PM +0530, Ashutosh Bapat wrote: > > Fujita-san, you are registered as a reviewer of this patch. Are you > > planning to look at it soon? > > Yeah, I'm planning to work on this immediately after fixing the issue > [1], because it still seems to me wise to work on it after addressing > that issue. (I'll post an updated version of the patch for that tomorrow.) Yep, it would be nice to have a full review here. But as far as I see the issue you were talking about is still in progress, right? Also looks like someone needs to take over this rather small patch.
(2018/11/30 2:58), Dmitry Dolgov wrote: >> On Mon, Oct 1, 2018 at 2:55 PM Etsuro Fujita<fujita.etsuro@lab.ntt.co.jp> wrote: >> >> (2018/10/01 19:42), Michael Paquier wrote: >>> On Mon, Jul 23, 2018 at 02:17:38PM +0530, Ashutosh Bapat wrote: >>> Fujita-san, you are registered as a reviewer of this patch. Are you >>> planning to look at it soon? >> >> Yeah, I'm planning to work on this immediately after fixing the issue >> [1], because it still seems to me wise to work on it after addressing >> that issue. (I'll post an updated version of the patch for that tomorrow.) > > Yep, it would be nice to have a full review here. But as far as I see the issue > you were talking about is still in progress, right? That's right. > Also looks like someone > needs to take over this rather small patch. I changed my mind; I thought it would be better to work on that issue before this, but Tom raised concerns about the patch I proposed for that issue, and I think it'll take much longer time to address that, so I'd like to get this done first. One thing we would need to discuss more about this is the name of a new function added by the patch. IIRC, we have three options so far [1]: 1) perform_one_foreign_dml proposed by Ashutosh 2) execute_dml_single_row proposed by Michael 3) execute_parameterized_dml_stmt proposed by me I'll withdraw my proposal; I think #1 and #2 would describe the functionality better than #3. My vote would go to #2 or execute_dml_stmt_single_row, which moves the function much more closer to execute_dml_stmt. I'd like to hear the opinions of others. Thanks! Best regards, Etsuro Fujita [1] https://www.postgresql.org/message-id/CAFjFpRc%2BMdhD2W%3Dofc06n3NriqPCy6ztNBehXTc_g_y9%3DP5k1w%40mail.gmail.com
(2018/11/30 19:55), Etsuro Fujita wrote: > One thing we would need to discuss more about this is the name of a new > function added by the patch. IIRC, we have three options so far [1]: > > 1) perform_one_foreign_dml proposed by Ashutosh > 2) execute_dml_single_row proposed by Michael > 3) execute_parameterized_dml_stmt proposed by me > > I'll withdraw my proposal; I think #1 and #2 would describe the > functionality better than #3. My vote would go to #2 or > execute_dml_stmt_single_row, which moves the function much more closer > to execute_dml_stmt. I'd like to hear the opinions of others. On second thought I'd like to propose another option: execute_foreign_modify because I think this would match the existing helper functions for updating foreign tables in postgres_fdw.c better, such as create_foreign_modify, prepare_foreign_modify and finish_foreign_modify. I don't really think the function name should contain "one" or "single_row". Like the names of the calling APIs (ie, ExecForeignInsert, ExecForeignUpdate and ExecForeignDelete), I think it's OK to omit such words from the function name. Here is an updated version of the patch. In addition to the naming, I tweaked the function a little bit to match other functions (mainly variable names), moved it to the place where the helper functions are defined, fiddled with some comments, and removed an extra include file that the original patch added. Best regards, Etsuro Fujita
Attachment
(2019/01/07 20:26), Etsuro Fujita wrote: > (2018/11/30 19:55), Etsuro Fujita wrote: >> One thing we would need to discuss more about this is the name of a new >> function added by the patch. IIRC, we have three options so far [1]: >> >> 1) perform_one_foreign_dml proposed by Ashutosh >> 2) execute_dml_single_row proposed by Michael >> 3) execute_parameterized_dml_stmt proposed by me >> >> I'll withdraw my proposal; I think #1 and #2 would describe the >> functionality better than #3. My vote would go to #2 or >> execute_dml_stmt_single_row, which moves the function much more closer >> to execute_dml_stmt. I'd like to hear the opinions of others. > > On second thought I'd like to propose another option: > execute_foreign_modify because I think this would match the existing > helper functions for updating foreign tables in postgres_fdw.c better, > such as create_foreign_modify, prepare_foreign_modify and > finish_foreign_modify. I don't really think the function name should > contain "one" or "single_row". Like the names of the calling APIs (ie, > ExecForeignInsert, ExecForeignUpdate and ExecForeignDelete), I think > it's OK to omit such words from the function name. Here is an updated > version of the patch. In addition to the naming, I tweaked the function > a little bit to match other functions (mainly variable names), moved it > to the place where the helper functions are defined, fiddled with some > comments, and removed an extra include file that the original patch added. If there are no objections, I'll commit that version of the patch. Best regards, Etsuro Fujita
On Wed, Jan 16, 2019 at 02:59:15PM +0900, Etsuro Fujita wrote: > (2019/01/07 20:26), Etsuro Fujita wrote: >> On second thought I'd like to propose another option: >> execute_foreign_modify because I think this would match the existing >> helper functions for updating foreign tables in postgres_fdw.c better, >> such as create_foreign_modify, prepare_foreign_modify and >> finish_foreign_modify. I don't really think the function name should >> contain "one" or "single_row". Like the names of the calling APIs (ie, >> ExecForeignInsert, ExecForeignUpdate and ExecForeignDelete), I think >> it's OK to omit such words from the function name. Here is an updated >> version of the patch. In addition to the naming, I tweaked the function >> a little bit to match other functions (mainly variable names), moved it >> to the place where the helper functions are defined, fiddled with some >> comments, and removed an extra include file that the original patch added. > > If there are no objections, I'll commit that version of the patch. I think that you could use PgFdwModifyState for the second argument of execute_foreign_modify instead of ResultRelInfo. Except of that nit, it looks fine to me, thanks for taking care of it. -- Michael
Attachment
Michael-san, (2019/01/16 15:54), Michael Paquier wrote: > On Wed, Jan 16, 2019 at 02:59:15PM +0900, Etsuro Fujita wrote: >> (2019/01/07 20:26), Etsuro Fujita wrote: >>> On second thought I'd like to propose another option: >>> execute_foreign_modify because I think this would match the existing >>> helper functions for updating foreign tables in postgres_fdw.c better, >>> such as create_foreign_modify, prepare_foreign_modify and >>> finish_foreign_modify. I don't really think the function name should >>> contain "one" or "single_row". Like the names of the calling APIs (ie, >>> ExecForeignInsert, ExecForeignUpdate and ExecForeignDelete), I think >>> it's OK to omit such words from the function name. Here is an updated >>> version of the patch. In addition to the naming, I tweaked the function >>> a little bit to match other functions (mainly variable names), moved it >>> to the place where the helper functions are defined, fiddled with some >>> comments, and removed an extra include file that the original patch added. >> >> If there are no objections, I'll commit that version of the patch. > > I think that you could use PgFdwModifyState for the second argument of > execute_foreign_modify instead of ResultRelInfo. Yeah, that is another option, but my favorite would be to use ResultRelInfo, as in the original patch by Ashutosh, because that makes it possible to write postgresExecForeignInsert, postgresExecForeignUpdate, and postgresExecForeignDelete as a single line. > Except of that nit, > it looks fine to me, thanks for taking care of it. Great! Thanks for the review! Best regards, Etsuro Fujita
(2019/01/16 20:30), Etsuro Fujita wrote: > (2019/01/16 15:54), Michael Paquier wrote: >> On Wed, Jan 16, 2019 at 02:59:15PM +0900, Etsuro Fujita wrote: >>> If there are no objections, I'll commit that version of the patch. >> >> I think that you could use PgFdwModifyState for the second argument of >> execute_foreign_modify instead of ResultRelInfo. > > Yeah, that is another option, but my favorite would be to use > ResultRelInfo, as in the original patch by Ashutosh, because that makes > it possible to write postgresExecForeignInsert, > postgresExecForeignUpdate, and postgresExecForeignDelete as a single line. > >> Except of that nit, >> it looks fine to me, thanks for taking care of it. > > Great! Thanks for the review! Pushed. I left that argument alone. I think we can change it later, if necessary :). Best regards, Etsuro Fujita
On Thu, Jan 17, 2019 at 02:44:25PM +0900, Etsuro Fujita wrote: > Pushed. I left that argument alone. I think we can change it later, if > necessary :). Sure, that's fine as well. Thanks for committing the patch. -- Michael