Thread: Method to pass data between queries in a multi-statement transaction

On Wed, Apr 17, 2019 at 3:04 PM Souvik Bhattacherjee <kivuosb@gmail.com> wrote:

Hello,

I'm trying to pass some values between queries in a multi-statement transaction. For example, consider the following representative multi-statement transaction:

begin;

select * from table1 t1, table2 t2 where t1.cid = t2.cid;

delete from table1 where cid in (values-to-be-populated-from-the-previous-query);

commit;

There are few if any situations where you need to immediately and completely pass all values from one query to another in the same transaction where the queries cannot just be combined into a single statement.  Your representative example is one that is easily combined into a single statement.

Now, the stuff you are trying seems to indicate you are trying to do something in C, inside the engine itself, with all of this.  If that is the case you may want to be more clear as to what you are attempting to do.  But as far as server SQL goes the only persistence area are tables/relations - including temporary ones.

David J.

There are few if any situations where you need to immediately and completely pass all values from one query to another in the same transaction where the queries cannot just be combined into a single statement.  Your representative example is one that is easily combined into a single statement.

> What if I need the result of the join to be stored into table3 as well as the tuples that participated in the query to be deleted from table1. The following can be done without the need to transfer values from the previous query into the next:

begin;

insert into table3 (id, attr1, attr2) (select t1.cid, t1.empname, t2.dept from table1 t1, table2 t2 where t1.cid = t2.cid);

delete from table1 where cid in (select c.cid from table1 t1, table2 t2 where t1.cid = t2.cid);

commit;

However note that we have to perform the join twice, which is not efficient. Now to make things worse, increase the number of tables to join while imposing the requirement of tuple deletion to apply to all or to a subset of the tables that participate in join.

Now, the stuff you are trying seems to indicate you are trying to do something in C, inside the engine itself, with all of this.  If that is the case you may want to be more clear as to what you are attempting to do.  But as far as server SQL goes the only persistence area are tables/relations - including temporary ones.

> I'm trying to modify the engine here.

-SB

On Wed, Apr 17, 2019 at 6:16 PM David G. Johnston <david.g.johnston@gmail.com> wrote:

On Wed, Apr 17, 2019 at 3:04 PM Souvik Bhattacherjee <kivuosb@gmail.com> wrote:

Hello,

I'm trying to pass some values between queries in a multi-statement transaction. For example, consider the following representative multi-statement transaction:

begin;

select * from table1 t1, table2 t2 where t1.cid = t2.cid;

delete from table1 where cid in (values-to-be-populated-from-the-previous-query);

commit;

There are few if any situations where you need to immediately and completely pass all values from one query to another in the same transaction where the queries cannot just be combined into a single statement.  Your representative example is one that is easily combined into a single statement.

Now, the stuff you are trying seems to indicate you are trying to do something in C, inside the engine itself, with all of this.  If that is the case you may want to be more clear as to what you are attempting to do.  But as far as server SQL goes the only persistence area are tables/relations - including temporary ones.

David J.

On Wed, Apr 17, 2019 at 5:30 PM Souvik Bhattacherjee <kivuosb@gmail.com> wrote:

> What if I need the result of the join to be stored into table3 as well as the tuples that participated in the query to be deleted from table1. The following can be done without the need to transfer values from the previous query into the next:

begin;

insert into table3 (id, attr1, attr2) (select t1.cid, t1.empname, t2.dept from table1 t1, table2 t2 where t1.cid = t2.cid);

delete from table1 where cid in (select c.cid from table1 t1, table2 t2 where t1.cid = t2.cid);

commit;

You can use INSERT...RETURNING in a WITH query:

postgres=# create table foo (a integer);

CREATE TABLE

postgres=# create table bar (b integer);

CREATE TABLE

postgres=# insert into bar values (42);

INSERT 0 1

postgres=# with i as (insert into foo values (42) returning a) delete from bar where b = (select a from i);

DELETE 1

-Michel

On Wed, Apr 17, 2019 at 6:16 PM David G. Johnston <david.g.johnston@gmail.com> wrote:

On Wed, Apr 17, 2019 at 3:04 PM Souvik Bhattacherjee <kivuosb@gmail.com> wrote:

Hello,

I'm trying to pass some values between queries in a multi-statement transaction. For example, consider the following representative multi-statement transaction:

begin;

select * from table1 t1, table2 t2 where t1.cid = t2.cid;

delete from table1 where cid in (values-to-be-populated-from-the-previous-query);

commit;

There are few if any situations where you need to immediately and completely pass all values from one query to another in the same transaction where the queries cannot just be combined into a single statement.  Your representative example is one that is easily combined into a single statement.

Now, the stuff you are trying seems to indicate you are trying to do something in C, inside the engine itself, with all of this.  If that is the case you may want to be more clear as to what you are attempting to do.  But as far as server SQL goes the only persistence area are tables/relations - including temporary ones.

David J.

On 4/17/19 5:30 PM, Souvik Bhattacherjee wrote:
> There are few if any situations where you need to immediately and
> completely pass all values from one query to another in the same
> transaction where the queries cannot just be combined into a single
> statement.  Your representative example is one that is easily combined
> into a single statement.
>
>  > What if I need the result of the join to be stored into table3 as
> well as the tuples that participated in the query to be deleted from
> table1. The following can be done without the need to transfer values
> from the previous query into the next:
>
> begin;
> insert into table3 (id, attr1, attr2) (select t1.cid, t1.empname,
> t2.dept from table1 t1, table2 t2 where t1.cid = t2.cid);
> delete from table1 where cid in (select c.cid from table1 t1, table2 t2
> where t1.cid = t2.cid);

Well the DELETE is not going to work as c.cid will error as undefined.

> commit;
>
> However note that we have to perform the join twice, which is not
> efficient. Now to make things worse, increase the number of tables to
> join while imposing the requirement of tuple deletion to apply to all or
> to a subset of the tables that participate in join.

You might want to take a look at CTE's:

https://www.postgresql.org/docs/11/queries-with.html

>
> Now, the stuff you are trying seems to indicate you are trying to do
> something in C, inside the engine itself, with all of this.  If that is
> the case you may want to be more clear as to what you are attempting to
> do.  But as far as server SQL goes the only persistence area are
> tables/relations - including temporary ones.
>
>> I'm trying to modify the engine here.
>
> -SB
>
> On Wed, Apr 17, 2019 at 6:16 PM David G. Johnston
> <david.g.johnston@gmail.com <mailto:david.g.johnston@gmail.com>> wrote:
>
>
>     On Wed, Apr 17, 2019 at 3:04 PM Souvik Bhattacherjee
>     <kivuosb@gmail.com <mailto:kivuosb@gmail.com>> wrote:
>
>         Hello,
>
>         I'm trying to pass some values between queries in a
>         multi-statement transaction. For example, consider the following
>         representative multi-statement transaction:
>
>         begin;
>         select * from table1 t1, table2 t2 where t1.cid = t2.cid;
>         delete from table1 where cid in
>         (values-to-be-populated-from-the-previous-query);
>         commit;
>
>
>     There are few if any situations where you need to immediately and
>     completely pass all values from one query to another in the same
>     transaction where the queries cannot just be combined into a single
>     statement.  Your representative example is one that is easily
>     combined into a single statement.
>
>     Now, the stuff you are trying seems to indicate you are trying to do
>     something in C, inside the engine itself, with all of this.  If that
>     is the case you may want to be more clear as to what you are
>     attempting to do.  But as far as server SQL goes the only
>     persistence area are tables/relations - including temporary ones.
>
>     David J.
>


--
Adrian Klaver
adrian.klaver@aklaver.com

On Thu, Apr 18, 2019 at 9:03 AM Souvik Bhattacherjee <kivuosb@gmail.com> wrote:

Thanks Michel.

However this only works if a is an unique attribute in the table that would help us to identify tuples that participated in the join. Consider the following join:

insert into table3 (id, level, empname, salary) 

(select  t0.cid, t0.level, t1.empname, t2.salary from table0 t0, table1 t1, table2 t2 where t0.cid = t1.cid and t1.pid = t2.pid);

Now if I want to delete those tuples from table2 that satisfied the join condition, I need to execute the join again with additional attributes.

Or just add pid to table3...

Also note that based on query plan, i.e. whether table0 and table1 were joined first followed by table1 and table2, we have to execute one additional join to get the tuples in table2 that satisfied the join condition (t1.pid = t2.pid).

???

Getting that information while the query is executed may not be difficult. There are other use cases in my application that require me to transfer the data from one query to the next within a transaction.

There may be some that benefit to some degree but its likely that you can write the application and queries in such a way to avoid a hard requirement.

Thus, what I'm looking for here is way to store the information and then pass that information to the next query efficiently.

For example, is it possible to define a struct of my choice, private to the current transaction, that would store the data and then pass it around to the next query in the transaction without having to materialize that struct (or deal with concurrency issues as in the hash table approach mentioned earlier) .

How much development and maintenance effort are you willing to spend here to gain what is likely to amount to only a bit of efficiency?  Many things are possible if you are going to modify the server code but why add grief?

David J.