Home > mailing lists

Thread: [patch] Imporve pqmq

[patch] Imporve pqmq

From

Xiaoran Wang

Date:

08 August, 06:23:44

Hi,

When I use the 'pqmq' recently, I found some issues, just fix them.

Allow the param 'dsm_segment *seg' to be NULL in function

'pq_redirect_to_shm_mq'. As sometimes the shm_mq is created
in shared memory instead of DSM.

Add function 'pq_leave_shm_mq' to allow the process to go
back to the previous pq environment.

Attachment

0001-Improve-pqmq.patch

Re: [patch] Imporve pqmq

From

Robert Haas

Date:

08 August, 22:24:11

On Wed, Aug 7, 2024 at 11:24 PM Xiaoran Wang <fanfuxiaoran@gmail.com> wrote:
>     When I use the 'pqmq' recently, I found some issues, just fix them.
>
>      Allow the param 'dsm_segment *seg' to be NULL in function
>     'pq_redirect_to_shm_mq'. As sometimes the shm_mq is created
>     in shared memory instead of DSM.

Under what circumstances does this happen?

>     Add function 'pq_leave_shm_mq' to allow the process to go
>     back to the previous pq environment.

In the code as it currently exists, a parallel worker never has a
connected client, and it talks to a shm_mq instead. So there's no need
for this. If a backend needs to communicate with both a connected
client and also a shm_mq, it probably should not use pqmq but rather
decide explicitly which messages should be sent to the client and
which to the shm_mq. Otherwise, it seems hard to avoid possible loss
of protocol sync.

--
Robert Haas
EDB: http://www.enterprisedb.com

Re: [patch] Imporve pqmq

From

Xiaoran Wang

Date:

09 August, 05:27:35

Robert Haas robertmhaas@gmail.com

03:24 (6小时前)

发送至我、 pgsql-hackers

On Wed, Aug 7, 2024 at 11:24 PM Xiaoran Wang <fanfuxiaoran@gmail.com> wrote:
> When I use the 'pqmq' recently, I found some issues, just fix them.
>
> Allow the param 'dsm_segment *seg' to be NULL in function
> 'pq_redirect_to_shm_mq'. As sometimes the shm_mq is created
> in shared memory instead of DSM.

>. Under what circumstances does this happen?

I just create a shm_mq in shared memory, compared with DSM, it is easier.

And don't need to attach and detach to the DSM.

This shm_mq in shared memory can meet my requirement, which is used

in two different sessions, one session A dumps some information into another

session B through the shm_mq. Session B is actually a monitor session, user can

use it to monitor the state of slow queries, such as queries in session A.

Yes, I can choose to use DSM in such situation. But I think it's better to let the 'pqmq'

to support the shm_mq not in DSM.

> Add function 'pq_leave_shm_mq' to allow the process to go
> back to the previous pq environment.

>. In the code as it currently exists, a parallel worker never has a
>. connected client, and it talks to a shm_mq instead. So there's no need
>. for this. If a backend needs to communicate with both a connected
> client and also a shm_mq, it probably should not use pqmq but rather
> decide explicitly which messages should be sent to the client and
> which to the shm_mq. Otherwise, it seems hard to avoid possible loss
> of protocol sync.

As described above, session B will send a signal to session A, then

session A handle the signal and send the message into the shm_mq.

The message is sent by pq protocol. So session A will firstly call

'pq_redirect_to_shm_mq' and then call 'pq_leave_shm_mq' to

continue to do its work.

Robert Haas <robertmhaas@gmail.com> 于2024年8月9日周五 03:24写道：

On Wed, Aug 7, 2024 at 11:24 PM Xiaoran Wang <fanfuxiaoran@gmail.com> wrote:
> When I use the 'pqmq' recently, I found some issues, just fix them.
>
> Allow the param 'dsm_segment *seg' to be NULL in function
> 'pq_redirect_to_shm_mq'. As sometimes the shm_mq is created
> in shared memory instead of DSM.

Under what circumstances does this happen?

> Add function 'pq_leave_shm_mq' to allow the process to go
> back to the previous pq environment.

In the code as it currently exists, a parallel worker never has a
connected client, and it talks to a shm_mq instead. So there's no need
for this. If a backend needs to communicate with both a connected
client and also a shm_mq, it probably should not use pqmq but rather
decide explicitly which messages should be sent to the client and
which to the shm_mq. Otherwise, it seems hard to avoid possible loss
of protocol sync.

--
Robert Haas
EDB: http://www.enterprisedb.com

Re: [patch] Imporve pqmq

From

Robert Haas

Date:

12 August, 19:28:29

On Thu, Aug 8, 2024 at 10:27 PM Xiaoran Wang <fanfuxiaoran@gmail.com> wrote:
> >     Add function 'pq_leave_shm_mq' to allow the process to go
> >     back to the previous pq environment.
>
> >.    In the code as it currently exists, a parallel worker never has a
> >.    connected client, and it talks to a shm_mq instead. So there's no need
> >.    for this. If a backend needs to communicate with both a connected
> >     client and also a shm_mq, it probably should not use pqmq but rather
> >     decide explicitly which messages should be sent to the client and
> >     which to the shm_mq. Otherwise, it seems hard to avoid possible loss
> >     of protocol sync.
>
>     As described above, session B  will send a signal to session A, then
>     session A handle the signal and send the message into the shm_mq.
>    The message is sent by pq protocol. So session A will firstly call
>    'pq_redirect_to_shm_mq'  and then  call 'pq_leave_shm_mq' to
>    continue to do its work.

In this kind of use case, there is really no reason to use the libpq
protocol at all. You would be better off just using a shm_mq directly,
and then you don't need this patch. See tqueue.c for an example of
such a coding pattern.

Using pqmq is very error-prone here. In particular, if a backend
unexpectedly hits an ERROR while the direct is in place, the error
will be sent to the other session rather than to the connected client.
This breaks wire protocol synchronization.

--
Robert Haas
EDB: http://www.enterprisedb.com

Re: [patch] Imporve pqmq

From

Xiaoran Wang

Date:

13 August, 10:38:55

Robert Haas <robertmhaas@gmail.com> 于2024年8月13日周二 00:28写道：

On Thu, Aug 8, 2024 at 10:27 PM Xiaoran Wang <fanfuxiaoran@gmail.com> wrote:
> > Add function 'pq_leave_shm_mq' to allow the process to go
> > back to the previous pq environment.
>
> >. In the code as it currently exists, a parallel worker never has a
> >. connected client, and it talks to a shm_mq instead. So there's no need
> >. for this. If a backend needs to communicate with both a connected
> > client and also a shm_mq, it probably should not use pqmq but rather
> > decide explicitly which messages should be sent to the client and
> > which to the shm_mq. Otherwise, it seems hard to avoid possible loss
> > of protocol sync.
>
> As described above, session B will send a signal to session A, then
> session A handle the signal and send the message into the shm_mq.
> The message is sent by pq protocol. So session A will firstly call
> 'pq_redirect_to_shm_mq' and then call 'pq_leave_shm_mq' to
> continue to do its work.

In this kind of use case, there is really no reason to use the libpq
protocol at all. You would be better off just using a shm_mq directly,
and then you don't need this patch. See tqueue.c for an example of
such a coding pattern.

Thanks for your reply and suggestion, I will look into that.

Using pqmq is very error-prone here. In particular, if a backend
unexpectedly hits an ERROR while the direct is in place, the error
will be sent to the other session rather than to the connected client.
This breaks wire protocol synchronization.

Yes, I found this problem too. Between the 'pq_beginmessage' and 'pq_endmessage',

any log should not be emitted to the client as it will be sent to the shm_mq

instead of client. Such as I sometimes set client_min_messages='debug1'

in psql, then it will go totally wrong. It maybe better to firstly write the 'msg'

into a StringInfoData, then send the 'msg' by libpq.

I agree that it is not good way to communicate between tow sessions.

--
Robert Haas
EDB: http://www.enterprisedb.com

Best regards !

Xiaoran Wang