Thread: parallel-processing multiple similar query tasks - any example?
multiple similar query tasks are as follows:
select * from a_table where country ='UK'
select * from a_table where country='France'
and so on
How best to parallel-processing such types of multiple similar query tasks?
Any example available?
Regards,
David
On Wed, Apr 27, 2022 at 4:34 PM Shaozhong SHI <shishaozhong@gmail.com> wrote:
multiple similar query tasks are as follows:select * from a_table where country ='UK'select * from a_table where country='France'and so onHow best to parallel-processing such types of multiple similar query tasks?Any example available?
You should search for how to run processes/commands in parallel in whatever client-side execution environment you want to use. There isn't anything specific to PostgreSQL here.
David J.
On Wed, Apr 27, 2022 at 4:44 PM David G. Johnston <david.g.johnston@gmail.com> wrote:
On Wed, Apr 27, 2022 at 4:34 PM Shaozhong SHI <shishaozhong@gmail.com> wrote:multiple similar query tasks are as follows:select * from a_table where country ='UK'select * from a_table where country='France'and so onHow best to parallel-processing such types of multiple similar query tasks?Any example available?You should search for how to run processes/commands in parallel in whatever client-side execution environment you want to use. There isn't anything specific to PostgreSQL here.
You should also read:
If you want to see when a single query can be executed using parallel workers.
David J.
On 2022-Apr-28, Shaozhong SHI wrote:
> multiple similar query tasks are as follows:
>
> select * from a_table where country ='UK'
> select * from a_table where country='France'
> and so on
>
> How best to parallel-processing such types of multiple similar query tasks?
>
> Any example available?
for country in UK France Germany Ireland; do
echo "select pg_sleep(1); select * from a_table where country = '${country//\'/''/}'"
done | \
xargs -d"\n" -P10 -n1 psql -X -c
Note the ${country/} stuff is a bash-ism.
--
Álvaro Herrera
No, No.
Why sleep(1)?
It should be all active - doing work concurrently.
Regards,
David
On Thu, 28 Apr 2022 at 16:17, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2022-Apr-28, Shaozhong SHI wrote:
> multiple similar query tasks are as follows:
>
> select * from a_table where country ='UK'
> select * from a_table where country='France'
> and so on
>
> How best to parallel-processing such types of multiple similar query tasks?
>
> Any example available?
for country in UK France Germany Ireland; do
echo "select pg_sleep(1); select * from a_table where country = '${country//\'/''/}'"
done | \
xargs -d"\n" -P10 -n1 psql -X -c
Note the ${country/} stuff is a bash-ism.
--
Álvaro Herrera
On 2022-Apr-28, Shaozhong SHI wrote: > Why sleep(1)? It is sleeping to show that they are running concurrently. If it runs five sleeps of one second each and the whole command lasts one second, then all sleeps ran in parallel. Had the whole command taken five seconds, you would know that the queries ran serially. > It should be all active - doing work concurrently. They are all active simultaneously. You just need to supply your own query, without any sleeps. -- Álvaro Herrera
Expand and explain please.
Regards,
David
On Thursday, 28 April 2022, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On Thursday, 28 April 2022, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2022-Apr-28, Shaozhong SHI wrote:
> Why sleep(1)?
It is sleeping to show that they are running concurrently. If it runs
five sleeps of one second each and the whole command lasts one second,
then all sleeps ran in parallel. Had the whole command taken five
seconds, you would know that the queries ran serially.
> It should be all active - doing work concurrently.
They are all active simultaneously. You just need to supply your own
query, without any sleeps.
--
Álvaro Herrera
On 2022-Apr-28, Shaozhong SHI wrote: > Expand and explain please. No, thanks. -- Álvaro Herrera
Well, I guess that does not work.
Never mind.
On Thursday, 28 April 2022, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
Regards,
David
On Thursday, 28 April 2022, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote:
On 2022-Apr-28, Shaozhong SHI wrote:
> Expand and explain please.
No, thanks.
--
Álvaro Herrera