Thread: Better way to process records in bash?
(This might be a bash question instead of a PG question, or it might be an A/B question.)
I need to process table records in a bash script. Currently, I read them using a while loop and redirection. The table isn't that big (30ish thousand rows), and performance is adequate, but am always looking for "better".
Here's the current code:
declare f1 f3 f8
while IFS='|' read f1 f3 f8; do
something f8 f3 f1
done < <(psql -XAt -c "select f1, f3, f8 from some.table_name;")
--
while IFS='|' read f1 f3 f8; do
something f8 f3 f1
done < <(psql -XAt -c "select f1, f3, f8 from some.table_name;")
Death to America, and butter sauce.
Iraq lobster!
Hi
What is this "something" that it cannot be calculated within the dB?
What is this "something" that it cannot be calculated within the dB?
On Thu, Sep 12, 2024 at 11:43 AM Thiemo Kellner <thiemo@gelassene-pferde.biz> wrote:
Hi
What is this "something" that it cannot be calculated within the dB?
Death to America, and butter sauce.
Iraq lobster!
## Ron Johnson (ronljohnsonjr@gmail.com):
> I need to process table records in a bash script. Currently, I read them
> using a while loop and redirection. The table isn't that big (30ish
> thousand rows), and performance is adequate, but am always looking for
> "better".
Use python, or any other sane language, and don't do bash in python.
Anything non-trivial is unreasonably hard to get really right when
using shell.
Take this example (even when allowing some fuzz with the variables):
> Here's the current code:
> declare f1 f3 f8
> while IFS='|' read f1 f3 f8; do
> something f8 f3 f1
> done < <(psql -XAt -c "select f1, f3, f8 from some.table_name;")
What happens when f3
a) is NULL?
b) contains a '|' character?
c) contains spaces? (this one depends on how exactly you're handling
the shell variables).
This might be fine if your data is only NOT NULL numbers, but that's
already quite a limitation on your data.
Regards,
Christoph
--
Spare Space
On Thu, Sep 12, 2024 at 6:08 PM Ron Johnson <ronljohnsonjr@gmail.com> wrote:
(This might be a bash question instead of a PG question, or it might be an A/B question.)I need to process table records in a bash script. Currently, I read them using a while loop and redirection. The table isn't that big (30ish thousand rows), and performance is adequate, but am always looking for "better".Here's the current code:declare f1 f3 f8
while IFS='|' read f1 f3 f8; do
something f8 f3 f1
done < <(psql -XAt -c "select f1, f3, f8 from some.table_name;")--Death to America, and butter sauce.Iraq lobster!
Another approach I've found more enjoyable in some cases,
is using select to_jsonb(...) on the resultset, and then pipe this to jq.
This will preserve (more than) decent performance,
something you'll have to give up should you move to python for example.
You'll still have to use bash at some point,
but jq will be more expressive before that happens.
So instead of psql -> bash, you'll have to psql --> jq --> (bash).
On Thu, Sep 12, 2024 at 3:30 PM Christoph Moench-Tegeder <cmt@burggraben.net> wrote:
## Ron Johnson (ronljohnsonjr@gmail.com):
> I need to process table records in a bash script. Currently, I read them
> using a while loop and redirection. The table isn't that big (30ish
> thousand rows), and performance is adequate, but am always looking for
> "better".
Use python, or any other sane language, and don't do bash in python.
Anything non-trivial is unreasonably hard to get really right when
using shell.
Take this example (even when allowing some fuzz with the variables):
> Here's the current code:
> declare f1 f3 f8
> while IFS='|' read f1 f3 f8; do
> something f8 f3 f1
> done < <(psql -XAt -c "select f1, f3, f8 from some.table_name;")
What happens when f3
a) is NULL?
b) contains a '|' character?
c) contains spaces? (this one depends on how exactly you're handling
the shell variables).
This might be fine if your data is only NOT NULL numbers, but that's
already quite a limitation on your data.
Good points. Shouldn't be relevant in my situation, but defensive programming never hurts.
Death to America, and butter sauce.
Iraq lobster!
You maybe even could use plpythonu.