Thread: Planet Postgres and the curse of AI
I've been noticing a growing trend of blog posts written mostly, if not entirely, with AI (aka LLMs, ChatGPT, etc.). I'm not sure where to raise this issue. I considered a blog post, but this mailing list seemed a better forum to generate a discussion.
The problem is two-fold as I see it.
First, there is the issue of people trying to game the system by churning out content that is not theirs, but was written by a LLM. I'm not going to name specific posts, but after a while it gets easy to recognize things that are written mostly by AI.
These blog posts are usually generic, describing some part of Postgres in an impersonal, mid-level way. Most of the time the facts are not wrong, per se, but they lack nuances that a real DBA would bring to the discussion, and often leave important things out. Code examples are often wrong in subtle ways. Places where you might expect a deeper discussion are glossed over.
The problem is two-fold as I see it.
First, there is the issue of people trying to game the system by churning out content that is not theirs, but was written by a LLM. I'm not going to name specific posts, but after a while it gets easy to recognize things that are written mostly by AI.
These blog posts are usually generic, describing some part of Postgres in an impersonal, mid-level way. Most of the time the facts are not wrong, per se, but they lack nuances that a real DBA would bring to the discussion, and often leave important things out. Code examples are often wrong in subtle ways. Places where you might expect a deeper discussion are glossed over.
So this first problem is that it is polluting the Postgres blogs with overly bland, moderately helpful posts that are not written by a human, and do not really bring anything interesting to the table. There is a place for posts that describe basic Postgres features, but the ones written by humans are much better. (yeah, yeah, "for now" and all hail our AI overlords in the future).
The second problem is worse, in that LLMs are not merely gathering information, but have the ability to synthesize new conclusions and facts. In short, they can lie. Or hallucinate. However you want to call it, it's a side effect of the way LLMs work. In a technical field like Postgres, this can be a very bad thing. I don't know how widespread this is, but I was tipped off about this over a year ago when I came across a blog suggesting using the "max_toast_size configuration parameter". For those not familiar, I can assure you that Postgres does not have, nor will likely ever have, a GUC with that name.
As anyone who has spoken with ChatGPT knows, getting small important details correct is not its forte. I love ChatGPT and actually use it daily. It is amazing at doing certain tasks. But writing blog posts should not be one of them.
Do we need a policy or a guideline for Planet Postgres? I don't know. It can be a gray line. Obviously spelling and grammar checking is quite okay, and making up random GUCs is not, but the middle bit is very hazy. (Human) thoughts welcome.
The second problem is worse, in that LLMs are not merely gathering information, but have the ability to synthesize new conclusions and facts. In short, they can lie. Or hallucinate. However you want to call it, it's a side effect of the way LLMs work. In a technical field like Postgres, this can be a very bad thing. I don't know how widespread this is, but I was tipped off about this over a year ago when I came across a blog suggesting using the "max_toast_size configuration parameter". For those not familiar, I can assure you that Postgres does not have, nor will likely ever have, a GUC with that name.
As anyone who has spoken with ChatGPT knows, getting small important details correct is not its forte. I love ChatGPT and actually use it daily. It is amazing at doing certain tasks. But writing blog posts should not be one of them.
Do we need a policy or a guideline for Planet Postgres? I don't know. It can be a gray line. Obviously spelling and grammar checking is quite okay, and making up random GUCs is not, but the middle bit is very hazy. (Human) thoughts welcome.
Cheers,
Greg
st 17. 7. 2024 v 19:22 odesílatel Greg Sabino Mullane <htamfids@gmail.com> napsal:
I've been noticing a growing trend of blog posts written mostly, if not entirely, with AI (aka LLMs, ChatGPT, etc.). I'm not sure where to raise this issue. I considered a blog post, but this mailing list seemed a better forum to generate a discussion.
The problem is two-fold as I see it.
First, there is the issue of people trying to game the system by churning out content that is not theirs, but was written by a LLM. I'm not going to name specific posts, but after a while it gets easy to recognize things that are written mostly by AI.
These blog posts are usually generic, describing some part of Postgres in an impersonal, mid-level way. Most of the time the facts are not wrong, per se, but they lack nuances that a real DBA would bring to the discussion, and often leave important things out. Code examples are often wrong in subtle ways. Places where you might expect a deeper discussion are glossed over.So this first problem is that it is polluting the Postgres blogs with overly bland, moderately helpful posts that are not written by a human, and do not really bring anything interesting to the table. There is a place for posts that describe basic Postgres features, but the ones written by humans are much better. (yeah, yeah, "for now" and all hail our AI overlords in the future).
The second problem is worse, in that LLMs are not merely gathering information, but have the ability to synthesize new conclusions and facts. In short, they can lie. Or hallucinate. However you want to call it, it's a side effect of the way LLMs work. In a technical field like Postgres, this can be a very bad thing. I don't know how widespread this is, but I was tipped off about this over a year ago when I came across a blog suggesting using the "max_toast_size configuration parameter". For those not familiar, I can assure you that Postgres does not have, nor will likely ever have, a GUC with that name.
As anyone who has spoken with ChatGPT knows, getting small important details correct is not its forte. I love ChatGPT and actually use it daily. It is amazing at doing certain tasks. But writing blog posts should not be one of them.
Do we need a policy or a guideline for Planet Postgres? I don't know. It can be a gray line. Obviously spelling and grammar checking is quite okay, and making up random GUCs is not, but the middle bit is very hazy. (Human) thoughts welcome.
It is very unpleasant to read a long article, and at the end to understand so there is zero valuable information. Terrible situation was on planet mariadb https://mariadb.org/planet/, but now it was cleaned. I am for some form of moderating - and gently touching an author that writes articles without extra value against documentation.
Regards
Pavel
Cheers,Greg
Hi Greg
I agree with you on the misuse of AI based tools, as per my experience with Postgres the solutions suggested wont work at times.
Its not bad to get help from these tools but put all the solutions from there is counter productive.
I think People should take care while using these tools while suggesting solutions for real world problems.
Regards
Kashif Zeeshan
On Wed, Jul 17, 2024 at 10:22 PM Greg Sabino Mullane <htamfids@gmail.com> wrote:
I've been noticing a growing trend of blog posts written mostly, if not entirely, with AI (aka LLMs, ChatGPT, etc.). I'm not sure where to raise this issue. I considered a blog post, but this mailing list seemed a better forum to generate a discussion.
The problem is two-fold as I see it.
First, there is the issue of people trying to game the system by churning out content that is not theirs, but was written by a LLM. I'm not going to name specific posts, but after a while it gets easy to recognize things that are written mostly by AI.
These blog posts are usually generic, describing some part of Postgres in an impersonal, mid-level way. Most of the time the facts are not wrong, per se, but they lack nuances that a real DBA would bring to the discussion, and often leave important things out. Code examples are often wrong in subtle ways. Places where you might expect a deeper discussion are glossed over.So this first problem is that it is polluting the Postgres blogs with overly bland, moderately helpful posts that are not written by a human, and do not really bring anything interesting to the table. There is a place for posts that describe basic Postgres features, but the ones written by humans are much better. (yeah, yeah, "for now" and all hail our AI overlords in the future).
The second problem is worse, in that LLMs are not merely gathering information, but have the ability to synthesize new conclusions and facts. In short, they can lie. Or hallucinate. However you want to call it, it's a side effect of the way LLMs work. In a technical field like Postgres, this can be a very bad thing. I don't know how widespread this is, but I was tipped off about this over a year ago when I came across a blog suggesting using the "max_toast_size configuration parameter". For those not familiar, I can assure you that Postgres does not have, nor will likely ever have, a GUC with that name.
As anyone who has spoken with ChatGPT knows, getting small important details correct is not its forte. I love ChatGPT and actually use it daily. It is amazing at doing certain tasks. But writing blog posts should not be one of them.
Do we need a policy or a guideline for Planet Postgres? I don't know. It can be a gray line. Obviously spelling and grammar checking is quite okay, and making up random GUCs is not, but the middle bit is very hazy. (Human) thoughts welcome.Cheers,Greg
On 7/17/24 10:21, Greg Sabino Mullane wrote: > I've been noticing a growing trend of blog posts written mostly, if not > entirely, with AI (aka LLMs, ChatGPT, etc.). I'm not sure where to raise > this issue. I considered a blog post, but this mailing list seemed a > better forum to generate a discussion. > > > Do we need a policy or a guideline for Planet Postgres? I don't know. It > can be a gray line. Obviously spelling and grammar checking is quite > okay, and making up random GUCs is not, but the middle bit is very hazy. > (Human) thoughts welcome. A policy would be nice, just not sure how enforceable it would be. How do you differentiate between the parrot that is AI and one that is human? I run across all manner of blog posts where folks have lifted content from the documentation or other sources without attribution, which is basically what AI generated content is. AI does like to embellish and make things up(ask the NYC lawyer suing the airlines about that), though that is a human trait as well. > > Cheers, > Greg > -- Adrian Klaver adrian.klaver@aklaver.com
On Wed, 2024-07-17 at 13:21 -0400, Greg Sabino Mullane wrote: > I've been noticing a growing trend of blog posts written mostly, if not entirely, with AI > (aka LLMs, ChatGPT, etc.). I'm not sure where to raise this issue. I considered a blog post, > but this mailing list seemed a better forum to generate a discussion. > > The problem is two-fold as I see it. > > First, there is the issue of people trying to game the system by churning out content that is not theirs [...] > > So this first problem is that it is polluting the Postgres blogs [...] > > The second problem is worse, in that LLMs are not merely gathering information, but have > the ability to synthesize new conclusions and facts. In short, they can lie. > > Do we need a policy or a guideline for Planet Postgres? I don't know. It can be a gray line. > Obviously spelling and grammar checking is quite okay, and making up random GUCs is not, > but the middle bit is very hazy. (Human) thoughts welcome. As someone who writes blogs and occasionally browses Planet Postgres, this has not struck me as a major problem. I just scrolled through it and nothing stood out to me - perhaps I am too naïve. There certainly are people who publish random short utterances, perhaps with the intention to hit the "top posters" list, but I don't think we need strong measures. If anything, I am most annoyed by articles that are just thinly veiled advertising, but there is already a policy controlling that. As long as there is not a flood of AI generated babble (and I cannot see one), I'd say that this will regulate itself: spewing empty content and lies is not going to reflect well on the author and his/her organization. PostgreSQL has excellent documentation. Anybody who blindly follows advice from a blog without checking with the documentation only has himself/herself to blame. Yours, Laurenz Albe
I wrote: > On Wed, 2024-07-17 at 13:21 -0400, Greg Sabino Mullane wrote: > > I've been noticing a growing trend of blog posts written mostly, if not entirely, with AI > > (aka LLMs, ChatGPT, etc.). I'm not sure where to raise this issue. I considered a blog post, > > but this mailing list seemed a better forum to generate a discussion. > > > > [...] > > > > Do we need a policy or a guideline for Planet Postgres? I don't know. It can be a gray line. > > Obviously spelling and grammar checking is quite okay, and making up random GUCs is not, > > but the middle bit is very hazy. (Human) thoughts welcome. > > As someone who writes blogs and occasionally browses Planet Postgres, this has not > struck me as a major problem. I just scrolled through it and nothing stood out to > me - perhaps I am too naïve. Seems like I *was* naïve - Álvaro has pointed me to a juicy example off-list. Still, I wouldn't make a policy specifically against AI generated content. That is hard to prove, and it misses the core of the problem. The real problem is low-level, counterfactual content, be it generated by an AI or not. Perhaps there could be a way to report misleading, bad content and a policy that says that you can be banned if you repeatedly write grossly misleading and counterfactual content. Stuff like "to improve performance, set fast_mode = on and restart the database". Yours, Laurenz Albe
On Fri, 19 Jul 2024 at 00:31, Laurenz Albe <laurenz.albe@cybertec.at> wrote: > Perhaps there could be a way to report misleading, bad content and a policy that says > that you can be banned if you repeatedly write grossly misleading and counterfactual > content. Stuff like "to improve performance, set fast_mode = on and restart the database". As a first step, maybe it's worth just privately writing to the offenders telling them what's been seen, giving them a chance to improve and letting them know what they're doing isn't going unnoticed. If I was doing this and someone pointed out lots of silly mistakes with something I'd published, I'd be very embarrassed and I'd reconsider my blog writing approach. It might also be worth considering if we want to have a policy on LLM usage in https://www.postgresql.org/about/policies/planet-postgresql/ . If we want to disallow blogs written by LLMs then we'd need to be careful about how we define that as doing something like using an LLM-based spell checker does not seem like it should be disallowed. But to what degree exactly should that be allowed? David
> But to what degree exactly should that be allowed?
Somewhat ironically, here's a distinction chatgpt and I came up with:
Human-edited or reviewed content: Content that has been substantially revised, corrected, or enhanced by a human after initial generation by LLMs. This includes using spell and grammar checking, manual edits for clarity or style, and content that reflects significant human input beyond the original LLM output.
On Thu, 2024-07-18 at 10:25 -0400, Greg Sabino Mullane wrote: > > But to what degree exactly should that be allowed? > > Somewhat ironically, here's a distinction chatgpt and I came up with: > > LLM-generated content: Content where the substantial part of the text is directly > created by LLMs without significant human alteration or editing. I have no problem with that definition, but it is useless as a policy: Even in a blog with glaring AI nonsense in it, how can you prove that the author did not actually edit and improve other significant parts of the text? Why not say that authors who repeatedly post grossly counterfactual or misleading content can be banned? Yours, Laurenz Albe
On Fri, Jul 19, 2024 at 3:22 AM Laurenz Albe <laurenz.albe@cybertec.at> wrote:
I have no problem with that definition, but it is useless as a policy:
Even in a blog with glaring AI nonsense in it, how can you prove that the
author did not actually edit and improve other significant parts of the text?
Well, we can't 100% prove it, but we can have ethical guidelines. We already have other guidelines that are open to interpretation (and plenty of planet posts bend the rules quite often, IMO, but that's another post).
Why not say that authors who repeatedly post grossly counterfactual or
misleading content can be banned?
Banned is a strong word, but certainly they can have the posts removed, and receive warnings from the planet admins. If the admins can point to a policy, that helps. Perhaps as you hint at, we need a policy to not just discourage AI-generated things, but also wrong/misleading things in general (which was not much of a problem before LLMs arrived, to be honest).
Cheers,
Greg
On Tue, 2024-07-23 at 10:38 -0400, Greg Sabino Mullane wrote: > > Why not say that authors who repeatedly post grossly counterfactual or > > misleading content can be banned? > > Perhaps as you hint at, we need a policy to not just discourage AI-generated > things, but also wrong/misleading things in general I have been known to make mistakes in my blogs... We shouldn't discourage people who happen to blog something wrong. That's why I used strong verbiage like "grossly counterfactual". Yours, Laurenz Albe
Hi,
attempting to make it as best/informative and as perfect as possible, it might seem slightly frustrating when
we see some AI generated content, especially when it is misleading readers.
However, I do agree with Lawrence that it is impossible to prove whether it is written by AI or a human.
AI can make mistakes and it might mistakenly point out that a blog is written by AI (which I know is difficult to implement).
I see Moderators spending some time reviewing content and accepting or warning if it is not related to Postgres.
AI may be adopted to help us score whether an article is related to Postgres and decline the submission/blog feed.
But, it is very impossible to use AI or some strategy to identify whether it is written by AI or human.
People may also use AI generated Images in their blogs, and they may be meaningful for their article.
Is it only the content or also the images ? It might get too complicated while implementing some rules.
Ultimately, Humans do make mistakes and we shouldn't discourage people assuming it is AI that made that mistake.
On Tue, Jul 23, 2024 at 11:51 AM Laurenz Albe <laurenz.albe@cybertec.at> wrote:
On Tue, 2024-07-23 at 10:38 -0400, Greg Sabino Mullane wrote:
> > Why not say that authors who repeatedly post grossly counterfactual or
> > misleading content can be banned?
>
> Perhaps as you hint at, we need a policy to not just discourage AI-generated
> things, but also wrong/misleading things in general
I have been known to make mistakes in my blogs...
We shouldn't discourage people who happen to blog something wrong.
That's why I used strong verbiage like "grossly counterfactual".
Yours,
Laurenz Albe
Regards,
Avinash Vallarapu
Avinash Vallarapu
On Tue, Jul 23, 2024 at 12:45 PM Avinash Vallarapu <avinash.vallarapu@gmail.com> wrote:
However, I do agree with Lawrence that it is impossible to prove whether it is written by AI or a human.AI can make mistakes and it might mistakenly point out that a blog is written by AI (which I know is difficult to implement).
Right - I am not interested in "proving" things, but I think a policy to discourage overuse of AI is warranted.
People may also use AI generated Images in their blogs, and they may be meaningful for their article.Is it only the content or also the images ? It might get too complicated while implementing some rules.
Only the content, the images are perfectly fine. Even expected, these days.
Ultimately, Humans do make mistakes and we shouldn't discourage people assuming it is AI that made that mistake.
Humans make mistakes. AI confidently hallucinates.
On Fri, Jul 19, 2024 at 3:22 AM Laurenz Albe <laurenz.albe@cybertec.at> wrote:
Why not say that authors who repeatedly post grossly counterfactual or
misleading content can be banned?
I like this, and feel we are getting closer. How about:
"Posts should be technically and factually correct. Use of AI should be used for minor editing, not primary generation"
(wordsmithing needed)
Cheers,
Greg
On 2024-08-20 22:44, Greg Sabino Mullane wrote: > On Fri, Jul 19, 2024 at 3:22 AM Laurenz Albe <laurenz.albe@cybertec.at> > wrote: > >> Why not say that authors who repeatedly post grossly counterfactual or >> misleading content can be banned? > > I like this, and feel we are getting closer. How about: > > "Posts should be technically and factually correct. Use of AI should be > used for minor editing, not primary generation" Sounds pretty sensible. :) + Justin
On Wed, Aug 21, 2024 at 02:19:22AM +1000, Justin Clift wrote: > On 2024-08-20 22:44, Greg Sabino Mullane wrote: > > On Fri, Jul 19, 2024 at 3:22 AM Laurenz Albe <laurenz.albe@cybertec.at> > > wrote: > > > > > Why not say that authors who repeatedly post grossly counterfactual or > > > misleading content can be banned? > > > > I like this, and feel we are getting closer. How about: > > > > "Posts should be technically and factually correct. Use of AI should be > > used for minor editing, not primary generation" > > Sounds pretty sensible. :) Agreed. Honestly, some of the AI is so bad that if you see something you suspect is AI generated, you can just ask the author what they meant by that paragraph, and they will not be able to answer. -- Bruce Momjian <bruce@momjian.us> https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
On Tue, Aug 20, 2024 at 8:33 AM Greg Sabino Mullane <htamfids@gmail.com> wrote: > > On Tue, Jul 23, 2024 at 12:45 PM Avinash Vallarapu <avinash.vallarapu@gmail.com> wrote: >> >> However, I do agree with Lawrence that it is impossible to prove whether it is written by AI or a human. >> AI can make mistakes and it might mistakenly point out that a blog is written by AI (which I know is difficult to implement). > > > Right - I am not interested in "proving" things, but I think a policy to discourage overuse of AI is warranted. > >> People may also use AI generated Images in their blogs, and they may be meaningful for their article. >> Is it only the content or also the images ? It might get too complicated while implementing some rules. > > > Only the content, the images are perfectly fine. Even expected, these days. > >> >> Ultimately, Humans do make mistakes and we shouldn't discourage people assuming it is AI that made that mistake. > > > Humans make mistakes. AI confidently hallucinates. > I think this is a key point, and one that we could focus on for purposes of discouragement. Ie. "Blogs that are found to repeatedly post incorrect information and/or AI style hallucinations may be restricted from contributing to the planet postgres feed. This will be determined on a case by case basis." While it is likely impossible to come up with a set of rules that will satisfy some of the more legalistic folks among us, this would be a simple warning that would at least encourage folks to make sure they aren't posting bad information and leave a door open for enforcement if needed. And yes, this assumes that the folks running planet will enforce if needed, though I don't think it requires heavy policing at this point. Robert Treat https://xzilla.net
> Posts should be technically and factually correct agreed and period. no need qualify how the nonsense was created. -john On Thu, Aug 22, 2024 at 4:13 PM Robert Treat <rob@xzilla.net> wrote: > > On Tue, Aug 20, 2024 at 8:33 AM Greg Sabino Mullane <htamfids@gmail.com> wrote: > > > > On Tue, Jul 23, 2024 at 12:45 PM Avinash Vallarapu <avinash.vallarapu@gmail.com> wrote: > >> > >> However, I do agree with Lawrence that it is impossible to prove whether it is written by AI or a human. > >> AI can make mistakes and it might mistakenly point out that a blog is written by AI (which I know is difficult to implement). > > > > > > Right - I am not interested in "proving" things, but I think a policy to discourage overuse of AI is warranted. > > > >> People may also use AI generated Images in their blogs, and they may be meaningful for their article. > >> Is it only the content or also the images ? It might get too complicated while implementing some rules. > > > > > > Only the content, the images are perfectly fine. Even expected, these days. > > > >> > >> Ultimately, Humans do make mistakes and we shouldn't discourage people assuming it is AI that made that mistake. > > > > > > Humans make mistakes. AI confidently hallucinates. > > > > I think this is a key point, and one that we could focus on for > purposes of discouragement. Ie. "Blogs that are found to repeatedly > post incorrect information and/or AI style hallucinations may be > restricted from contributing to the planet postgres feed. This will be > determined on a case by case basis." While it is likely impossible to > come up with a set of rules that will satisfy some of the more > legalistic folks among us, this would be a simple warning that would > at least encourage folks to make sure they aren't posting bad > information and leave a door open for enforcement if needed. And yes, > this assumes that the folks running planet will enforce if needed, > though I don't think it requires heavy policing at this point. > > Robert Treat > https://xzilla.net > > -- Fast is fine, But accuracy is final. You must learn to be slow in a hurry. - Wyatt Earp