Thread: BUG #17193: Issue with pg_output login
The following bug has been logged on the website: Bug reference: 17193 Logged by: DBA TEAM Email address: sushant.kamboj@getfareye.com PostgreSQL version: 9.6.23 Operating system: Ubuntu 20.04.2 LTS Description: We identified that there was an issue with the encoding with UTF8 characters with pg_output plugin. The data was visible correctly in the database but while pushing the data through the CDC pipeline there was some distortion with the Greek letters. Note:- We are pushing data through debezium.
On Tue, Sep 14, 2021, at 10:44 AM, PG Bug reporting form wrote:
We identified that there was an issue with the encoding with UTF8 characterswith pg_output plugin. The data was visible correctly in the database butwhile pushing the data through the CDC pipeline there was some distortionwith the Greek letters.
You don't provide enough information. What's your database encoding?
IIRC Debezium only supports database whose encoding is UTF-8. Since pgoutput
doesn't do encoding conversion and it uses binary output, it seems the issue is
in the Debezium side.
Hi Euler,
Thanks for your reply.
I will check from the debezium side and will try to provide some more logs over this case.
On Thu, Sep 16, 2021 at 12:17 AM Euler Taveira <euler@eulerto.com> wrote:
On Tue, Sep 14, 2021, at 10:44 AM, PG Bug reporting form wrote:We identified that there was an issue with the encoding with UTF8 characterswith pg_output plugin. The data was visible correctly in the database butwhile pushing the data through the CDC pipeline there was some distortionwith the Greek letters.You don't provide enough information. What's your database encoding?IIRC Debezium only supports database whose encoding is UTF-8. Since pgoutputdoesn't do encoding conversion and it uses binary output, it seems the issue isin the Debezium side.
Thanks and Regards
Sushant Kamboj
Database Administrator
+91-9999518201
Hi Euler,
As we have verified in our environment ,your above statement looks like a contradiction since this is working fine when we are using wal2json plugin instead of pgoutput .
Since pgoutput is a native plugin, kindly let us know any settings or parameter change that we can do to fix this on postgresql or debezium side.
FYI:We are using postgresql 12.5 RDS
On Thu, Sep 16, 2021 at 12:17 AM Euler Taveira <euler@eulerto.com> wrote:
On Tue, Sep 14, 2021, at 10:44 AM, PG Bug reporting form wrote:We identified that there was an issue with the encoding with UTF8 characterswith pg_output plugin. The data was visible correctly in the database butwhile pushing the data through the CDC pipeline there was some distortionwith the Greek letters.You don't provide enough information. What's your database encoding?IIRC Debezium only supports database whose encoding is UTF-8. Since pgoutputdoesn't do encoding conversion and it uses binary output, it seems the issue isin the Debezium side.
Thanks and Regards
Sushant Kamboj
Database Administrator
+91-9999518201
On Thu, Sep 16, 2021, at 9:36 AM, Sushant Kamboj wrote:
As we have verified in our environment ,your above statement looks like a contradiction since this is working fine when we are using wal2json plugin instead of pgoutput .
The main difference is: wal2json uses text output and pgoutput uses binary
output. I'm not sure if it matters for Debezium.
Since pgoutput is a native plugin, kindly let us know any settings or parameter change that we can do to fix this on postgresql or debezium side.FYI:We are using postgresql 12.5 RDS
You don't provide enough information for an investigation. You can start with a
reproducible test case.
--
Euler Taveira
On Tue, Sep 21, 2021 at 5:47 PM Euler Taveira <euler@eulerto.com> wrote:
On Thu, Sep 16, 2021, at 9:36 AM, Sushant Kamboj wrote:As we have verified in our environment ,your above statement looks like a contradiction since this is working fine when we are using wal2json plugin instead of pgoutput .The main difference is: wal2json uses text output and pgoutput uses binaryoutput. I'm not sure if it matters for Debezium.Since pgoutput is a native plugin, kindly let us know any settings or parameter change that we can do to fix this on postgresql or debezium side.FYI:We are using postgresql 12.5 RDSYou don't provide enough information for an investigation. You can start with areproducible test case.--Euler Taveira
At Debezium documentation at [Debezium connector for PostgreSQL :: Debezium Documentation], we can see:
Debezium currently supports databases with UTF-8 character encoding only. With a single byte character encoding, it is not possible to correctly process strings that contain extended ASCII code characters.
Debezium currently supports databases with UTF-8 character encoding only. With a single byte character encoding, it is not possible to correctly process strings that contain extended ASCII code characters.
I believe this is not PostgreSQL BUG, but a Debezium Limitation.