Thread: Validity of using the test_decoding plugin for production?
Hi all, I'm considering using the test_decoding[1] plugin in conjunction with Amazon's recent update for RDS with PostgreSQL[2] which includes logical replication support. The concern I currently have is the stability and output format of the test_decoding plugin. Specifically if it's built to handle general purpose replication, has the stability to perform in a production environment, and if there are any concerns about depending on it for critical infrastructure given my current understanding is that new output plugins cannot be used with RDS. Comments to the above or experiences using test_decoding in a production setup would be appreciated, whether inside RDS or not. Thanks, -Josh [1]: https://www.postgresql.org/docs/9.6/static/test-decoding.html [2]: https://aws.amazon.com/blogs/aws/amazon-rds-for-postgresql-new-minor-versions-logical-replication-dms-and-more/
On Tue, Nov 1, 2016 at 12:57 AM, Joshua Kehn <josh@kehn.us> wrote: > I'm considering using the test_decoding[1] plugin in conjunction with > Amazon's recent update for RDS with PostgreSQL[2] which includes logical > replication support. The concern I currently have is the stability and > output format of the test_decoding plugin. Specifically if it's built to > handle general purpose replication, has the stability to perform in a > production environment, and if there are any concerns about depending on > it for critical infrastructure given my current understanding is that > new output plugins cannot be used with RDS. > > Comments to the above or experiences using test_decoding in a production > setup would be appreciated, whether inside RDS or not. Personally, I have an image of test_decoding as something people can use as a base to implement their own decoder plugin to satisfy their requirements. Based on that, it is not something that would be used for general purposes regarding replication to replicate logical changes. How are you going to use it anyway? First deparse its output and adapt it to your needs? You would give yourself a favor to hack your own thing, and that would finish by being more performant as their is not need to do a post-receive format change of the data received by a single change. Now regarding the stability of the thing, the code is in a pretty nice shape. -- Michael
> On Tue, Nov 1, 2016, at 04:00 AM, Michael Paquier wrote: > How are you going to use it anyway? First deparse its output > and adapt it to your needs? That's the intention. Take the output of test_decoding, transform it into a reasonable object form, and feed that into an events stream for real-time consumption. The initial question was suitability of the test_decoding plugin, given that it is the only one I see available on RDS and requiring a switch to a self-managed Postgres instance—to use a separate plugin—would kill this idea. Are there any success/failure stories about using logical decoding as a way of generating an events stream? There's some new question marks over the consumer side, whether to use pg_recvlogical or something with a peek/write to stream/pop cycle to protect the consumer from failure. Given a single reader, what is the common failure mode when consuming the output? Best, -Josh
On Wed, Nov 2, 2016 at 4:01 AM, Joshua Kehn <josh@kehn.us> wrote: > That's the intention. Take the output of test_decoding, transform it > into a reasonable object form, and feed that into an events stream for > real-time consumption. Hm... > The initial question was suitability of the test_decoding plugin, given > that it is the only one I see available on RDS and requiring a switch to > a self-managed Postgres instance—to use a separate plugin—would kill > this idea. OK, and no custom binaries can be installed in this case. IMO you would have a far easier time moving to something else if you are willing to fetch custom changes or to use some of your own plugins. Vendor-locking with RDS is your problem here, and trying to use the stream from test_decoding to then modify it in the way you want is likely going to embark you in a journey you'll regret in the long term. I don't know the opinion of others on the matter... But IMO test_decoding is an example of output plugin, so there is no need to keep its output stable across major releases if any needs for another patch or feature pops in. > Are there any success/failure stories about using logical decoding as a > way of generating an events stream? There's some new question marks over > the consumer side, whether to use pg_recvlogical or something with a > peek/write to stream/pop cycle to protect the consumer from failure. pglogical has been one solution around that worked on logical decoding, and as far as I recall it uses its own protocol to replicate changes. I never had a look at it in details but that may be something to look into if you are looking for some story in this area. -- Michael