I am using mtls ssl encryption with postgres. It works and even accepts an encrypted pk8 file. I found two things that perhaps will help your team out.
The pk8 file must use this flag on the openssl pkcs8 command to generate the pk8 file -v1 pbeWithSHA1And3-KeyTripleDES-CBC and the PgJDBC docs are wrong about this and state to use -v1 PBE-MD5-DES which doesn't work. I think the docs just weren't updated when the encryption was updated at some point in the past.
When I use postgres_fdw to make the same JDBC connection it only works if I use an unencrypted pk8 file to hold the sslkey.
I am using postgres in the alpine docker container (16.3-alpine3.20). Please let me know if and when this changes and I can use encrypted pk8 files to hold keys in postgres_fdw too.