Thread: escape vs. bytea in config
Dear Supportlist,
I came across the "bytea_output" configuration variable inside postgresql.conf. It seems to be affecting the "digest" function, but in a way I cannot understand.
When my postgresql 14 server executes the digest function, it considers the "bytea_output" variable, but the output of the function is *NOT* the same as the "encode" function. It differs only slightly.
Please have a view of the following two conditions: First with "bytea_output = hex", second with "bytea_output = escape". In both ways the result differs, which leads to my password-comparison to turn out false.
Regards,
Mathias
-- /etc/postgresql/14/main/postgresql.conf
-- bytea_output = 'hex'
db=> select digest('mtmayr', 'sha256');
┌────────────────────────────────────────────────────────────────────┐
│ digest │
├────────────────────────────────────────────────────────────────────┤
│ \xbf1995dd67c794df2de75983acd41f66c750682ab75ca3a935538b575884fd47 │
└────────────────────────────────────────────────────────────────────┘
db=> select encode(digest('mtmayr', 'sha256'), 'hex');
┌──────────────────────────────────────────────────────────────────┐
│ encode │
├──────────────────────────────────────────────────────────────────┤
│ bf1995dd67c794df2de75983acd41f66c750682ab75ca3a935538b575884fd47 │
└──────────────────────────────────────────────────────────────────┘
-- /etc/postgresql/14/main/postgresql.conf
-- bytea_output = 'escape'
db=> select encode(digest('mtmayr', 'sha256'), 'escape');
┌────────────────────────────────────────────────────────────────────────────────────────────┐
│ encode │
├────────────────────────────────────────────────────────────────────────────────────────────┤
│ \277\x19\225\335g\307\224\337-\347Y\203\254\324\x1Ff\307Ph*\267\\\243\2515S\213WX\204\375G │
└────────────────────────────────────────────────────────────────────────────────────────────┘
db=> select digest('mtmayr', 'sha256');
┌────────────────────────────────────────────────────────────────────────────────────────────┐
│ digest │
├────────────────────────────────────────────────────────────────────────────────────────────┤
│ \277\031\225\335g\307\224\337-\347Y\203\254\324\037f\307Ph*\267\\\243\2515S\213WX\204\375G │
└────────────────────────────────────────────────────────────────────────────────────────────┘
-- ^^ differences ^^^^
-- /\/\athias /\/\ayrhofer
On Tue, 2023-03-07 at 11:53 +0100, Mathias Mayrhofer wrote:
> -- /etc/postgresql/14/main/postgresql.conf
> -- bytea_output = 'escape'
>
> db=> select encode(digest('mtmayr', 'sha256'), 'escape');
> ┌────────────────────────────────────────────────────────────────────────────────────────────┐
> │ encode │
> ├────────────────────────────────────────────────────────────────────────────────────────────┤
> │ \277\x19\225\335g\307\224\337-\347Y\203\254\324\x1Ff\307Ph*\267\\\243\2515S\213WX\204\375G │
> └────────────────────────────────────────────────────────────────────────────────────────────┘
>
> db=> select digest('mtmayr', 'sha256');
> ┌────────────────────────────────────────────────────────────────────────────────────────────┐
> │ digest │
> ├────────────────────────────────────────────────────────────────────────────────────────────┤
> │ \277\031\225\335g\307\224\337-\347Y\203\254\324\037f\307Ph*\267\\\243\2515S\213WX\204\375G │
> └────────────────────────────────────────────────────────────────────────────────────────────┘
>
> -- ^^ differences ^^^^
They are actually the same: octal 31 is hexadecimal 19.
The difference is between the "encode" function and the type putput function of "bytea".
Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com
On 2023-03-07 11:53:29 +0100, Mathias Mayrhofer wrote: > I came across the "bytea_output" configuration variable inside postgresql.conf. > It seems to be affecting the "digest" function, but in a way I cannot > understand. No, it just affects the way bytea values are displayed. The return value of the digest() function is a 32 byte bytea value in both cases. But you can choose wether you want to display bytea values as 2 hex digits per byte or as a mixture of printable ascii characters and octal escapes. > When my postgresql 14 server executes the digest function, it considers the > "bytea_output" variable, but the output of the function is *NOT* the same as > the "encode" function. It differs only slightly. I don't think the docs say anywhere that these are the same. In particular, https://www.postgresql.org/docs/15/datatype-binary.html#id-1.5.7.12.9 says that "the entire string is preceded by the sequence \x (to distinguish it from the escape format". > -- /etc/postgresql/14/main/postgresql.conf > -- bytea_output = 'escape' > > db=> select encode(digest('mtmayr', 'sha256'), 'escape'); > ┌────────────────────────────────────────────────────────────────────────────────────────────┐ > │ encode │ > ├────────────────────────────────────────────────────────────────────────────────────────────┤ > │ \277\x19\225\335g\307\224\337-\347Y\203\254\324\x1Ff\307Ph*\267\\\243\2515S\213WX\204\375G │ > └────────────────────────────────────────────────────────────────────────────────────────────┘ This is a bit weirder. Why are there two bytes encoded as hex instead of octal? It turns out this is again an artefact of displaying the value. https://www.postgresql.org/docs/14/functions-binarystring.html says: | The escape format converts zero bytes and bytes with the high bit set | into octal escape sequences (\nnn), and it doubles backslashes. Other | byte values are represented literally. So the byte 0x19 is converted to a single character U+0019 (EM) which is then displayed as '\x19', while bytes >= 0x80 are converted to four-character escape sequences. hp -- _ | Peter J. Holzer | Story must make more sense than reality. |_|_) | | | | | hjp@hjp.at | -- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!"