Yes. The output plugin declares whether it requires the *output method* to support binary data. pg_logical_slot_peek_changes *can not* support binary data because it outputs data as text. pg_logical_slot_peek_binary_changes *can* support binary data because it returns bytea (and thus it also can output text, because that's essentially a subset of binary data).
Thanks for the explanations. This would meritate more details within the docs, like what those two modes actually do and what the user can expect as differences, advantages and disadvantages if he chooses one or the other when starting decoding...