No, a textual output plugin is *NOT* allowed to produce binary output. That'd violate e.g. pg_logical_slot_peek_changes's return type because it's only declared to return text.
A textual output plugin can call pg_logical_slot_peek_binary_changes and pg_logical_slot_peek_changes as well,
and a binary output plugin can only call pg_logical_slot_peek_binary_changes, and will error out with pg_logical_slot_peek_changes: