Would it be reasonable to just take the simple approach with same algorithm I used in the shell script? Basically: If the psql client uses a local UNIX domain socket, or localhost TCP connection, use the string output by "hostname" system command. From the C perspective, this is just calling the uname(2) and/or gethostname(2) system call.
All other remote connections would then fall back to the current %M functionality.