If you use EXPLAIN (VERBOSE), you will see that the function gets inlined in the fast case.
That saves the overhead of a function call.
The IMMUTABLE function cannot be inlined because to_char() is not IMMUTABLE.
So, the punishment for lying about the volatility of one's function is to prohibit it from being inlined even in a case where had you been truthful about the volatility it would have been inlined.
