Eh, and pg_upgrade [other switches] --upgrade-extensions sounds good too ...
Ultimately I believe this is the solution, however we still need to teach extensions how to upgrade themselves or emit a message saying they can't, or even ignore if it truly is a NOP.
If it's opt-in and simple I don't really care but I doubt I would use it as personally I'd rather the upgrade not touch my application at all (to the extent possible) and just basically promise that I'll get a reliable upgrade. Then I'll go ahead and ensure I have the backups of the new version and that my application works correctly, then just run the "ALTER EXTENSION" myself. But anything that will solve pain points for same-PostgreSQL-version extension upgrading is great.
I may have not communicated this clearly. In this case the application worked fine, the extension worked fine. The issue only arose when doing a dump and restore of the database and then the only reason it failed was due to trying to revoke permissions from pg_stat_statements_reset.
There may have been other things that were not working correctly but since it did not cause any errors it was difficult to know.
As Robert points out in the next message this is not a particularly great user experience
