Eh, and pg_upgrade [other switches] --upgrade-extensions sounds good too ...
Ultimately I believe this is the solution, however we still need to teach extensions how to upgrade themselves or emit a message saying they can't, or even ignore if it truly is a NOP.
If it's opt-in and simple I don't really care but I doubt I would use it as personally I'd rather the upgrade not touch my application at all (to the extent possible) and just basically promise that I'll get a reliable upgrade. Then I'll go ahead and ensure I have the backups of the new version and that my application works correctly, then just run the "ALTER EXTENSION" myself. But anything that will solve pain points for same-PostgreSQL-version extension upgrading is great.
I would say that it probably should be "--upgrade-extension=aaa --upgrade_extension=bbb" though if we are going to the effort to offer something.
