The fragility there is not an issue in a mostly read-only application, but it definitely would be a concern in other cases.
While we accept that visibility map is good for read only application, why can't we make it optional? Atleast if there is a way for a person to drop the visibility map for a table(if it gets created by default), the application need not incur the overhead for those tables, when it knows it is update intensive / with batch jobs.
Again not to deviate from my initial question, can we make a decision regarding unstable/mutable functions / broken data types ?