With some system of symlinks to make it all work with defaults for those who don't care, a libc could have /usr/share/locale/en_US@CLDR34.UTF-8 etc so you could setlocale(LC_COLLATE, "en_US@CLDR34"), or something. I suppose they don't want to promise to be able to interpret the old data in future releases, and, as you say, sometimes the changes are in C code, due to bugs or algorithm changes, not the data.
If I understand correctly, files in /usr/share/locale aren’t enough because those only have the tailoring rules, and core algorithm and data (before applying locale-specific tweaks) also change between versions. I’m pretty sure glibc works similar to UCA in this regard (albeit based on ISO 14651 and not CDLR), and the Unicode link above is a good illustration of default collation rules that underly the locale-specific tweaks.
