Did you change the system-wide autovacuum_analyze_scale_factor? If so, don't do that. You can use a table's storage parameters to set a custom autovacuum_analyze_scale_factor just for individual tables. So just the table with the troublesome gin index:
No I did it just to the problematic table:
ALTER TABLE my_table SET (autovacuum_vacuum_scale_factor=0);
ALTER TABLE my_table SET (autovacuum_analyze_scale_factor=0);
My system-wide autovacuum_analyze_threshold is 50 so I think that's ok.
There is a bulk load going on right now so a lot of tables are needing vacuuming. I really need to increase my autovacuum_max_workers.
Killing existing vacuums won't help (at least not until change your system wide setting back and so the above instead) because it will likely just restart on the same table it was on when you killed it.
You're right, I gave up trying, and am back running a manual VACUUM. It's slow, but I can't bounce the db right now.