Some connections should only come in through the VIP, while others (like replication) must come in through .4 and others (f.e. administrators, can come in from .4 or .5).
(Why don't we use a connection pooler? The 3rd party application has only been validated against direct connections to PG. Bugs in PgPool caused problems in prod.)