I know 20ms is nothing but it shows me that there is a problem with my configuration. I want to find it.
This is a dangerous assumption. This is no configuration you can come up with which will cause the planner to be within 20% of perfection in all cases. Given the other plans you've shown and discussed, I think this is just chasing our own tail.