Of course this can only be used after the outer join relids has been added by add_outer_joins_to_relids(). This calculation is performed multiple times during planning. I'm not sure if this has performance issues. Maybe we can calculate it only once and store the result in some place (such as in JoinPath)?
In the v2 patch, I added a member in JoinPath to store the relid set of any outer joins that will be calculated at this join, and this would avoid repeating this calculation when creating nestloop/merge/hash join plan nodes. Also fixed a comment in v2.