Hmm. The planner should avoid using a merge join if it knows that to be true. Maybe analyze'ing that table would prompt it to use some other join method?
The planner has updated stats on the table and wants to use a nested loop:
But the nested loop version is around 8 seconds slower so I forced the issue. But thanks to this conversation I now understand what's happening with the row count. This understanding helped make the nested loops' plan easier to understand. Unfortunately, there doesn't seem to be any hope for the merge join variant in terms of being easily understood. The uninitiated sees a scan node and its parent sort node and their brain defaults to thinking: the sort node will produce the same number of rows as the node feeding it.