You analysis looks good to me, I think I missed to drop the newly created slot when changing the logic to support row filter. Here is a small patch to drop the slots in each cycle, it can fix the issue on my machine.
IIUC, the issue exists from PG15~HEAD. The current patch can apply on PG16~HEAD, If it looks ok, I will test and prepare the patch for the other branch.
This patch looks good to me. I tested again and didn't reproduce the problem.