It seems as if the planner is not using the PRIMARY KEY as index which was my assumption.
pgsql-performance by date:
Соглашаюсь с условиями обработки персональных данных