Read table rows in chunks - Mailing list pgsql-hackers

From Sushrut Shivaswamy
Subject Read table rows in chunks
Date
Msg-id CAH5mb99Ej7P9+tpR9_hPpO8rMEqJK264WCAW8uz6Lhm0PRH5VQ@mail.gmail.com
Whole thread Raw
Responses Re: Read table rows in chunks
Re: Read table rows in chunks
List pgsql-hackers
Hey,

I"m trying to read the rows of a table in chunks to process them in a background worker.
I want to ensure that each row is processed only once.

I was thinking of using the `SELECT * ... OFFSET {offset_size} LIMIT {limit_size}` functionality for this but I"m running into issues.

Some approaches I had in mind that aren't working out:
 - Try to use the transaction id to query rows created since the last processed transaction id 
      - It seems Postgres does not expose row transaction ids so this approach is not feasible
 - Rely on OFFSET / LIMIT combination to query the next chunk of data 
      - SELECT * does not guarantee ordering of rows so it's possible older rows repeat or newer rows are missed in a chunk

Can you please suggest any alternative to periodically read rows from a table in chunks while processing each row exactly once.

Thanks,
Sushrut



pgsql-hackers by date:

Previous
From: Marina Polyakova
Date:
Subject: Re: cpluspluscheck/headerscheck require build in REL_16_STABLE
Next
From: "Anton A. Melnikov"
Date:
Subject: Re: Refactoring backend fork+exec code