Thoughts on how to avoid a massive integer update. - Mailing list pgsql-general

From Fehrle, Brian
Subject Thoughts on how to avoid a massive integer update.
Date
Msg-id C7D3EE42-66B9-44AF-83F4-CB7F0F42486E@comscore.com
Whole thread Raw
Responses Re: Thoughts on how to avoid a massive integer update.  (Adrian Klaver <adrian.klaver@aklaver.com>)
Re: Thoughts on how to avoid a massive integer update.  (Rob Sargent <robjsargent@gmail.com>)
Re: Thoughts on how to avoid a massive integer update.  (Rob Sargent <robjsargent@gmail.com>)
Re: Thoughts on how to avoid a massive integer update.  ("Peter J. Holzer" <hjp-pgsql@hjp.at>)
Re: Thoughts on how to avoid a massive integer update.  ("David G. Johnston" <david.g.johnston@gmail.com>)
List pgsql-general

Hi all,

 

This is a shot in the dark in hopes to find a magic bullet to fix an issue I have, I can’t personally think of any solution myself.

 

I have a database with hundreds of terabytes of data, where every table has an integer column referencing a small table. For reasons out of my control and cannot change, I NEED to update every single row in all these tables, changing the integer value to a different integer.

 

Since I have to deal with dead space, I can only do a couple tables at a time, then do a vacuum full after each one.
Another option is to build a new table with the new values, then drop the old one and swap in the new, either way is very time consuming.

 

Initial tests suggest this effort will take several months to complete, not to mention cause blocking issues on tables being worked on.

 

Does anyone have any hackery ideas on how to achieve this in less time? I was looking at possibly converting the integer column type to another that would present the integer differently, like a hex value, but everything still ends up requiring all data to be re-written to disk. In a well designed database (I didn’t design it :) ), I would simply change the data in the referenced table (200 total rows), however the key being referenced isn’t just an arbitrary ID, it’s actual ‘data’, and must be changed.

 

Thanks for any thoughts or ideas,

  • Brian F

pgsql-general by date:

Previous
From: Adrian Klaver
Date:
Subject: Re: 12.2: Howto check memory-leak in worker?
Next
From: Adrian Klaver
Date:
Subject: Re: Thoughts on how to avoid a massive integer update.