Thread: Best Practices for Managing Schema Changes Dynamically with libpq

Best Practices for Managing Schema Changes Dynamically with libpq

From

Sasmit Utkarsh

Date:

03 December, 23:43:46

Dear PostgreSQL Community Team,

I am working on a project that uses libpq along with C language to interact with PostgreSQL, and we face challenges with managing schema changes dynamically in production while avoiding downtime. Specifically, we need guidance on handling table structure changes/additions without tightly coupling these changes to application updates.

Current Approach:
Schema changes are deployed first, followed by application updates to align with the new structure.

Challenges:
Ensuring application stability during the transitional phase when the schema and code are not fully in sync.
Handling table structure changes (e.g., adding new columns) dynamically without requiring immediate code changes.

Questions:
Are there recommended best practices for managing such schema changes with libpq?
How can we efficiently handle table additions/updates while keeping the application and database in sync dynamically?

I would appreciate any guidance, patterns, or examples that can help us implement a robust solution.

Thank you for your time and support!

Regards,

Sasmit Utkarsh

+91-7674022625

Re: Best Practices for Managing Schema Changes Dynamically with libpq

From

Ron Johnson

Date:

04 December, 00:10:05

On Tue, Dec 3, 2024 at 12:44 PM Sasmit Utkarsh <utkarshsasmit@gmail.com> wrote:

[snip]

How can we efficiently handle table additions/updates while keeping the application and database in sync dynamically?

Enumerate all relevant column names in SELECT and INSERT statements. That way, the application still works when you add columns or alter(*) data types. Dropping columns will still break your app.

*Altering to from a numeric type to a text type might still kill your app, if it can't convert the string into your app's int or float type..

Death to <Redacted>, and butter sauce.

Don't boil me, I'm still alive.

<Redacted> lobster!

Re: Best Practices for Managing Schema Changes Dynamically with libpq

From

Adrian Klaver

Date:

04 December, 02:50:54

On 12/3/24 09:43, Sasmit Utkarsh wrote:
> Dear PostgreSQL Community Team,
> 
> I am working on a project that uses libpq along with C language to 
> interact with PostgreSQL, and we face challenges with managing schema 
> changes dynamically in production while avoiding downtime. Specifically, 
> we need guidance on handling table structure changes/additions without 
> tightly coupling these changes to application updates.
> 
> *Current Approach:*
> Schema changes are deployed first, followed by application updates to 
> align with the new structure.
> 
> *Challenges:*
> Ensuring application stability during the transitional phase when the 
> schema and code are not fully in sync.
> Handling table structure changes (e.g., adding new columns) dynamically 
> without requiring immediate code changes.
> 
> *Questions:*
> Are there recommended best practices for managing such schema changes 
> with libpq?

I use Sqitch(https://sqitch.org/). You have to squint but it is libpq, 
of a sort, as it uses psql to do its changes.

> How can we efficiently handle table additions/updates while keeping the 
> application and database in sync dynamically?

There is way too many variations that enter into the above to give a 
complete concrete answer in anything less then a short book.

My general rule for this is to create a map of the process in outline 
form. Personally I still think better on paper and I pull out a legal 
pad and pencil and write out a work flow that goes from where I am to 
where I want to be. This starts with the 10000 foot view that I then 
drill down in to get the specific actions. I use a pencil as the drill 
down process often uncovers flaws in the 10000 foot view. This by the 
way was a method my 7th grade math teacher taught the class back way 
back when.

> 
> I would appreciate any guidance, patterns, or examples that can help us 
> implement a robust solution.
> 
> Thank you for your time and support!
> 
> Regards,
> Sasmit Utkarsh
> +91-7674022625

-- 
Adrian Klaver
adrian.klaver@aklaver.com

Re: Best Practices for Managing Schema Changes Dynamically with libpq

From

Erik Wienhold

Date:

06 December, 08:06:03

On 2024-12-03 18:43 +0100, Sasmit Utkarsh wrote:
> I am working on a project that uses libpq along with C language to interact
> with PostgreSQL, and we face challenges with managing schema changes
> dynamically in production while avoiding downtime. Specifically, we need
> guidance on handling table structure changes/additions without tightly
> coupling these changes to application updates.
> 
> *Current Approach:*
> Schema changes are deployed first, followed by application updates to align
> with the new structure.
> 
> *Challenges:*
> Ensuring application stability during the transitional phase when the
> schema and code are not fully in sync.
> Handling table structure changes (e.g., adding new columns) dynamically
> without requiring immediate code changes.

What you're looking for is the "Expand and Contract" pattern[1][2].  The
transitional phase between expand and contract has to support both old
and new code until the old code is migrated as well.  How you keep the
schema compatible with the old code for some time depends on the kind of
schema changes.  Some use cases from the top of my head:

1) expand:     add unconstrained columns
   transition: adapt code to use new columns
   contract:   add constraints

2) expand:     rename tables/columns
   transition: add (updatable) views that expose the old names until the
               code is adapted to the new names
   contract:   drop views

3) expand:     add columns with constraints
   transition: backfill new columns with triggers
   contract:   drop triggers

[1] https://www.tim-wellhausen.de/papers/ExpandAndContract/ExpandAndContract.html
[2] https://martinfowler.com/articles/evodb.html#everything_refactoring

-- 
Erik