Problem is we have multiple databases and some of the table can have millions on rows so queries like
update my_table set column1='some'
takes up to 30 mins and more and sometimes transaction fails due connection lost or some.
The idea is some how split original query into parts in runtime and run them.
I hope liquibase have solution to do it but didnt find it.
2
Answers
I do include these type of statements in my Liquibase changelogs whenever I want to ensure that an update is propagating across all my environments. So this a valid usecase in my experience.
Usually, what I do, is shard up the update by using the id column, like
update my_table set column1='some' where id<500000
update my_table set column1='some' where id>=500000
You’ll have to decide what where clause makes the most sense for your data.
IMO, in case of
PostgreSQL
the idea of "splitting original query into parts" within single connection/transaction does not make sense at all (I do not see any performance benefits there), moreover it is not clear what does the statement "sometimes transaction fails due connection lost or some" mean – if you are experiencing infrastructure issues you are supposed to resolve them via configuring/tuning infrastructure.The liquibase way of performing such migrations is following:
I. define SQL migration as:
II. operator is supposed to perform following steps:
BTW, if we are talking about theoretical possibilities of getting performance benefits I would think about something like: