> On Jul 14, 2024, at 12:33 PM, BIPUL KHAN <bipul364@gmail.com> wrote:
>
> Hello,
> We have developed and run a large-scale mobile financial system using a microservice architecture, which includes 110
databases(big and small). We use PostgreSQL as the primary database inside Kubernetes via the CrunchyData operator.
> Now I would like to share my problem. Our data storage has grown significantly since 2020, and we are using Ceph
(rook-Ceph).We now have 21 TB of data.
> Our current situation is critical as we urgently need to move our data center. Our source and destination data
centershave a 1 GB/s data transfer rate. We attempted to add a new Ceph node with the same cluster, which started data
rebalancing,but it’s taking a significant amount of time. This directly and severely impacts our primary transactions,
whichinvolve reading and writing to the database.
> What is the best option for moving data from one data center to a new one? We would appreciate your expertise. Would
youhappento know how to move it any other way?
> It is mentioned that the two different data centers are different Kubernetes clusters.
> I would appreciate some ideas. Feel free to let me know if you need any other adjustments!
> Let me know if there are any further changes you'd like!
Plan to migrate off Ceph as soon as possible. It is simply not suited for this--yes, I am speaking from experience. The
rebalancingwill hammer you when you can least afford it. WekaFS, Pure Storage, heck NFS. Or node-local storage, XFS or
ZFS.Really, just about anything else. (It might help if you clarify whether "move our data center" really means what I
assumed,that you own hardware and are moving it, or whether you're going to another cloud provider.
You're probably going to have to migrate your databases one at a time--even if you temporarily upgraded your bandwidth
to10Gbps you'd have too long of a down time. So, one at a time, set up a standby streaming replica, then when they're
allpopulated and up-to-date, take a very brief downtime (seconds) to switch over. This assumes that 1Gbps can actually
handleyour stream of normal updates with enough room left over to be transferring the base files. (Current versions of
Crunchysupport the notion of cross-cluster replication.)
