Question posted in
Some documentation to get you started can be found here.

Migrate a (storm+nimbus) cluster to a new Zookeeper, without loosing the information or having downtime – Apache

I have a nimbus+storm cluster using Zookeeper, and I wish to move my cluster and point it to a new Zookeeper. Do you know if this is possible? Can I keep all the information of the old zookeeper and save it in the new one? Is it possible to do it without downtime?

I have looked in the internet for this procedure but I have not found much.

Would it be as simples as change the storm.yml file in both the master . and worker nodes? Do I need a restart afterwards?

# storm.zookeeper.servers:
#     - "server1"
#     - "server2"
Tags:

2

Answers


  1. 0

    If you just change storm.yml, you’d be pointing Storm at a new empty Zookeeper cluster, and it will be like you just installed Storm from scratch. More likely, you want to grow your Zookeeper cluster to include your new machines, then update storm.yml to point at the new machines, then shrink the cluster to exclude the machines you want to move away from. That way, your Zookeeper quorum is preserved even though you’ve moved to other physical machines.

    This is easier to do on Zookeeper 3.5 with dynamic reconfiguration http://zookeeper.apache.org/doc/r3.5.5/zookeeperReconfig.html. I’m unsure whether Storm will run on Zookeeper 3.5, but you may consider investigating whether you can upgrade to 3.5 before growing/shrinking the cluster.

    Otherwise you will have to do a rolling restart to add the new Zookeeper nodes, then do another one to remove the old machines once the cluster has stabilized.

    Login or Signup to reply.
  2. 0

    Let me suggest a hack here. This was a script provided by microsoft for migration on HD Insight cluster , but you can change it and use it for your need.

    The script can be downloaded from : https://github.com/hdinsight/hdinsight-storm-examples/tree/master/tools/zkdatatool-1.0 and you can read more about it here :
    https://blogs.msdn.microsoft.com/azuredatalake/2017/02/24/restarting-storm-eventhub/

    I have used it in the past when i had to migrate some stuff between PaaS clusters and i can confirm it works ok!

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search