Question posted in
Some documentation to get you started can be found here.

How to scale out apache atlas

There is no info provided in atlas document on how to scale it.

Apache atlas is connected to cassandra or hbase in the backend which can scale out ,but I dont know how apache atlas engine ( rest web-service and request processor ) can scale out.

I can install multiple instances of it on different machine and have load balancer in front of it to fan out the request. But would this model help ? Does it do any kind of locking and do db transaction, so that this model would not work.

Does someone know how apache atlas scales out ?

Thanks.

Tags:

2

Answers


  1. 0

    So Apache Atlas runs Kafka as the message queue under the covers, and in my experience, the way they have designed the Kafka queue (consumer group that says you should ONLY have ONE consumer) is the choke point.

    Not only that, when you look at the code, the consumer has a poll time for the broker of 1 sec hard coded into the consumer. Put these two together, and that means that if the consumer can’t process the messages from the various producers (HIVE, Spark, etc) within that second, the broker then disengages the ONLY consumer, and waits for a non-existent consumer to pick up messages…

    I need to design something similar, but this is as far as I have got…

    Hope that helps somewhat…

    Login or Signup to reply.
  2. 0

    Please refer to this page. http://atlas.apache.org/#/HighAvailability

    Atlas does not support actual horizontal scale-out.
    All the requests are handled by the ‘Active instance’. the ‘Passive instances’ just forward all the requests to the ‘Active instance’.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search