I am trying to replicate Kafka cluster with MirrorMaker 2.0. I am using following mm2.properties: name = mirror-site1-site2 topics = .* connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector tasks.max = 1 plugin.path=/usr/share/java/kafka/plugin clusters = site1, site2 # for demo, source and target clusters are…
I read the consumer group example in here(provided by Shopify/sarama), and I wonder that how would I know how many consumers are in a consumer group, are there any ways I could control the number of consumers? Thanks a lot
We have a 3 node Kafka cluster deployment with 5 topics and 6 partitions per topic. we have configured the replication factor =3 , we are seeing very strange problem that number of file descriptor have been crossed the ulimit…
this is my first post on Stackoverflow, i hope i didnt choose the wrong section. Context : Kafka HEAP size is configured on following file : /etc/systemd/system/kafka.service With following parameter : Environment="KAFKA_HEAP_OPTS=-Xms6g -Xmx6g" OS is "CentOS Linux release 7.7.1908". Kafka…
I'm trying to set up Apache Kafka to communicate between two virtual machines running CentOS on the same network. I originally set up a Kafka producer and consumer on one machine and everything was running smoothly. I then set up…
I want to stream data using spark from Kafka topic in Horton works. I have started the zookeeper and Kafka server. Then I have successfully built a maven project in Eclipse with all the streaming dependencies. Given below is the…
I want to stream data using spark from Kafka topic in Horton works. I have started the zookeeper and Kafka server. Then I have successfully built a maven project in Eclipse with all the streaming dependencies. Given below is the…
I want to stream data using spark from Kafka topic in Horton works. I have started the zookeeper and Kafka server. Then I have successfully built a maven project in Eclipse with all the streaming dependencies. Given below is the…
I need to stream live tweets from twitter API and then analyse them. I should use kafka to get tweets or spark streaming directly or both ?
I just started the testing of Kafka Stream to Spark using Pyspark library. I have been running the whole setup on Jupyter Notebook. I am trying to get data from the Twitter Streaming. Twitter Streaming Code: import json import tweepy…