I would like to start the Confluent control center using docker as prescribed in the document: https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html This is the docker-compose.yaml file they provided; --- version: '2' services: zookeeper: image: confluentinc/cp-zookeeper:5.5.0 hostname: zookeeper container_name: zookeeper ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT:…
I am building Kafka CDC but following the document, it runs many docker-run commands. I want to put it all into a docker-compose.yml but I fail at 1 last command I can not convert to The below are the commands…
I am consuming Avro data coming from Debezium I made the kafka consumer as follows: The Java POJO import lombok.Data; @Data public class Shop { Long shopId; Double latitude, longitude; String name; String phoneNumber; String placeId; double rating; String website;…
I am trying to expose cp-kafka brokers publicly using the ingress-nginx and I happen to see this Stack Overflow question. The answers only shows one broker is exposed outside the cluster. Say, If I have 3 brokers running, how can…
my question is rather specific, so I will be ok with a general answer, which will point me in the right direction. Description of the problem: I want to deliver specific task data from multiple producers to a particular consumer…
I am trying to build a Kafka-Connect image in Docker: FROM confluentinc/cp-kafka-connect RUN confluent-hub install --no-prompt wepay/kafka-connect-bigquery:1.6.1 RUN confluent-hub install --no-prompt confluentinc/connect-transforms:latest RUN mkdir -p /usr/share/landoop-plugins COPY kafka-connect-redis-1.2.2-2.1.0-all.jar /usr/share/landoop-plugins/ but it runs as appuser Step 4/4 : RUN id --->…
I've been trying to retrieve events by its position using Microsoft.Azure.EventHubs. I've been told that there is a way to calculate an event position using its offset or sequenceNumber, so everytime I'm adding an event to an EventBatch, I cache…
I need to send data to an external api, but this API has a limit of requests per endpoint(i.e: 60 requests per minute). The data come from Kafka, then every message goes to redis(because I can send a request with…
I was trying to make kafka consumer which would collect messages for a particular amount of time after that I could manually commit with the messages that has been collected. But I could not find a method or api from…
is there any way, in Kafka, to produce a message once several related messages have been consumed ? (without having to manually control it at the application code...) The use case would be to pick a huge file, split it…