I am very new to docker and Kafka, and have a simple kafka python publisher shown below The following are in my dockerfile: FROM python:3.10 WORKDIR /app COPY . /app RUN pip install --user pip==23.0.1 && pip install pipenv &&…
I am tyring to setup the AWS MSK connector to the DynamoDB but I cant understand how can I specify the DunamoDB key. connector.class=io.confluent.connect.aws.dynamodb.DynamoDbSinkConnector table.name.format=test_sink_table_2 confluent.topic.bootstrap.servers="someserver" tasks.max=1 topics=spk-enriched-stream aws.dynamodb.endpoint=https://dynamodb.eu-west-3.amazonaws.com confluent.topic.security.protocol=SASL_SSL confluent.topic.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; value.converter.schemas.enable=false value.converter=org.apache.kafka.connect.converters.ByteArrayConverter key.converter=org.apache.kafka.connect.storage.StringConverter confluent.topic.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler aws.dynamodb.region=eu-west-3 confluent.topic.sasl.mechanism=AWS_MSK_IAM My application…
We are trying to use mysql as source db and redis as sink to build an ETL I have specified the sink as redis using following properties debezium.sink.type=redis debezium.sink.redis.batch.size=500 debezium.sink.redis.address=redis:6379 debezium.sink.redis.message.format=compact But starting the debezium is throwing the error that…
I am trying to user a docker-compose to spin up a broker, zookeeper and 2 of my own apps. They all build and run like normal but then when I try to create a kafka producer or consumer from within…
In production, we have a microservice with two instances and consume from the same topic but with the same group id(and different clientId). As a result, if we have 5 partitions, the first instance takes three and the other one…
I need to parse data from Kafka in ClickHouse using Kafka Engine, as example, I have created a table, in https://fiddle.clickhouse.com/0e89bec6-4e76-410a-9fc4-cf58ace5f34f, CREATE TABLE json(name String, data Array(Map(String, String)) ) ENGINE = Memory; INSERT INTO JSON FORMAT JSONEachRow {"name": "asd", "data":[{"id":"1",…
From Debezium official page, there's this picture showing that multiple Debezium connector can connect to the same Kafka. So I have 2 databases, 2 Debeziums, 1 Kafka running in docker-compose, but it seems like only 1 debezium sent update to…
Is it possible to use the AWS CLI tool to list the Kafka MSK topics and display the configuration of them?
I'm trying to deploy kafka on local k8s, then I need to connect to it by application and using offset explorer so, using kubectl I created zookeeper service and deployment using this yml file apiVersion: v1 kind: Service metadata: labels:…
I was reading through the Microsoft Azure Event Hub documentation , but couldn't find mention of dead-letter-topic and/or dead-letter-queue . I know that Kafka has support for dead-letter-topic and dead-letter-queue My question is , does Microsoft Azure Event Hub have…