I am writing integration tests and created kafka topic using docker command docker exec kafka-broker kafka-topics.sh --create --bootstrap-server localhost:9093 --partitions 1 --replication-factor 1 --topic test-topic in github workflow, Topic is created successfully. Using kafka topic in testcase to write data…
I have a Kafka cluster deployed by Strimzi and Apicurio Registry for Kafka schema registry. I am hoping to use AvroConverter in the JDBC sink connector to sink data from Kafka to TimescaleDB. Here is my Kafka Connect Dockerfile: FROM…
I would like to use Kafka in KRaft mode instead of Zookeeper on AWS, since Kraft mode seems to be the future of Kafka. As I understand, basic solution from AWS for Kafka is AWS MSK. But in MSK documentation…
I am very new to docker and Kafka, and have a simple kafka python publisher shown below The following are in my dockerfile: FROM python:3.10 WORKDIR /app COPY . /app RUN pip install --user pip==23.0.1 && pip install pipenv &&…
I am tyring to setup the AWS MSK connector to the DynamoDB but I cant understand how can I specify the DunamoDB key. connector.class=io.confluent.connect.aws.dynamodb.DynamoDbSinkConnector table.name.format=test_sink_table_2 confluent.topic.bootstrap.servers="someserver" tasks.max=1 topics=spk-enriched-stream aws.dynamodb.endpoint=https://dynamodb.eu-west-3.amazonaws.com confluent.topic.security.protocol=SASL_SSL confluent.topic.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; value.converter.schemas.enable=false value.converter=org.apache.kafka.connect.converters.ByteArrayConverter key.converter=org.apache.kafka.connect.storage.StringConverter confluent.topic.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler aws.dynamodb.region=eu-west-3 confluent.topic.sasl.mechanism=AWS_MSK_IAM My application…
We are trying to use mysql as source db and redis as sink to build an ETL I have specified the sink as redis using following properties debezium.sink.type=redis debezium.sink.redis.batch.size=500 debezium.sink.redis.address=redis:6379 debezium.sink.redis.message.format=compact But starting the debezium is throwing the error that…
I am trying to user a docker-compose to spin up a broker, zookeeper and 2 of my own apps. They all build and run like normal but then when I try to create a kafka producer or consumer from within…
In production, we have a microservice with two instances and consume from the same topic but with the same group id(and different clientId). As a result, if we have 5 partitions, the first instance takes three and the other one…
I need to parse data from Kafka in ClickHouse using Kafka Engine, as example, I have created a table, in https://fiddle.clickhouse.com/0e89bec6-4e76-410a-9fc4-cf58ace5f34f, CREATE TABLE json(name String, data Array(Map(String, String)) ) ENGINE = Memory; INSERT INTO JSON FORMAT JSONEachRow {"name": "asd", "data":[{"id":"1",…
From Debezium official page, there's this picture showing that multiple Debezium connector can connect to the same Kafka. So I have 2 databases, 2 Debeziums, 1 Kafka running in docker-compose, but it seems like only 1 debezium sent update to…