I'm learning Kafka via Conduktor. I ran this command on their get-started page: curl -L https://releases.conduktor.io/console -o docker-compose.yml && docker compose up -d --wait && echo "Conduktor started on http://localhost:8080" Now when I visit the console and view the connection,…
I have the following project: https://github.com/ably/kafka-connect-ably Running the dockerfile locally works perfectly well. I have tried a few methods to get it working in k8s... I have tried Kompose. This created the correct .yaml files, as well as a persistent…
I'm building a Go application that reads messages from a Kafka topic and writes them to a PostgreSQL database. I've set up a loop that reads messages from Kafka using a kafka.Reader and inserts them into the database using a…
My Kafka cluster is IAM auth enabled. I am successfully able to produce and consume messages from topic test-topic2 by assuming the correct IAM role if the policy is as follows; { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowKafkaTopicWrite", "Effect":…
I am trying to sink data from Kafka to Postgres and the following is what I have configured - My zookeeper, Kafka, Schema registry & Kafka Connect is working fine Even i have configured a Mongo Sink Connector which works…
my docker-compose.yml file is: version: '3' services: kafka1: image: confluentinc/cp-kafka:7.3.2 volumes: - ./init-scripts/kafka:/docker-entrypoint-initdb.d ports: - "9092:9092" environment: KAFKA_ZOOKEEPER_CONNECT: zoo1:2181 KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:9093 KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL://localhost:9093 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 depends_on: - zoo1 networks: - mynetwork zoo1: image: confluentinc/cp-zookeeper:7.3.2 ports: -…
Ran the command docker-compose up -d enter image description here got this error : Error response from daemon: lease "moby-image-sha256:c2740b69f111bd711c867e337c12bab4b3d720c987e0d09549fe95e6badc6ba8": not found No idea what this means. I am on a M1.This is my docker files : version: '3.9' services:…
I am writing integration tests and created kafka topic using docker command docker exec kafka-broker kafka-topics.sh --create --bootstrap-server localhost:9093 --partitions 1 --replication-factor 1 --topic test-topic in github workflow, Topic is created successfully. Using kafka topic in testcase to write data…
I have a Kafka cluster deployed by Strimzi and Apicurio Registry for Kafka schema registry. I am hoping to use AvroConverter in the JDBC sink connector to sink data from Kafka to TimescaleDB. Here is my Kafka Connect Dockerfile: FROM…
I would like to use Kafka in KRaft mode instead of Zookeeper on AWS, since Kraft mode seems to be the future of Kafka. As I understand, basic solution from AWS for Kafka is AWS MSK. But in MSK documentation…