Hi I have been stuck in a problem since two weeks, I am trying to upload a CSV in kafka partition using a CSV source connector. First, I tried on windows 11 using first.row.as.header:true, Source connector was created but it's…
I'm a newbie to Kafka and started experimenting. I'm using this tutorial https://medium.com/high-alpha/data-stream-processing-for-newbies-with-kafka-ksql-and-postgres-c30309cfaaf8 I ran the following docker-compose.yml version: '2' services: postgres: image: 'debezium/postgres:11' environment: POSTGRES_PASSWORD: postgres command: postgres -c config_file=/home/config/postgresql.conf ports: - '5432:5432' volumes: - 'C:postgres-kafkaconfig:/home/config' - 'C:postgres-kafkaconfigpg_hba.conf:/home/config/pg_hba.conf' -…
I have a stack which consists of 3 Kafka instances a producer and a consumer and a schema registry. The producer sends a topic "Person" all the time and the consumer just receives it. I have created an AVRO schema…
I'm learning Kafka via Conduktor. I ran this command on their get-started page: curl -L https://releases.conduktor.io/console -o docker-compose.yml && docker compose up -d --wait && echo "Conduktor started on http://localhost:8080" Now when I visit the console and view the connection,…
I have the following project: https://github.com/ably/kafka-connect-ably Running the dockerfile locally works perfectly well. I have tried a few methods to get it working in k8s... I have tried Kompose. This created the correct .yaml files, as well as a persistent…
I'm building a Go application that reads messages from a Kafka topic and writes them to a PostgreSQL database. I've set up a loop that reads messages from Kafka using a kafka.Reader and inserts them into the database using a…
My Kafka cluster is IAM auth enabled. I am successfully able to produce and consume messages from topic test-topic2 by assuming the correct IAM role if the policy is as follows; { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowKafkaTopicWrite", "Effect":…
I am trying to sink data from Kafka to Postgres and the following is what I have configured - My zookeeper, Kafka, Schema registry & Kafka Connect is working fine Even i have configured a Mongo Sink Connector which works…
my docker-compose.yml file is: version: '3' services: kafka1: image: confluentinc/cp-kafka:7.3.2 volumes: - ./init-scripts/kafka:/docker-entrypoint-initdb.d ports: - "9092:9092" environment: KAFKA_ZOOKEEPER_CONNECT: zoo1:2181 KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:9093 KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL://localhost:9093 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 depends_on: - zoo1 networks: - mynetwork zoo1: image: confluentinc/cp-zookeeper:7.3.2 ports: -…
Ran the command docker-compose up -d enter image description here got this error : Error response from daemon: lease "moby-image-sha256:c2740b69f111bd711c867e337c12bab4b3d720c987e0d09549fe95e6badc6ba8": not found No idea what this means. I am on a M1.This is my docker files : version: '3.9' services:…