apache-kafka Questions

How to configure kafka connect jdbc sink connector for postgresql with ssl postgres authentication?

November 27, 2023
Noy Gafni
2 Answers

I have a working jdbc sink connector configuration for postgres using username and password configuration: { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "connection.url": "jdbc:postgresql://localhost:5432/postgres", "connection.user": "postgres-user", "connection.password": "postgres-pw", "topics.regex": "my-topic", "transforms": "unwrap", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "false", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false",…

VIEW QUESTION

Ubuntu – Cannot deserialize instance of `java.lang.String` out of START_OBJECT, through reference chain: java.util.LinkedHashMap["key.schema"

November 19, 2023
Vineet Mishra
2 Answers

Hi I have been stuck in a problem since two weeks, I am trying to upload a CSV in kafka partition using a CSV source connector. First, I tried on windows 11 using first.row.as.header:true, Source connector was created but it's…

VIEW QUESTION

Postgresql – Debezium Connect does not create topic

November 5, 2023
paolo_tn
2 Answers

I'm a newbie to Kafka and started experimenting. I'm using this tutorial https://medium.com/high-alpha/data-stream-processing-for-newbies-with-kafka-ksql-and-postgres-c30309cfaaf8 I ran the following docker-compose.yml version: '2' services: postgres: image: 'debezium/postgres:11' environment: POSTGRES_PASSWORD: postgres command: postgres -c config_file=/home/config/postgresql.conf ports: - '5432:5432' volumes: - 'C:postgres-kafkaconfig:/home/config' - 'C:postgres-kafkaconfigpg_hba.conf:/home/config/pg_hba.conf' -…

VIEW QUESTION

Docker – How to enable Kafka schema validation

October 12, 2023
Nico
2 Answers

I have a stack which consists of 3 Kafka instances a producer and a consumer and a schema registry. The producer sends a topic "Person" all the time and the consumer just receives it. I have created an AVRO schema…

VIEW QUESTION

Docker – How do you connect your Java Producer to a preconfigured Kafka in Red Panda?

September 18, 2023
Rigo Sarmiento
2 Answers

I'm learning Kafka via Conduktor. I ran this command on their get-started page: curl -L https://releases.conduktor.io/console -o docker-compose.yml && docker compose up -d --wait && echo "Conduktor started on http://localhost:8080" Now when I visit the console and view the connection,…

VIEW QUESTION

How can I load a Dockerfile into K8s (AKS) with a .properties file?

July 28, 2023
Matt
2 Answers

I have the following project: https://github.com/ably/kafka-connect-ably Running the dockerfile locally works perfectly well. I have tried a few methods to get it working in k8s... I have tried Kompose. This created the correct .yaml files, as well as a persistent…

VIEW QUESTION

How to handle errors when reading from Kafka and writing to PostgreSQL in Go?

July 17, 2023
Towhid Zeinali
2 Answers

I'm building a Go application that reads messages from a Kafka topic and writes them to a PostgreSQL database. I've set up a loop that reads messages from Kafka using a kafka.Reader and inserts them into the database using a…

VIEW QUESTION

Amazon web services – AWS MSK kafka Not authorized to access topics

June 20, 2023
Anum Sheraz
2 Answers

My Kafka cluster is IAM auth enabled. I am successfully able to produce and consume messages from topic test-topic2 by assuming the correct IAM role if the policy is as follows; { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowKafkaTopicWrite", "Effect":…

VIEW QUESTION

Postgresql – JDBC Sink – Value schema must be of type Struct

June 20, 2023
Rathore
2 Answers

I am trying to sink data from Kafka to Postgres and the following is what I have configured - My zookeeper, Kafka, Schema registry & Kafka Connect is working fine Even i have configured a Mongo Sink Connector which works…

VIEW QUESTION

Docker – None of my applications can see my kafka even though they are running on the same network?

June 17, 2023
Berk Sunduri
2 Answers

my docker-compose.yml file is: version: '3' services: kafka1: image: confluentinc/cp-kafka:7.3.2 volumes: - ./init-scripts/kafka:/docker-entrypoint-initdb.d ports: - "9092:9092" environment: KAFKA_ZOOKEEPER_CONNECT: zoo1:2181 KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:9093 KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL://localhost:9093 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 depends_on: - zoo1 networks: - mynetwork zoo1: image: confluentinc/cp-zookeeper:7.3.2 ports: -…

VIEW QUESTION