I have a dockerized spring-boot web service that needs to produce to a Kafka topic. I'm running Kafka with Zookeeper locally on Ubuntu. My web service is running in Docker on the same machine. I have confirmed with netstat -tlpn…
i want to send this simple pyspark dataframe to kafka and i done everything but i always get an error. i try it with simple python producer script and it works and the pyspark read stream work my only problem…
I have a JDBC sink connector retrieving data from a Postgres source connector, and I need to copy the data from my "parametros" source table to my "parametros_sistema" target table. I'm using the org.apache.kafka.connect.transforms.ReplaceField$Value transformer to target the specific columns…
I am trying to connect kafka and aws lambda on a different hosting provider. When I connect via AWS EC2, it works fine because they are in the same VPC, but when I try to connect kafka and lambda on…
I have an event schema stored in JSON format in Azure Event Hubs Schema Registry. I need to read that schema in my C# code, without hard coding the schema in the code. Schema needs to be read into a…
I'm having authentication issues when I try to connect a producer following the instruction of the helm chart installation: I'm installing this kafka helm chart: https://artifacthub.io/packages/helm/bitnami/kafka?modal=install I can perfectly install it with helm install, and this is the output from…
I have a working jdbc sink connector configuration for postgres using username and password configuration: { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "connection.url": "jdbc:postgresql://localhost:5432/postgres", "connection.user": "postgres-user", "connection.password": "postgres-pw", "topics.regex": "my-topic", "transforms": "unwrap", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "false", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false",…
Hi I have been stuck in a problem since two weeks, I am trying to upload a CSV in kafka partition using a CSV source connector. First, I tried on windows 11 using first.row.as.header:true, Source connector was created but it's…
I'm a newbie to Kafka and started experimenting. I'm using this tutorial https://medium.com/high-alpha/data-stream-processing-for-newbies-with-kafka-ksql-and-postgres-c30309cfaaf8 I ran the following docker-compose.yml version: '2' services: postgres: image: 'debezium/postgres:11' environment: POSTGRES_PASSWORD: postgres command: postgres -c config_file=/home/config/postgresql.conf ports: - '5432:5432' volumes: - 'C:postgres-kafkaconfig:/home/config' - 'C:postgres-kafkaconfigpg_hba.conf:/home/config/pg_hba.conf' -…
I have a stack which consists of 3 Kafka instances a producer and a consumer and a schema registry. The producer sends a topic "Person" all the time and the consumer just receives it. I have created an AVRO schema…