I am writing my first Kafka based project and experience some issues with integration. Kafka and Kafka Connect. Here is my docker-compose.yaml version: '3.3' services: database: image: 'postgres:16.2-alpine' restart: always shm_size: 128mb ports: - 5432:5432 env_file: - ./database/.env volumes: -…
Hello guys i wanna ask some question about CDC mysql to kafka with azure database for mysql. I already following this tutorial: https://techcommunity.microsoft.com/t5/azure-database-for-mysql-blog/cdc-in-azure-database-for-mysql-flexible-server-using-kafka/ba-p/2780943 but got stuck while creating kafka conncetor on this part, and the error: {"error_code":400,"message":"Connector configuration is invalid…
I have a JDBC sink connector retrieving data from a Postgres source connector, and I need to copy the data from my "parametros" source table to my "parametros_sistema" target table. I'm using the org.apache.kafka.connect.transforms.ReplaceField$Value transformer to target the specific columns…
I have a working jdbc sink connector configuration for postgres using username and password configuration: { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "connection.url": "jdbc:postgresql://localhost:5432/postgres", "connection.user": "postgres-user", "connection.password": "postgres-pw", "topics.regex": "my-topic", "transforms": "unwrap", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "false", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false",…
Hi I have been stuck in a problem since two weeks, I am trying to upload a CSV in kafka partition using a CSV source connector. First, I tried on windows 11 using first.row.as.header:true, Source connector was created but it's…
I want to continuously update a document's field named value {'_id': 'count', 'value':0} by a certain number. My MongoSinkConnector has document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy.ProvidedInValueStrategy writemodel.strategy=com.mongodb.kafka.connect.sink.writemodel.strategy.UpdateOneTimestampsStrategy I'm using a python script to produce messages to the appropiate topic self._aio_producer.produce( topic='mongo', value=json.dumps( { "_id":"count", "$inc":{"value":len(task['payload'].split(','))}…
I have the following project: https://github.com/ably/kafka-connect-ably Running the dockerfile locally works perfectly well. I have tried a few methods to get it working in k8s... I have tried Kompose. This created the correct .yaml files, as well as a persistent…
I am trying to sink data from Kafka to Postgres and the following is what I have configured - My zookeeper, Kafka, Schema registry & Kafka Connect is working fine Even i have configured a Mongo Sink Connector which works…
I have a Kafka cluster deployed by Strimzi and Apicurio Registry for Kafka schema registry. I am hoping to use AvroConverter in the JDBC sink connector to sink data from Kafka to TimescaleDB. Here is my Kafka Connect Dockerfile: FROM…
I am tyring to setup the AWS MSK connector to the DynamoDB but I cant understand how can I specify the DunamoDB key. connector.class=io.confluent.connect.aws.dynamodb.DynamoDbSinkConnector table.name.format=test_sink_table_2 confluent.topic.bootstrap.servers="someserver" tasks.max=1 topics=spk-enriched-stream aws.dynamodb.endpoint=https://dynamodb.eu-west-3.amazonaws.com confluent.topic.security.protocol=SASL_SSL confluent.topic.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; value.converter.schemas.enable=false value.converter=org.apache.kafka.connect.converters.ByteArrayConverter key.converter=org.apache.kafka.connect.storage.StringConverter confluent.topic.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler aws.dynamodb.region=eu-west-3 confluent.topic.sasl.mechanism=AWS_MSK_IAM My application…