I have a Kafka cluster deployed by Strimzi and Apicurio Registry for Kafka schema registry. I am hoping to use AvroConverter in the JDBC sink connector to sink data from Kafka to TimescaleDB. Here is my Kafka Connect Dockerfile: FROM…
I am tyring to setup the AWS MSK connector to the DynamoDB but I cant understand how can I specify the DunamoDB key. connector.class=io.confluent.connect.aws.dynamodb.DynamoDbSinkConnector table.name.format=test_sink_table_2 confluent.topic.bootstrap.servers="someserver" tasks.max=1 topics=spk-enriched-stream aws.dynamodb.endpoint=https://dynamodb.eu-west-3.amazonaws.com confluent.topic.security.protocol=SASL_SSL confluent.topic.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; value.converter.schemas.enable=false value.converter=org.apache.kafka.connect.converters.ByteArrayConverter key.converter=org.apache.kafka.connect.storage.StringConverter confluent.topic.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler aws.dynamodb.region=eu-west-3 confluent.topic.sasl.mechanism=AWS_MSK_IAM My application…
I want to control the throughtput of a JDBC source kafka connector. I have a lot of data stored in a PostgreSQL table and I wand to ingest it into a Kafka topic. However I would like to avoid a…
I have a database in MongoDB. That is updating every minute from various user PCs. Now I want to store the data into Amazon S3 bucket (preferable in Parquet, otherwise CSV). But I do not want to store the full…
As the title states, I'm using debezium Postgres source connector and I would like MongoDB sink connector to group kafka topics in different collection and databases (also different dbs to isolate unrelated data) according to their names. While inquiring I…
I following the Kafka quick start guide. I'm stuck as step 6 though, Import/Export your data as streams of Events... https://kafka.apache.org/documentation/ The error indicates that Kafka can't find the connector. I've added it to the connect-standalone.properties file in a variety…
I am running a docker file to execute a debezium service as entrypoint. It works fine and executes debezium service when I use the exact file path as parameter to Entrypoint command in my docker file command in docker file…
We have several collections in Mongo based on n tenants and want the kafka connector to only watch for specific collections. Below is my mongosource.properties file where I have added the pipeline filter to listen only to specific collections.It works…
I am using this Github repo and folder path I found: https://github.com/entechlog/kafka-examples/tree/master/kafka-connect-standalone The issue I am having is that, as a matter of access control, I must specify my group ID by adding a prefix to it, let's call it…
I am trying to build a Kafka-Connect image in Docker: FROM confluentinc/cp-kafka-connect RUN confluent-hub install --no-prompt wepay/kafka-connect-bigquery:1.6.1 RUN confluent-hub install --no-prompt confluentinc/connect-transforms:latest RUN mkdir -p /usr/share/landoop-plugins COPY kafka-connect-redis-1.2.2-2.1.0-all.jar /usr/share/landoop-plugins/ but it runs as appuser Step 4/4 : RUN id --->…