Question 117112 in Mongodb Question posted in
The official documentation can be found here.

Kafka connect- Mongodb Sink Connector

I am new to kafka connector. I have been explore about it about a week. I have used create and update the mongodb via mongodb connector curl commands. I am bit struggling to understand the concept and implementation of below.

  1. We are registering curl command to connector at every time with unique name before producing the message. How it will be automated?. For example, If I pass the data from my application to producer should I call the curl command for each and every request?

2)I need to maintain the history collection based on that I need to pass two collection and two topics (one for updating and one for creating). How will I manage with curl configuration.
I will paste my curl update configuration below,

curl -X POST -H "Content-Type: application/json" -d '{"name":"test-students-update",
 "config":{"topics":"topicData",
 "connector.class":"com.mongodb.kafka.connect.MongoSinkConnector",
 "tasks.max":"1",
 "connection.uri":"mongodb://localhost:27017",
 "database":"quickstart",
 "collection":"topicData",
 "key.converter":"org.apache.kafka.connect.storage.StringConverter",
 "value.converter":"org.apache.kafka.connect.storage.StringConverter",
 "key.converter.schemas.enable":"false",
 "value.converter.schemas.enable":"false",
 "document.id.strategy":"com.mongodb.kafka.connect.sink.processor.id.strategy.BsonOidStrategy",
 "document.id.strategy":"com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy",
 "value.projection.list":"tokenNumber",
 "value.projection.type":"whitelist",
 "writemodel.strategy":"com.mongodb.kafka.connect.sink.writemodel.strategy.UpdateOneTimestampsStrategy"
}}' localhost:8083/connectors
Tags:

2

Answers


  1. 0

    Not sure what do you mean by automating CURL command for your MongoDB Sink Connector, and what is the need of running CURL command every time. Kindly clarify.

    The existing MongoDB Sink connector can easily be integrated with the Confluent Hub and can run as a standalone service to serve the purpose of data UPSERT.

    You can have a look at https://www.mongodb.com/docs/kafka-connector/current/sink-connector/fundamentals/#std-label-kafka-sink-fundamentals

    Login or Signup to reply.
  2. 0

    every time with unique name before producing the message

    This is not necessary. Post the connector once and it’ll start a Kafka consumer and wait for data, just like any other Kafka client.

    pass the data from my application to producer should I call the curl command for each and every request

    As stated, no.

    How it will be automated

    You don’t necessarily need to use curl. If you’re using Kubernetes, there are CRDs for KafkaConnect. Otherwise, Terraform providers work with Connect API as well. Or you can continue to use curl in some ci/cd pipeline, but it only needs ran once to start the connector

    need to pass two collection and two topics (one for updating and one for creating).

    The collection field in the connector can only reference one collection. Therefore, you’d need two separate connectors, and therefore all Kafka events would be inserted or updated to those individual collections, and not reference on another unless your schema model uses ObjectId references

    Alternatively, redesign your producer to send to one topic, then inserts and updates (and deletes) can happen based on the key of the record into one collection

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search