Question 20458 in Mysql Question posted in
The official MySQL documentation can be found here.

How to stream mysql events to redis using debezium?

We are trying to use mysql as source db and redis as sink to build an ETL

I have specified the sink as redis using following properties

debezium.sink.type=redis
debezium.sink.redis.batch.size=500
debezium.sink.redis.address=redis:6379
debezium.sink.redis.message.format=compact

But starting the debezium is throwing the error that below two keys do not have valid values. These keys are for kafka, so how to modify this to redis?

schema.history.internal.kafka.topic
schema.history.internal.kafka.bootstrap.servers

Exact error: "The ‘schema.history.internal.kafka.topic’ value is invalid: A value is required"

Ref: https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-example-configuration

Is this even possible to have Redis as sink with mysql connector?

Tags:

2

Answers


  1. 0

    It is one of the values that are required and don’t have default value as per debezium docs, you must include the topic where it will track the DDL updates, and the kafka cluster that it will point to should be the same that the kafka-connect is using, below is the statement from the docs regarding schema.history.internal.kafka.bootstrap.servers

    A list of host/port pairs that the connector uses for establishing an initial connection to the Kafka cluster. This connection is used for retrieving the database schema history previously stored by the connector, and for writing each DDL statement read from the source database. Each pair should point to the same Kafka cluster used by the Kafka Connect process.

    Regarding the redis part, I didn’t try it myself, but as per the docs, yes it is doable, check this link (Debezium Redis Stream)

    Login or Signup to reply.
  2. 0

    Debezium Server can optionally store 3 types of data in the sink:

    1. change records
    2. schema history (needed for MySql and Oracle)
    3. Offset

    In order to store schema history in Redis, add the following configuration:

    debezium.source.schema.history.internal=io.debezium.storage.redis.history.RedisSchemaHistory

    In order to store offsets in Redis, add the following configuration:

    debezium.source.offset.storage=io.debezium.storage.redis.offset.RedisOffsetBackingStore
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search