Question 114162 in PostgreSQL Question posted in
The official documentation can be found here.

Postgresql – Topic not created when using debezium/connect Docker image

A Kafka topic for my table is not created when using the debezium/connect Docker image. Here’s how I’m starting the container:

docker run -it --rm --name debezium -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs 
-e OFFSET_STORAGE_TOPIC=my-connect-offsets  -e BOOTSTRAP_SERVERS=192.168.56.1:9092 
-e CONNECT_NAME=my-connector -e CONNECT_CONNECTOR_CLASS=io.debezium.connector.postgresql.PostgresConnector 
-e CONNECT_TOPIC_PREFIX=my-prefix -e CONNECT_DATABASE_HOSTNAME=host.docker.internal -e CONNECT_DATABASE_PORT=5432 
-e CONNECT_DATABASE_USER=postgres -e CONNECT_DATABASE_PASSWORD=root -e DATABASE_SERVER_NAME=mydb 
-e CONNECT_DATABASE_DBNAME=mydb -e CONNECT_TABLE_INCLUDE_LIST=myschema.my_table -e CONNECT_PLUGIN_NAME=pgoutput 
debezium/connect

I’ve tried using CONNECT__ instead of CONNECT_, but I get the same results. A topic for the table is not created if I use the API:

curl -H 'Content-Type: application/json' 127.0.0.1:8083/connectors --data '
{
  "name": "prism",  
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector", 
    "topic.prefix": my-connector",
    "database.hostname": "host.docker.internal", 
    "database.port": "5432", 
    "database.user": "postgres", 
    "database.password": "root", 
    "database.server.name": "mydb", 
    "database.dbname" : "mydb", 
    "table.include.list": "myschema.my_table",
    "plugin.name": "pgoutput"
  }
}'

The topics my-connect-configs and my-connect-offsets, specified by CONFIG_STORAGE_TOPIC and OFFSET_STORAGE_TOPIC, are created.

http://localhost:8083/connectors/my-connector/status shows this:

{"name":"my-connector","connector":{"state":"RUNNING","worker_id":"172.17.0.3:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"172.17.0.3:8083"}],"type":"source"}

I was able to create a topic when using bin/connect-standalone.sh instead of the Docker image as per this question.

Automatic topic creation is enabled and I don’t see any errors/warnings in the log.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    The issue was that the underlying table didn't have any data, and, therefore, the topic was not created. The topic is created either if the table has data when the connector is started or if rows are added while the connector is running.


  2. 0
    1. Check kafka connect container logs, what msg it shows when it tries to insert the data to kafka cluster.
    2. You need to enable auto topic creation in kafka broker config, checkout this doc
    3. Make sure that table exist in the db and "table.include.list": "myschema.my_table", is correct. For experimentation you can remove this config temporary.

    You can use UI platform created by redpanda team to manage topic, broker and kafka-connect config – here

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search