I am writing my first Kafka based project and experience some issues with integration. Kafka and Kafka Connect. Here is my docker-compose.yaml
version: '3.3'
services:
database:
image: 'postgres:16.2-alpine'
restart: always
shm_size: 128mb
ports:
- 5432:5432
env_file:
- ./database/.env
volumes:
- ./database/data:/var/lib/postgresql/data/
- ./database/init.sql:/docker-entrypoint-initdb.d/init.sql
networks:
priv-aeroflot-pilot-net:
ipv4_address: 172.16.254.3
depends_on:
- kafka
- kafka-connect
zookeeper:
image: zookeeper:3.7.0
ports:
- "2181:2181"
volumes:
# - ./kafka/zookeeper/data:/var/lib/zookeeper/data
- ./kafka/zookeeper/data:/data
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
networks:
priv-aeroflot-pilot-net:
ipv4_address: 172.16.254.5
kafka:
image: apache/kafka:3.7.0
ports:
- 9092:9092
volumes:
# - ./kafka/connect-plugins:/opt/kafka/connect-plugins
- ./kafka/config:/opt/kafka/custom-config
- ./kafka/logs:/var/lib/kafka/data
environment:
KAFKA_BROKER_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
networks:
priv-aeroflot-pilot-net:
ipv4_address: 172.16.254.2
depends_on:
- zookeeper
# kafka-connect:
# image: apache/kafka:3.7.0
# command: bash -c "/opt/kafka/bin/connect-standalone.sh /opt/kafka/config/connect-standalone.properties"
# volumes:
# - ./kafka/connect-plugins:/opt/kafka/connect-plugins
# - ./kafka/connect-standalone.properties:/opt/kafka/config/connect-standalone.properties
# - ./kafka/connect-standalone-worker.properties:/opt/kafka/config/connect-standalone-worker.properties
# networks:
# priv-aeroflot-pilot-net:
# ipv4_address: 172.16.254.4
# depends_on:
# - kafka
# adminer:
# image: adminer
# restart: always
# ports:
# - 8080:8080
# networks:
# priv-aeroflot-pilot-net:
# ipv4_address: 172.16.254.1
networks:
priv-aeroflot-pilot-net:
driver: bridge
ipam:
config:
- subnet: 172.16.254.0/28
Kafka connect is temporarily commented out to fix first issues Kafka Zookeeper vs KRaft conflict.
At this moment the Kafka service exits with issue:
The kafka configuration file appears to be for a legacy cluster. Formatting is only supported for clusters in KRaft mode.
Zookeeper
is going to be removed in the next release of Apache Kafka, but in 3.7 release it is just marked deprecated. Zookeeper
and KRaft
are mutually exclusive mechanisms and should not work together, but neither I omit KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
from KRaft
configuration or omit configs related to KRaft
I receive an exception.
ConfigException: Missing required configuration `zookeeper.connect` which has no default value.
or like another above.
General context is I want my Kafka data being mirrored into postgres. I already have working Kafka service queue where is producer put messages and consumers reads them and I want now to have this messages stored in PostgreSQL db in addition to Kafka queue. I have t write PostgreSQL sink connector and choose either Zookeeper
or KRaft
orchestrate mode. I avoid usage proprietary solutions like Confluent and keep as pure as it possible on the plain Kafka, althought fully open-sourced solutions are ok.
Could you help me please figure out with my current issue? Just for case of simplicity I am fine with Zookeeper
and postpone with KRaft
integration for future.
2
Answers
I haven't resolved this issue on docker based
apache/kafka
images, but I have solved my task by migrating onconfluentinc/cn-kafka
docker based images. Here is a finaldocker-compose.yaml
:Confluent shares this images under
Apache 2.0
license which is ok for current goal, however I am still interested to find a solution on pureApache Kafka
.It seems kafka is already ditching zookeeper in their docker image.
The Kafka docker image uses kraft config file: https://github.com/apache/kafka/blob/3.7.0/docker/jvm/Dockerfile#L81