I am trying to user a docker-compose to spin up a broker, zookeeper and 2 of my own apps.
They all build and run like normal but then when I try to create a kafka producer or consumer from within my app, it throws an exception saying there are no available brokers.
Here is my docker-compose.yml:
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
container_name: zookeeper
networks:
- kafka_network
environment:
ZOOKEEPER_CLIENT_PORT: '2181'
kafka:
container_name: kafka
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
networks:
- kafka_network
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENERS: EXTERNAL_SAME_HOST://:29092,INTERNAL://:9092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_CONFLUENT_TOPIC_REPLICATION_FACTOR: 3
csir-api:
build:
context: app/
depends_on:
- kafka
networks:
- kafka_network
csir-api-entry:
build:
context: app-entry/
ports:
- 8080:8080
depends_on:
- kafka
networks:
- kafka_network
networks:
kafka_network:
name: kafka_docker_example_net
Here is how I am trying to connect within my app:
consumer = KafkaConsumer(
['clickup-task-created', 'clickup-task-updated', 'sheets-task-updated', 'sheets-task-created'],
bootstrap_servers=['kafka:9092'],
auto_offset_reset='earliest',
enable_auto_commit=True,
group_id='my-group',
value_deserializer=lambda x: loads(x.decode('utf-8')))
producer = KafkaProducer(
bootstrap_servers=['kafka:9092'],
value_serializer=lambda x:
dumps(x).encode('utf-8'),
)
Here is the error being thrown:
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
I expected the producer and consumer to connect on the same network and allow me to consume and produce to the kafka broker.
2
Answers
according to your config I would try
Using
kafka:9092
is correct from another container.If you’re just running
docker compose up
, then the Kafka container takes a few seconds to properly startup, meanwhile, you python code isn’t seeming to wait, therefore will try to connect immediately, and fail.Since none of your topics exist, I suggest using AdminClient functions in a retry loop to try and create them while waiting for the brokers to become available