skip to Main Content

I am trying to user a docker-compose to spin up a broker, zookeeper and 2 of my own apps.
They all build and run like normal but then when I try to create a kafka producer or consumer from within my app, it throws an exception saying there are no available brokers.

Here is my docker-compose.yml:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    networks: 
      - kafka_network
    environment:
      ZOOKEEPER_CLIENT_PORT: '2181'

  kafka:
    container_name: kafka
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    networks: 
      - kafka_network
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENERS: EXTERNAL_SAME_HOST://:29092,INTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CONFLUENT_TOPIC_REPLICATION_FACTOR: 3

  csir-api:
    build:
      context: app/
    depends_on:
      - kafka
    networks:
      - kafka_network

  csir-api-entry:
    build:
      context: app-entry/
    ports: 
      - 8080:8080
    depends_on:
      - kafka
    networks:
      - kafka_network

networks:
  kafka_network:
    name: kafka_docker_example_net

Here is how I am trying to connect within my app:

consumer = KafkaConsumer(
    ['clickup-task-created', 'clickup-task-updated', 'sheets-task-updated', 'sheets-task-created'],
     bootstrap_servers=['kafka:9092'],
     auto_offset_reset='earliest',
     enable_auto_commit=True,
     group_id='my-group',
     value_deserializer=lambda x: loads(x.decode('utf-8')))

    producer = KafkaProducer(
        bootstrap_servers=['kafka:9092'],
        value_serializer=lambda x: 
        dumps(x).encode('utf-8'),
    )

Here is the error being thrown:

kafka.errors.NoBrokersAvailable: NoBrokersAvailable

I expected the producer and consumer to connect on the same network and allow me to consume and produce to the kafka broker.

2

Answers


  1. according to your config I would try

    bootstrap_servers=['kafka:29092']
    
    Login or Signup to reply.
  2. Using kafka:9092 is correct from another container.

    If you’re just running docker compose up, then the Kafka container takes a few seconds to properly startup, meanwhile, you python code isn’t seeming to wait, therefore will try to connect immediately, and fail.

    Since none of your topics exist, I suggest using AdminClient functions in a retry loop to try and create them while waiting for the brokers to become available

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search