skip to Main Content

I’m using the Wurstmeister Docker Image to try to run a very basic single broker Kafka server just for testing my application.

It’s a simple docker-compose.yml:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - '2181:2181'
  kafka:
    image: wurstmeister/kafka
    ports:
      - '9092:9092'
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

I can enter the container, and create topics using the standard binaries, e.g.

> docker exec -it ${container} kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092
> docker exec -it ${container} kafka-topics.sh --list --bootstrap-server localhost:9092
my-topic

Displaying container information shows the port 9092 exposed:

> docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS         PORTS                                                NAMES
14b0210787cf   wurstmeister/zookeeper     "/bin/sh -c '/usr/sb…"   11 seconds ago   Up 9 seconds   22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   catalog-ingestor-service_zookeeper_1
9ecb59c1bfdd   wurstmeister/kafka         "start-kafka.sh"         11 seconds ago   Up 9 seconds   0.0.0.0:9092->9092/tcp                               catalog-ingestor-service_kafka_1

Using kcat (kafkacat), which I installed using homebrew, I try to connect and get an error:

> kcat -b localhost:9092 -L
%3|1645810235.301|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Failed to resolve 'localhost:9092': nodename nor servname provided, or not known (after 5002ms in state CONNECT)
% ERROR: Failed to acquire metadata: Local: Broker transport failure (Are the brokers reachable? Also try increasing the metadata timeout with -m <timeout>?)

I’m pretty sure I should be trying to connect on localhost:9092, but even when I ping it, it doesn’t seem to resolve:

> ping localhost:9092
ping: cannot resolve localhost:9092: Unknown host

Also, I assume I don’t need to explicitly create a network / bridge across the Kafka Broker and Zookeeper, since zookeeper:2181 already resolves that bridge?

2

Answers


  1. Well the ping with a port will not help do check whether you can reach a port or not.

    Anyway, I was having a similar issue with Confluent Kafka and I had to explicitly expose the port. See the following yaml:

    ---
    version: '2'
    services:
      zookeeper:
        image: confluentinc/cp-zookeeper:6.2.0
        container_name: zookeeper
        restart: always
        expose:
          - '2181'
        environment:
          ZOOKEEPER_SERVER_ID: 1
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
          ZOOKEEPER_INIT_LIMIT: 5
          ZOOKEEPER_SYNC_LIMIT: 2
          ZOOKEEPER_SERVERS: 0.0.0.0:2888:3888
          ZOOKEEPER_MAX_CLIENT_CNXNS: 0
        volumes:
          - /Users/pacone/broker/zookeeper-logs/data:/var/lib/zookeeper/data
          - /Users/pacone/broker/zookeeper-logs/log:/var/lib/zookeeper/log
    
      kafka-one:
        image: confluentinc/cp-kafka:6.2.0
        container_name: kafka-one
        restart: always
        expose: 
          - '9092'
        depends_on:
          - zookeeper
        ports:
          - '9092:9092'
          - '29092:29092'
    
    environment:
      KAFKA_BROKER_ID: 10
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-one:29092,EXTERNAL://localhost:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_DEFAULT_REPLICATION_FACTOR: 1
      KAFKA_NUM_PARTITIONS: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000
    volumes:
      - /Users/yourUsername/broker/kafka-logs/data:/var/lib/kafka/data
    

    With previous config I am able to connect from Java:
    enter image description here

    Cheers

    Login or Signup to reply.
  2. can you check if the 9092 port is getting used by any other process? if yes then try to change it.
    Also, try to make the docker-compose version 3

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search