Question 84373 in Docker Question posted in
The official documentation can be found here.

Failing to connect to single Kafka broker in local Docker container

I’m using the Wurstmeister Docker Image to try to run a very basic single broker Kafka server just for testing my application.

It’s a simple docker-compose.yml:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - '2181:2181'
  kafka:
    image: wurstmeister/kafka
    ports:
      - '9092:9092'
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

I can enter the container, and create topics using the standard binaries, e.g.

> docker exec -it ${container} kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092
> docker exec -it ${container} kafka-topics.sh --list --bootstrap-server localhost:9092
my-topic

Displaying container information shows the port 9092 exposed:

> docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS         PORTS                                                NAMES
14b0210787cf   wurstmeister/zookeeper     "/bin/sh -c '/usr/sb…"   11 seconds ago   Up 9 seconds   22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   catalog-ingestor-service_zookeeper_1
9ecb59c1bfdd   wurstmeister/kafka         "start-kafka.sh"         11 seconds ago   Up 9 seconds   0.0.0.0:9092->9092/tcp                               catalog-ingestor-service_kafka_1

Using kcat (kafkacat), which I installed using homebrew, I try to connect and get an error:

> kcat -b localhost:9092 -L
%3|1645810235.301|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Failed to resolve 'localhost:9092': nodename nor servname provided, or not known (after 5002ms in state CONNECT)
% ERROR: Failed to acquire metadata: Local: Broker transport failure (Are the brokers reachable? Also try increasing the metadata timeout with -m <timeout>?)

I’m pretty sure I should be trying to connect on localhost:9092, but even when I ping it, it doesn’t seem to resolve:

> ping localhost:9092
ping: cannot resolve localhost:9092: Unknown host

Also, I assume I don’t need to explicitly create a network / bridge across the Kafka Broker and Zookeeper, since zookeeper:2181 already resolves that bridge?

Tags:

2

Answers


  1. 0

    Well the ping with a port will not help do check whether you can reach a port or not.

    Anyway, I was having a similar issue with Confluent Kafka and I had to explicitly expose the port. See the following yaml:

    ---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.0
    container_name: zookeeper
    restart: always
    expose:
      - '2181'
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
      ZOOKEEPER_SERVERS: 0.0.0.0:2888:3888
      ZOOKEEPER_MAX_CLIENT_CNXNS: 0
    volumes:
      - /Users/pacone/broker/zookeeper-logs/data:/var/lib/zookeeper/data
      - /Users/pacone/broker/zookeeper-logs/log:/var/lib/zookeeper/log

  kafka-one:
    image: confluentinc/cp-kafka:6.2.0
    container_name: kafka-one
    restart: always
    expose: 
      - '9092'
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
      - '29092:29092'

environment:
  KAFKA_BROKER_ID: 10
  KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
  KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-one:29092,EXTERNAL://localhost:9092
  KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  KAFKA_DEFAULT_REPLICATION_FACTOR: 1
  KAFKA_NUM_PARTITIONS: 1
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
  KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
  KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 6000
volumes:
  - /Users/yourUsername/broker/kafka-logs/data:/var/lib/kafka/data

    With previous config I am able to connect from Java:
    enter image description here

    Cheers

    Login or Signup to reply.
  2. 0

    can you check if the 9092 port is getting used by any other process? if yes then try to change it.
    Also, try to make the docker-compose version 3

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search