Question 334964 in Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Ubuntu – Dockerized spring boot app can't access host machine's kafka instance through localhost

I have a dockerized spring-boot web service that needs to produce to a Kafka topic. I’m running Kafka with Zookeeper locally on Ubuntu. My web service is running in Docker on the same machine. I have confirmed with netstat -tlpn that the Kafka broker is active on port 9092.

If I run docker run my-image-name:latest --network="host", the Kafka producer cannot access the bootstrap broker.

KafkaService.java

@Component
public class KafkaService {

    private String bootstrapServerUrl;

    private String topicName;

    private KafkaProducer<String, HandleCaptureKafkaMessage> kafkaProducer;

        // create Producer properties
    public KafkaService(@Value("${kafka-bootstrap-server-url}") String bootstrapServerUrl, @Value("${kafka-topic-name}") String topicName) {
        this.bootstrapServerUrl = bootstrapServerUrl;
        this.topicName = topicName;
        System.out.println("SERVERURL " + bootstrapServerUrl);
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServerUrl);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, HandleCaptureKafkaMessageSerializer.class.getName());

        kafkaProducer = new KafkaProducer<>(properties);
    }

    public void sendMessage(HandleCaptureKafkaMessage handleCaptureKafkaMessage) {
        kafkaProducer.send(new ProducerRecord<>(topicName, null, handleCaptureKafkaMessage), null);
    }

}

Output log

2024-02-05T19:13:00.425Z  INFO 1 --- [           main] o.a.k.clients.producer.ProducerConfig    : ProducerConfig values:
        acks = -1
        auto.include.jmx.reporter = true
        batch.size = 16384
        bootstrap.servers = [localhost:9092]
        buffer.memory = 33554432
...
2024-02-05T19:13:00.498Z  INFO 1 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Instantiated an idempotent producer.
2024-02-05T19:13:00.554Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.6.1
2024-02-05T19:13:00.559Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 5e3c2b738d253ff5
2024-02-05T19:13:00.560Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1707160380540
2024-02-05T19:13:00.830Z  INFO 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Node -1 disconnected.
2024-02-05T19:13:00.832Z  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-02-05T19:13:00.832Z  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

Any ideas?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Following the instructuions in Olakunle Abiola's comment, setting listeners=PLAINTEXT://0.0.0.0:9092 in Kafka's server.properties seems to have worked!


  2. 0

    From what you’ve explained, this seems to be happening because localhost or 127.0.0.1 inside the Docker container refers to the container itself, not the host machine where Kafka is running.

    Since you’ve tried --network="host" without success, it’s important to note that the this option works on Linux hosts but does not work the same on Windows because of the way networking is implemented in Docker Desktop for Windows.

    Instead of using localhost or 127.0.0.1, use the IP address of your WSL2 instance where Kafka is running. To find the WSL2 IP address, you can run the following command from within your WSL2 terminal:

    ip addr show eth0

    Look for the inet line which gives you the IP address, something like 192.168.x.x. Use this IP address as your Kafka bootstrap server URL in your application properties:

    kafka-bootstrap-server-url=192.168.x.x:9092

    If this doesn’t work, you can use the docker special DNS name host.docker.internal which resolves to the internal IP address used by the host. This DNS is intended for such cases where you need to access a service running on the host machine from within a Docker container. Update your Kafka bootstrap server address to this:

    kafka-bootstrap-server-url=host.docker.internal:9092

    UPDATE

    ————————————

    Can you check your Kafka’s server.properties file, typically found in /etc/kafka or /opt/kafka-*, depending on how it was installed.

    Set the listeners property to listen on all interfaces or the specific host IP: listeners=PLAINTEXT://0.0.0.0:9092.

    You’ll need to restart Kafka for this to take effect.

    Also, in your Spring Boot application’s configuration, set the kafka-bootstrap-server-url to either *localhost:9092 or :9092, depending on whichever you set in your properties file.

    And then give this another go :

    docker run --network="host" my-image-name:latest

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search