I’m new to Kafka and I’m trying to run a Kafka service on my local machine and use it to transfer some data from one .NET project to another.
I’m using docker-compose.yml file to create two docker containers for zookeeper and Kafka from wurstmeister.
In Kafka definition in environment variables there is KAFKA_ADVERTISED_HOST_NAME which I set to 127.0.0.1.
In docker hub of wurstmeister/kafka says I quote
"modify the KAFKA_ADVERTISED_HOST_NAME in docker-compose.yml to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host IP if you want to run multiple brokers.)".
When I set my topic to have more than 1 replica and more than 1 partition I get this message
"Error while executing topic command : Replication factor: 2 larger than available brokers: 1.".
What is the right IP address to define in KAFKA_ADVERTISED_HOST_NAME which will allow me to get more than 1 broker?
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
KAFKA_CREATE_TOPICS: "simpletalk_topic:2:2"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
2
Answers
it should be ip address of your machine.
Linux: use
ifconfig
command and your IP will beinet <IP_ADDRESS>
in windows you will get this by using
ipconfig
Note – Running multiple brokers on one machine does not provide true fault tolerance.
You only need one broker for this.
First, suggest reading https://github.com/wurstmeister/kafka-docker/wiki/Connectivity
KAFKA_ADVERTISED_HOST_NAME
is deprecated. Don’t use it.It has been replaced by
KAFKA_ADVERTISED_LISTENERS
, which can both contain${DOCKER_HOST_IP:-127.0.0.1}
(or your host IP fromifconfig
) since this is the IP your client will use (from the host).Since brokers are clients of themselves, you also need to advertise the container names. And, if you want to containerize your client, those addresses would also be used by your app.
Beyond Kafka networking configs,
KAFKA_BROKER_ID
needs to be different between each, and your error is suggesting you need to override all the other replication factor broker configs.All in all.
Set
bootstrap.servers="localhost:9092,localhost:9093"
Further reading – Connect to Kafka running in Docker
Example usage
Note that topic creation doesn’t seem to automatically work. You should ideally be using your application to actually create / check topic existence.
producing some data… and consuming it