Question 83429 in Docker Question posted in
The official documentation can be found here.

Unable to install confluent-kafka: "fatal error: librdkafka/rdkafka.h: No such file or directory" – Docker

I am using the confluent-kafka Python client in my project. I’m trying to create a Docker image with this client.

I am facing the following error:-

#11 8.015 [pipenv.exceptions.InstallError]:       In file included from /tmp/pip-install-so_whhii/confluent-kafka_9d9553bf46cf489bb25fcb2ac7698747/src/confluent_kafka/src/Admin.c:17:
#11 8.015 [pipenv.exceptions.InstallError]:       /tmp/pip-install-so_whhii/confluent-kafka_9d9553bf46cf489bb25fcb2ac7698747/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: librdkafka/rdkafka.h: No such file or directory
#11 8.015 [pipenv.exceptions.InstallError]:          23 | #include <librdkafka/rdkafka.h>
#11 8.015 [pipenv.exceptions.InstallError]:             |          ^~~~~~~~~~~~~~~~~~~~~~
#11 8.015 [pipenv.exceptions.InstallError]:       compilation terminated.
#11 8.015 [pipenv.exceptions.InstallError]:       error: command '/usr/bin/gcc' failed with exit code 1
#11 8.016 [pipenv.exceptions.InstallError]:       [end of output]

Based on my search it is related to Apple M1 build for librdkafka.

Tags:

7

Answers


  1. 0

    Prebuilt binary wheels of confluent-kafka-python for your Apple M1 seems to not exist.

    Thus, in order to install the package, pip tries to build it itself from source. But librdkafka looks like to not be installed as per librdkafka/rdkafka.h: No such file or directory error you got.

    You can find specific instructions to install from source for different platforms here.

    Login or Signup to reply.
  2. 0

    The Confluent engineers are obviously very focused on their paying customers, and many, many months after the release of Python 3.10, they still haven’t released 3.10 wheels that include the binaries for the package.

    https://github.com/confluentinc/confluent-kafka-python/issues/1219

    You have two choices. 1, run Python 3.9; 2, install librdkafka-dev via apt before installing confluent-kafka (for a debian base image).

    The issue is the same with Apple M1 – i.e, there are no pre-built wheels for the platform containing the binaries, but it is much easier to fix (by installing librdkafka-dev) if you are trying to build a amd64 Linux image.

    Login or Signup to reply.
  3. 0

    Please refer to this Github issue. The problem with M1 is that Homebrew is installed in a different location and so these variables need to be added to the environment by including these lines in your .zshrc file

    export C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include
export LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib
    Login or Signup to reply.
  4. 0

    I solved this problem trying with Python 3.9.13 on a virtual environment.

    Login or Signup to reply.
  5. 0

    On a non-M1 mac, Python 3.10.2, I solved this with brew install librdkafka.

    Login or Signup to reply.
  6. 0

    If you are installing confluent-kafka inside Docker container on M1, this helped me.

    RUN apt-get update && 
  apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && 
  cd /tmp && git clone https://github.com/edenhill/librdkafka.git && 
  cd librdkafka && git checkout tags/v1.9.0 && 
  ./configure && make && make install && 
  cd ../ && rm -rf librdkafka
    Login or Signup to reply.
  7. 0

    I added these to my docker-compose.yml (according to https://stackoverflow.com/a/74020511/20575677 and https://github.com/confluentinc/confluent-kafka-python/issues/65#issuecomment-269964346)

    RUN apt-get update && 
  apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && 
  cd /tmp && git clone https://github.com/edenhill/librdkafka && 
  cd librdkafka && git checkout tags/v2.0.2 && 
  ./configure && make && make install && 
  ldconfig &&
  cd ../ && rm -rf librdkafka

RUN pip install confluent-kafka==2.0.2
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search