Question 80729 in Docker Question posted in
The official documentation can be found here.

Connecting to MariaDB from docker-compose Airflow – No module named 'MySQLdb' error

I’ve modified the docker-compose for Airflow (apache/airflow:2.5.0-python3.10) and added in a MariaDB service: to emulate the target DB for a project in dev.

In the _PIP_ADDITIONAL_REQUIREMENTS I’ve included pymsql and I am attempting to create a MySQL Connection under Admin | Connections and using the MySQL connector to bridge over to Maria (before asking I used host.docker.internal as the host name in the Connections| Add Connection field for host).

apache-airflow-providers-mysql is already installed as part of the official docker compose for Airflow (and I checked it manually on the container.)

However, every time I hit the Test button I get a No module named 'MySQLdb' which I assumed the pip extras install took care of.

Also, I am assuming here I can merely use MySQL over the wire to connect to the mariadb via python but if there additional libraries or installs needed (for example, libmariadb3 libmariadb-dev python3-mysqldb or similar to be installed, please let me know. I assume someone else has had to do this already, though perhaps not from docker… =] I am trying to avoid building my own image and use the existing docker-compose, but if it’s unavoidable and I need to do a build ., lemme know.).

I am concerned this may be due to the fact I am on an M1 Mac laptop running this (as 2 requirements, I’ve now dug down and researched have a platform_machine != "aarch64" against them 8-/ Surely there is a way around this though, yeah? Esp with docker? (do I need to do a custom build?). Also, do not understand why these particular components would not work on arm64 at this point that macs have been chipped like that over 2 years.

thanks!
PS> Maria is not supported as a backend in Airflow but Connections to it as a database should work much like anything else that can connect via sqlalchemy etc.

Set up the Connection in Airflow and then installed the extra pip for pymysql. However, I still seem to be getting the same No module named ‘MySQLdb’ error on test connection.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    The issue appears to be one where Airflow's MySQL operators have a != aarch64 for the Apple OSX and my machine was an M1 chipped laptop.

    How did I get around this? In my docker-compose for the airflow components I set a directive of platform: x86_64 in the service(s) and then rebuild the container. After that, I had no problem connecting with the MySQL connector to the MariaDB that we had as a target.

    note: this greatly increased the startup time for the docker compose instance (from like 20s to 90s-ish, but works). As a solution for dev, worked great.


  2. 0

    You need to install the MySql provider for Airflow:

    pip install apache-airflow-providers-mysql

    or add it to your requirements file, it should install all what you need to use MySql/MariaDB as connection.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search