I've modified the docker-compose for Airflow (apache/airflow:2.5.0-python3.10) and added in a MariaDB service: to emulate the target DB for a project in dev. In the _PIP_ADDITIONAL_REQUIREMENTS I've included pymsql and I am attempting to create a MySQL Connection under Admin…
I have airflow dags running on Google Cloud Composer that trains machine learning models on some training data and stores the model with the best accuracy. I want to make a docker container/image that has the best model and deploy…
I am currently working with a local installation of Airflow on Ubuntu, and this works perfectly for simple DAGs. I now have a DAG that has dependencies on other modules, and I am stuck with a ModuleNotFound error that stops…
I realise this is a duplicate of thousands of posts across the internet, but I have only found solutions for version <2.0, and those solutions do not apply to >2.0. Basically, I have installed Apache Airflow on an Ubuntu system.…
I just recently installed airflow 2.1.4 with docker containers, I've successfully set up the postgres, redis, scheduler, 2x local workers, and flower on the same machine with docker-compose. Now I want to expand, and set up workers on other machines.…