Question 112438 in Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Ubuntu – ModuleNotFoundError while importing Airflow DAG

I am currently working with a local installation of Airflow on Ubuntu, and this works perfectly for simple DAGs.

I now have a DAG that has dependencies on other modules, and I am stuck with a ModuleNotFound error that stops me from importing the DAG.

The my_dag.py file has a line that tries to import from the module1.py python file. I’m attempting to do this with:

from my_project.common_modules.module1 import *

I then get a Broken DAG import error that says ModuleNotFoundError: No module named ‘my_project’.

My folder structure is:

dags
| .airflowignore 
| my_project
              | __init__.py
              | common_modules
              |              |  __init__.py
              |              | module1.py
              | my_dag.py

I have been following the guide here – https://airflow.apache.org/docs/apache-airflow/2.2.0/modules_management.html.

I have checked that my PYTHONPATH and airflow.cfg file correctly reflects the ~/airflow/dags folder. The content of my .airflowignore file is simply my_project/common_modules/.* (I have tried with and without an airflowignore file). I have also tried placing the common module in various locations, including the plugins folder, with the same issue in every combination.

I would welcome assistance.

Tags:

3

Answers


  1. 0

    Try moving your module to plugins. A structure like:

    dags
| .airflowignore 
| my_project
              | my_dag.py
plugins
| my_project
              | __init__.py
              | common_modules
              |              |  __init__.py
              |              | module1.py

    And then the import should still work as:

    from my_project.common_modules.module1 import *

    This method works for me and it is how we structure imports in our Airflow environment.

    See this resource: https://airflow.apache.org/docs/apache-airflow/2.3.2/plugins.html

    Login or Signup to reply.
  2. 0

    add the "my_project" folder to sys.path before the import line

    import os
import sys

sys.path.append("my_project full path")
from my_project.common_modules.module1 import *
    Login or Signup to reply.
  3. 0 
    from my_project.common_modules.module1 import *

    Your original way of importing the module has no problem as long as airflow could identify where you put your /dags.

    You mentioned that

    ... airflow.cfg file correctly reflects the ~/airflow/dags folder

    I think you mean you defined dags_folder = ~/airflow/dags

    If that is the case, please check if your project directory is ~/airflow, instead of something like ~/my_project/airflow. Otherwise, you would need to change the config in airflow.cfg

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search