Question 81963 in Docker Question posted in
The official documentation can be found here.

new dags not shown on docker airflow

Problem: new dags not shown on docker airflow, no error when running airflow dags list-import-errors

Docker image: official airflow image

Dags path inside docker-compose.yaml (this is the default path):

  volumes:
    - ./dags:/opt/airflow/dags

I put the dag file inside the dags folder on the main directory, as shown below:
enter image description here

However, the dags is still not shown on both webserver UI and airflow dags list. Running airflow dags list-import-errors also yield no result.

When I open the docker terminal, I can see my dag inside the dags folder via ls command. I also tried to make the owner root by using chown, but both of my dag still not shown up on the list.

The airflow run successfully (via docker compose) as I can see example dags, but not my own dags.

Any help will be appreciated. Thanks!

Tags:

3

Answers


  1. 0

    I would try a few things:

    • rename files to sth like gcp_dag.py and python_dag.py
    • ensure import airflow is present in each file
    • ensure you create DAG object in each file
    • add __init__.py (empty) file to dags folder

    It would be also helpful to see contents of at least one of those files.

    Login or Signup to reply.
  2. 0
    1. do not name your python DAGs test_* that probably does not matter but this is usually a convention for unit tests
    2. make sure there is a DAG word in your dags (or change https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-discovery-safe-mode to False) – the DAGs can be ignored if they do not contain "DAG" or "airflow"
    3. Exec into your scheduler container and check if it can see and read the DAGs and whether the env vars are properly set airflow info run in your scheduler container (providing that it is run with the same env as the running scheduler) should show all the configuration
    4. Check if your scheduler is runnning
    5. look at the logs of the scheduler and see if it is scanning the right folders – enabling DEBUG log might be helpful https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#logging-level
    6. see if you can see the dags with airflow dags sub commands
    Login or Signup to reply.
  3. 0

    It might also be a problem with permissions. Check what user is assigned to the DAG files or better try to view the files from inside the container, like this:

    docker exec -it <container> bash
cat /opt/airflow/dags/test_gcp.py
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search