Question 81267 in Docker Question posted in
The official documentation can be found here.

Any Airflow configuration that will make new DAGS dropped in ./dags folder show up in the UI more quickly? – Docker

I’m trying to learn a little Airflow by running the official airflow docker compose image
[ per instructions here: https://airflow.apache.org/docs/apache-airflow/2.0.1/start/docker.html ].
I was able to launch all the services such that when I dropped my test DAG in the ./dags
folder I eventually saw my DAG in the UI and could run it.

But this took several minutes. And continues to take several minutes every time I ‘deploy’ a new dag into ./dags.

I’m wondering if there is some configuration option that would cause the scanning of the
./dags folder to happen faster so I don’t have to wait around as much. Thanks in advance !

Tags:

2

Answers


  1. 0

    As far as I know there are several reasons for slowness of your DAG’s loading.

    1. If you have a lot (thousands) of DAGs (dynamic DAGs are slower, depends on how you generate them).
    2. Your DAG is too complex – see Reducing DAG complexity.
    3. You can test your code speed – DAG Loader Test
    Login or Signup to reply.
  2. 0

    In airflow.cfg file you may want to change this, default is 5 minutes.

    dag_dir_list_interval = 300

    After changing the value, just restart webserver, scheduler, triggerer, worker docker containers i.e. without killing them or doing docker-compose down. e.g.

    docker restart your_airflow_webserver_container_id
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search