Problem: new dags not shown on docker airflow, no error when running airflow dags list-import-errors Docker image: official airflow image Dags path inside docker-compose.yaml (this is the default path): volumes: - ./dags:/opt/airflow/dags I put the dag file inside the dags…
I realise this is a duplicate of thousands of posts across the internet, but I have only found solutions for version <2.0, and those solutions do not apply to >2.0. Basically, I have installed Apache Airflow on an Ubuntu system.…
We have a bunch of Sensor tasks running in reschedule mode with the default poke_interval of 60 seconds. These tasks run for some time perfectly fine but sometimes fails and the last log I can see is that they are…
My team and I are on Airflow v2.1.0 using the Celery executor with Redis. Recently we’ve noticed some jobs are occasionally running until we kick them (many hours, sometimes days—basically until someone notices). There doesn’t seem to be a particular…
Everyone, I am using Airflow(v.1.10.12) for one of the project to schedule the Jobs daily. Wanted to execute multiple DAG’s(14) in parallel. Have updated the concurrency parameters in cfg file. However, have observed the behavior that: Scheduler starts the execution…
I'm running the quickstart for KubernetesPodOperator secret using the link below : https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator Code used below : from airflow import models from airflow.contrib.kubernetes import secret from airflow.contrib.operators import kubernetes_pod_operator # A Secret is an object that contains a small amount…
I try to implement Apache Airflow with the CeleryExecutor. For the database I use Postgres, for the celery message queue I use Redis. When using LocalExecutor everything works fine, but when I set the CeleryExecutor in the airflow.cfg and want…
Airflow scheduler process crashes if we start turning on the DAG and Trigger DAG from Airflow web-server. Airflow Version - **v1.10.4 Redis server v=5.0.7 executor = CeleryExecutor broker_url = 'redis://:password@redis-host:2287/0' sql_alchemy_conn = postgresql+psycopg2://user:password@host/dbname result_backend = 'db+postgresql://user:password@host/dbname' Crashes with Below Error…