PROBLEM How to automatically move all DAG files to Airflow Docker and not just have only the latest added file moved and renamed to 'example_python_operator'? Guides That I have Tried 1. docker run -d -p 8080:8080 -v /home/user/airflow/dags:/usr/local/airflow/dags puckel/docker-airflow webserver…
Problem: The start date of my DAG is not being set properly, can anyone tell me why? Here is sample code: default_args = { "owner": "hello", "email_on_failure": "false", "retries": 1, "retry_delay": timedelta(minutes=1), "start_date": datetime(2022, 7, 20), "catchup": True, "schedule_interval": "@weekly",…
Problem: new dags not shown on docker airflow, no error when running airflow dags list-import-errors Docker image: official airflow image Dags path inside docker-compose.yaml (this is the default path): volumes: - ./dags:/opt/airflow/dags I put the dag file inside the dags…
I am trying to use class EcsCreateClusterOperator in MWAA but I get the following error: Broken DAG: [/usr/local/airflow/dags/etl_basic_ecs/etl_ecs_basic_dag.py] Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/local/airflow/dags/etl_basic_ecs/etl_ecs_basic_dag.py", line 5, in <module> from airflow.providers.amazon.aws.operators.ecs import (…
I am trying to create a glue job with this configuration 'NumberOfWorkers': 2, 'WorkerType': 'G.1X'. Here's my code for job creation. job_name = "glue_job" submit_glue_job = AwsGlueJobOperator( task_id="glue_job", job_name=job_name, wait_for_completion=True, # num_of_dpus=10, retry_limit=0, script_location=f"s3://bucket/etl.py", s3_bucket=GLUE_EXAMPLE_S3_BUCKET, iam_role_name=GLUE_CRAWLER_ROLE.split("/")[-1], create_job_kwargs={ 'GlueVersion': '3.0', 'NumberOfWorkers':…
I am deploying Airflow with official Helm chart and trying to understand why it requires stateful set for worker deployment. When it makes perfect sense for redis and postgtre I am not sure why this is requirement for worker
Quick question, do i need to docker compose up on airflow when i amend a secret in kubectl? I've changed a password using the command line and kubectl in vscode and just want to know if it is necessary to…
I am currently working with a local installation of Airflow on Ubuntu, and this works perfectly for simple DAGs. I now have a DAG that has dependencies on other modules, and I am stuck with a ModuleNotFound error that stops…
Does anybody know how to unblock admin>configuration in UI of airflow which is running with docker-compose? When I try to open it it shows '# Your Airflow administrator chose not to expose the configuration, most likely for security reasons.'
I realise this is a duplicate of thousands of posts across the internet, but I have only found solutions for version <2.0, and those solutions do not apply to >2.0. Basically, I have installed Apache Airflow on an Ubuntu system.…