I am trying to add a running instance of MinIO to Airflow connections, I thought it should be as easy as this setup in the GUI (never mind the exposed credentials, this is a blocked of environment and will be…
I have an airflow task where I try and load a file into an s3 bucket. I have airflow running on a Ec2 instance. Im running AF version 2.4.3 I have done pip install 'apache-airflow[amazon]' I start up my AF…
I'm trying to learn a little Airflow by running the official airflow docker compose image [ per instructions here: https://airflow.apache.org/docs/apache-airflow/2.0.1/start/docker.html ]. I was able to launch all the services such that when I dropped my test DAG in the ./dags…
I want to send notification when Airflow DockerOperator can't create Docker image. Notification can be sent in case of DockerOperator execution errors using on_failure_callback. To be more specific, I want to catch 2 errors. private Docker repository is not running(10.11.12.13…
The basic concept of Airflow does not allow triggering a Dag on an irregular interval. Actually, I want to trigger a dag every time a new file is placed on a remote server (like HTTPS, sftp, s3 ...) But Airflow…
I extracted data from an API using Airflow. The data is extracted from the API and saved on cloud storage in JSON format. The next step is to insert the data into an SQL DB. I have a few questions:…
I'm trying to move from a DTL script to this a DLT DBT script. The problem is that in one of my tables I also have some other DTL processes that writes in. I read https://docs.getdbt.com/guides/migration/tools/migrating-from-stored-procedures/1-migrating-from-stored-procedures but didn't figure out…
I have airflow dags running on Google Cloud Composer that trains machine learning models on some training data and stores the model with the best accuracy. I want to make a docker container/image that has the best model and deploy…
I updated my Airflow setup from 2.3.3 to 2.4.0. and I started to get these errors on the UI DAG <dag name> seems to be missing from DagBag. Scheduler log shows ERROR - DAG < dag name> not found in…
I have a compute job I wish to scale dynamically according to charge. Since the compute is done on AWS batch, I want to use the array_size parameter, here is how I do it with the AwsBatchOperator in airflow :…