how to connect an s3 bucket w/ airflow
I have an airflow task where I try and load a file into an s3 bucket. I have airflow running on a Ec2 instance. Im running AF version 2.4.3 I have done pip install 'apache-airflow[amazon]' I start up my AF…
Any Airflow configuration that will make new DAGS dropped in ./dags folder show up in the UI more quickly? – Docker
I'm trying to learn a little Airflow by running the official airflow docker compose image [ per instructions here: https://airflow.apache.org/docs/apache-airflow/2.0.1/start/docker.html ]. I was able to launch all the services such that when I dropped my test DAG in the ./dags…
How Airflow DockerOperator send notification when Docker image creation error
I want to send notification when Airflow DockerOperator can't create Docker image. Notification can be sent in case of DockerOperator execution errors using on_failure_callback. To be more specific, I want to catch 2 errors. private Docker repository is not running(10.11.12.13…
Redis – How to trigger DAG in Airflow everytime an external event state is True (Event based triggering)
The basic concept of Airflow does not allow triggering a Dag on an irregular interval. Actually, I want to trigger a dag every time a new file is placed on a remote server (like HTTPS, sftp, s3 ...) But Airflow…
Azure – Insert JSON data into SQL DB using Airflow/python
I extracted data from an API using Airflow. The data is extracted from the API and saved on cloud storage in JSON format. The next step is to insert the data into an SQL DB. I have a few questions:…
Postgresql – Moving from DTL processes to DLT processes (using DBT)
I'm trying to move from a DTL script to this a DLT DBT script. The problem is that in one of my tables I also have some other DTL processes that writes in. I read https://docs.getdbt.com/guides/migration/tools/migrating-from-stored-procedures/1-migrating-from-stored-procedures but didn't figure out…
How to make a docker image/container on Google Cloud Composer and deploy it?
I have airflow dags running on Google Cloud Composer that trains machine learning models on some training data and stores the model with the best accuracy. I want to make a docker container/image that has the best model and deploy…
Redis – DAG seems to be missing from DagBag error in Airflow 2.4.0
I updated my Airflow setup from 2.3.3 to 2.4.0. and I started to get these errors on the UI DAG <dag name> seems to be missing from DagBag. Scheduler log shows ERROR - DAG < dag name> not found in…
Dynamic arrayjob size AwsBatchOperator – Amazon web services
I have a compute job I wish to scale dynamically according to charge. Since the compute is done on AWS batch, I want to use the array_size parameter, here is how I do it with the AwsBatchOperator in airflow :…
How to automatically move all DAG files to Airflow Docker and not just have only the latest added file moved and renamed to 'example_python_operator'?
PROBLEM How to automatically move all DAG files to Airflow Docker and not just have only the latest added file moved and renamed to 'example_python_operator'? Guides That I have Tried 1. docker run -d -p 8080:8080 -v /home/user/airflow/dags:/usr/local/airflow/dags puckel/docker-airflow webserver…