I'm creating a dev environment to use airflow for testing. I'm using the docker-compose.yaml file available on Airflow website. I would like to know if it is possible to set my connections and variables in this file. I know that…
I'm trying to run an airflow image using docker. A while ago it was working normally. However, I ran other applications on localhost without using docker (using VisualStudio) and when I rerun my airflow after that, localhost was no longer…
I created a custom image with the following Dockerfile: FROM apache/airflow:2.1.1-python3.8 USER root RUN apt-get update && apt-get -y install gcc gnupg2 && curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - && curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list RUN apt-get update && ACCEPT_EULA=Y apt-get…
I'm trying to set up airflow instance using docker-compose as described in official docs and I'm stuck at airflow-init part. It looks like there is no connectivity between containers, but I don't know how to fix it. I use literally…
Problem Definition I am trying to use two docker-compose.yml files (each in separate directories) on the same host machine, one for Airflow and the other for another application. I have put Airflow's containers in the same named network as my…
I’m setting up a distributed Airflow cluster where everything else except the celery workers are run on one host and processing is done on several hosts. The airflow2.0 setup is configured using the yaml file given at the Airflow documentation…
My team and I are on Airflow v2.1.0 using the Celery executor with Redis. Recently we’ve noticed some jobs are occasionally running until we kick them (many hours, sometimes days—basically until someone notices). There doesn’t seem to be a particular…
I am using the following docker-compose image, I got this image from: https://github.com/apache/airflow/blob/main/docs/apache-airflow/start/docker-compose.yaml version: "3" x-airflow-common: &airflow-common image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.0-python3.7} environment: &airflow-common-env AIRFLOW__CORE__EXECUTOR: CeleryExecutor AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0 AIRFLOW__CORE__FERNET_KEY: "" AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: "true" AIRFLOW__CORE__LOAD_EXAMPLES: "false" AIRFLOW__API__AUTH_BACKEND: "airflow.api.auth.backend.basic_auth" _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-apache-airflow-providers-apache-spark} volumes:…
I run Airflow using Docker-compose and I wanted to create a DAG with a DockerOperator in order to run a docker container. The thing is I keep on getting the same error when looking at the logs for the DAG…
I am trying to install Airflow 2.0.1 with ansible on CentOS8 machine. Python version 3.8.1. I made pip 20.2.4 as suggested in Airflow docs. I am using postgresql and airflow db check is successful. But the db init task gives…