I'm working with Airflow DAG and Kubernetes for the first time. I have a Python script that connects to AWS S3 and reads some files. This works fine if I run it in a Docker container/image using bash. But when…
I'm trying to set Airflow as a scheduler for my already existing project. After opening the airflow I get error message: Broken DAG: [/opt/airflow/dags/test_dag.py] Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/opt/airflow/dags/test_dag.py", line 4,…
I am setting up the airflow in my macOS using Docker and I am encountering this below error while doing the setup. Any help would be appreciated. ERROR: Get "https://registry-1.docker.io/v2/": x509: certificate is valid for *.integration.livongo.com, integration.livongo.com, not registry-1.docker.io Thanks.
We have a bunch of Sensor tasks running in reschedule mode with the default poke_interval of 60 seconds. These tasks run for some time perfectly fine but sometimes fails and the last log I can see is that they are…
I am trying to setup a GCP connection with apache airflow 2.0 in docker from MAC OS using JSON key file, I got this error when I trigger the DAG: ERROR - [Errno 2] No such file or directory: '/Users/my_user/PycharmProjects/airflow_docker/plugins/key.json'…
I am new to Docker so I have followed the official guide here for running Airflow with it-https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html When I start Airflow all services with 'docker-compose up' in Ubuntu, I get "ERROR - Triggerer's async thread was blocked for ...…
I am using Airflow through Docker-compose. The same docker-compose.yml has an image of Postgres as well. The config looks like below - postgres: image: postgres:13 ports: - 5432:5432 environment: POSTGRES_USER: airflow POSTGRES_PASSWORD: airflow POSTGRES_DB: airflow volumes: - postgres-db-volume:/var/lib/postgresql/data healthcheck: test:…
As the title says, i want to setup Airflow that would run on a cluster (1 master, 2 nodes) using Docker swarm. Current setup: Right now i have Airflow setup that uses the CeleryExecutor that is running on single EC2.…
I just recently installed airflow 2.1.4 with docker containers, I've successfully set up the postgres, redis, scheduler, 2x local workers, and flower on the same machine with docker-compose. Now I want to expand, and set up workers on other machines.…
I am using Airflow and PostgreSQL in Docker. So I set up a PostgreSQL database on port 5433. Container (384eaa7b6efb). This is where I have my data which I want to fetch with my dag in Airflow. docker ps CONTAINER…