OK, I am probably very stupid but anyways; How can I install additional pip packages via the docker-compose file of airflow? I am assuming that their should be a standard functionality to pick up a requirements.txt or something. When inspecting…
I get below error when tried to run the following breeze build command to build airflow docker. I clone the git airflow master branch to build this image. Build Command: ./breeze build-image --production-image --python 3.7 --install-airflow-version 2.0.0 --additional-extras=jdbc --additional-python-deps="pandas" --additional-runtime-apt-deps="default-jre-headless"…
I am using this helm chart to deploy airflow https://github.com/apache/airflow/tree/master/chart Apache Airflow version: 2.0.0 Kubernetes version: v1.19.4 What happened: I get this error when try to execute tasks using kubernetes [2021-01-14 19:39:17,628] {dagbag.py:440} INFO - Filling up the DagBag from…
Everyone, I am using Airflow(v.1.10.12) for one of the project to schedule the Jobs daily. Wanted to execute multiple DAG’s(14) in parallel. Have updated the concurrency parameters in cfg file. However, have observed the behavior that: Scheduler starts the execution…
I'm running the quickstart for KubernetesPodOperator secret using the link below : https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator Code used below : from airflow import models from airflow.contrib.kubernetes import secret from airflow.contrib.operators import kubernetes_pod_operator # A Secret is an object that contains a small amount…
I am trying to create a composer environment with the approach of infrastructure as code. For this, I need to be able to store and retrieve airflow variables programmatically, and keep them versioned somewhere. The following command lets me list…
I am running Airflow on Kubernetes from the stable helm chart. I'm running this in an AWS environment. This error exists with and without mounting any external volumes for log storage. I tried to set the configuration of the [logs]…
While doing follow up investigations on the stackoverflow question: Airflow DockerOperator cannot find some images but can find others. I tried loading a docker image using an airflow bash operator replicating exactly how I run it from my terminal and…
I like to re-run or run a DAG from composer, and below command is what i have used, but i got some exceptions like this kubeconfig entry generated for europe-west1-leo-stage-bi-db7ea92f-gke. Executing within the following Kubernetes cluster namespace: composer-1-7-7-airflow-1-10-1-db7ea92f command terminated…
I am trying to run Airflow Celery worker with redis backend but cant get the connection between the worker and redis to work. Here is how the configuration looks like: broker_url = redis://:<my-password>@localhost:6379/0 I am getting error: ValueError: invalid literal…