I have airflow dags running on Google Cloud Composer that trains machine learning models on some training data and stores the model with the best accuracy. I want to make a docker container/image that has the best model and deploy…
I'm running the quickstart for KubernetesPodOperator secret using the link below : https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator Code used below : from airflow import models from airflow.contrib.kubernetes import secret from airflow.contrib.operators import kubernetes_pod_operator # A Secret is an object that contains a small amount…
I am trying to create a composer environment with the approach of infrastructure as code. For this, I need to be able to store and retrieve airflow variables programmatically, and keep them versioned somewhere. The following command lets me list…
I like to re-run or run a DAG from composer, and below command is what i have used, but i got some exceptions like this kubeconfig entry generated for europe-west1-leo-stage-bi-db7ea92f-gke. Executing within the following Kubernetes cluster namespace: composer-1-7-7-airflow-1-10-1-db7ea92f command terminated…