I've been asked to learn Docker so that we can eventually setup our web scraping scripts in Apache Airflow Docker + Kubernetes clusters. If I am using these terms the wrong way, it's because I am a beginner at Docker.…
Basicly I'm trying to call a function from another files in airflow. I want to run a python function from a different files using airflow-docker I am using this image : https://airflow.apache.org/docs/apache-airflow/2.3.1/docker-compose.yaml to run an airflow Everything run smoothly when…
I have a running Apache AirFlow started with docker-compose. Also I have a docker-compose project which I'd like to manage with AirFlow. example_workers/docker-compose.yml: version: '3' services: my_app: build: context: ./my_app/ environment: TEST: 1 INDOCKER: 1 restart: "no" volumes: - ./my_app:/my_app…
I am testing some stuff where I have to init my Postgres DB DDL into airflow Postgres DB when I compose-up it should automatically init for one time as it will be cached afterward as airflow DB works usually. Thanks
I'm currently attempting to build a data pipeline using the puckel docker-airflow 1.10.9 image on windows. When the functions are called to the DAG from within the DAG file, it works well and I can access the output of the…
what is the best way to clean xcom table in airflow? thats running in docker with postgres db. i've try to delete some data with query (Delete from xcom) and try running this reference: https://cloud.google.com/composer/docs/cleanup-airflow-database but it doesn't work, xcom…
i want to extend the airflow on docker with providers hdfs: https://airflow.apache.org/docs/docker-stack/build.html#examples-of-image-extending the Dockerfile looks like: FROM apache/airflow:2.2.4 ARG DEV_APT_DEPS=" curl gnupg2 apt-transport-https apt-utils build-essential ca-certificates gnupg dirmngr freetds-bin freetds-dev gosu krb5-user ldap-utils libffi-dev libkrb5-dev libldap2-dev libpq-dev libsasl2-2 libsasl2-dev libsasl2-modules…
To preface I'm fairly new to Docker, Airflow & Stackoverflow. I've got an instance of Airflow running in Docker on an Ubuntu (20.04.3) VM. I'm trying to get Openpyxl installed on build in order to use it as the engine…
I have difficulties to understand how the paths in airflow work. I created this repository so that it is easy to see what I mean: https://github.com/remo2479/airflow_example/blob/master/dags/testdag.py I created this repository from scratch according to the manual on the airflow page.…
I'm working with Airflow DAG and Kubernetes for the first time. I have a Python script that connects to AWS S3 and reads some files. This works fine if I run it in a Docker container/image using bash. But when…