I am developing and testing with airflow inside a docker container. The container is created using the image ghcr.io/apache/airflow/main/ci/python3.9 from apache/airflow. Inside the container, airflow and python are installed: $ which airflow /usr/local/bin/airflow $ which python /usr/local/bin/python I tried to…
I have created an AWS MWAA environment and correctly pointed it to an S3 bucket as well as an "dags" folder inside the S3 bucket. I have created a simple dag file (python) and added it to my /dags/ directory…
I have Clickhouse-server and Airflow all are Docker containers in WSL. They're both running and i even managed to connect to Clickhouse somehow, but when trying to query i get the following traceback in Airflow logs: [2024-06-04, 12:17:49 UTC] {logging_mixin.py:188}…
I have already set up the following infrastructure: host (main): airflow (with all the services) slave server airflow (redis, postgres, worker) Cannot see any tutorial or couldn't find any clue about how to do it, but i don't have any…
I am creating a apache airflow app which requires the use of python packages pyportfolioopt or cvxpy. In the dockerfile i am currently using apache/airflow:2.8.4 as the base image. When i run docker-compose up --build using the files below, i…
Context I'm trying to setup an airflow/dbt docker infrastructure with Snowflake as a database, AWS S3 as a file management system and MongoDB as a data source. Issue I installed apache-airflow-providers-snowflake, I can find airflow.providers.snowflake.transfers.copy_into_snowflake but can't find airflow.providers.snowflake.transfers.s3_to_snowflake !…
I have a task running on AirFlow that has two steps: fetches data from MSSql server as a dataframe; stores it in a PostGres database; I'm using the MsSqlHook and PostgresHook to manage the connections. The fetching is going ok,…
I have issue with connecting to airflow database I can do it locally with such code import pandas as pd from sqlalchemy import create_engine import os df = pd.read_csv('wynik_zgloszenia.csv', sep = '#') engine = create_engine(f'postgresql+psycopg2://postgres:mysecretpassword@localhost:8001/postgres',client_encoding='utf8') df.to_sql('permissions', engine, index=False, if_exists='replace') I…
Hi I am completely new to docker and run into the dreaded Docker usr/bin/env: ‘bashr’: No such file or directory issue. I understand there are number of answers posted here but none of them were able to help me as…
I'm trying to build a minimal datapipeline using docker, postgres, and airflow. My docker-compose.yaml file can be found here and is exteneded from airflow's documentation here. I've extended it to include a seperate postgres database where I will write data,…