I use MWAA to run my SQL jobs, which are executed on RDS MySQL Community. For example, I start with job 'A', and once it's completed, I proceed to run job 'B'. In job 'A', I use insert SQL to…
I have been struggling with an issue in my Apache Airflow DAG for some time now. I am trying to use the snowflake-connector-python module in my DAG, and it is included in my requirements.txt file. However, when I try to…
I'm running following script in MWAA and my local env. from airflow import DAG, XComArg from airflow.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperator from airflow.models.connection import Connection from airflow.operators.dummy_operator import DummyOperator from airflow.utils.task_group import TaskGroup # from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook from airflow.decorators import dag,…
I am trying to setup Airflow on Managed Apache Airflow. Everything seems to be working fine except for my AWS Redshift connection. I am using the Connections tab on UI and editing redshift_default with my values. It's working fine locally…
I am trying to ssh into my own AWS MWAA instance in order to install some system dependencies. I'm coming from GCP so this is a bit different for me. I can't find the exact EC2 it is hosted on.…
Is there a way to run a specific batch script on every worker node that MWAA spins up? Is there any feature in airflow that can do this?
we've set up AWS SecretsManager as a secrets backend to Airflow (AWS MWAA) as described in their documentation. Unfortunately, nowhere is explained where the secrets are to be found and how they are to be used then. When I supply…
I'm trying to query postgres from an MWAA instance of airflow. I'm not sure if there is a conflict due to airflow itself having a different version of postgres for its metadata or what, but I get this error when…
I have a MWAA environment and I have to create another one by Terraform. The environment creation is not an issue, but the 'metadata' of my old enviroment. I want to import all variables and connections, programatically, but I haven't…
I keep receiving an "Internal Server Error" redirect when I click the "Open Airflow UI" link in the Amazon MWAA console; it's just a blank white screen with "Internal Server Error" in big black text. I was being successfully redirected…