I am trying to use class EcsCreateClusterOperator in MWAA but I get the following error:
Broken DAG: [/usr/local/airflow/dags/etl_basic_ecs/etl_ecs_basic_dag.py] Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/airflow/dags/etl_basic_ecs/etl_ecs_basic_dag.py", line 5, in <module>
from airflow.providers.amazon.aws.operators.ecs import (
ImportError: cannot import name 'EcsCreateClusterOperator' from 'airflow.providers.amazon.aws.operators.ecs' (/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/ecs.py)
This library works perfectly in my local environment installed with:
pip install apache-airflow-providers-amazon==5.0.0
I also added this library (version 5.0.0) to my requirements.txt file before deploying to MWAA, in the UI seems like the provider is installed:
Screenshot of installed Amazon Provider 5.0.0
The log of the update shows that the library is downloaded:
Here is the documentation of the class: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/operators/ecs/index.html this class was introduced recently (version 5.0.0).
By default MWAA (Airflow 2.2.2) has the Amazon Provider with an older version (2.4), seems like Airflow is really loading the older version of the library.
Any help would be appreciated, thank you.
2
Answers
MWAA with Airflow version 2.2.2 is constrained to
apache-airflow-providers-amazon==2.4.0
. See the constraints file for more details.References:
I had the same issue with
EksCreateClusterOperator
:Looks like is case sensitive. So instead of using
EcsCreateClusterOperator
change it toECSCreateClusterOperator
(capital ECS) how it is showing on airflow github. That fixed my issue.