Question 18983 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

MWAA Airflow job getting SCRAM error when connecting to postgres – Amazon Web Sevices

I’m trying to query postgres from an MWAA instance of airflow. I’m not sure if there is a conflict due to airflow itself having a different version of postgres for its metadata or what, but I get this error when connecting to postgres:

  File "/usr/local/airflow/dags/transactions/transactions.py", line 62, in load_ss_exposures_to_s3
    ss_conn = psycopg2.connect(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: SCRAM authentication requires libpq version 10 or above

Locally I have psycopg2 version 2.9.5 and libpq version 140005. MWAA is using psycopg2 2.9.5 and libpq 90224. Is there a way for me to force MWAA to use another version? Maybe through airflow plugins? Airflow version is 2.4.3.

Tags:

2

Answers


  1. 0

    To update the libs version, in AWS MWAA, you can add a requirements file to S3, and configure MWAA to install it by selecting the file and updating the environment (doc).

    For your problem, you can try installing the same version you have in localhost, but if you are using graviton VM in MWAA (arm64), it can be another problem, like installing psycopg2-binary which is not compatible with arm64, and in this case you need to install psycopg2:

    pip install psycopg2
    Login or Signup to reply.
  2. 0

    In case anyone else encounters this issue when upgrading to MWWA Airflow 2.4.3, I managed to resolve this issue by adding psycopg2-binary to the requirements.txt file. I didn’t specify a version.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search