Question 19591 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

how to connect an s3 bucket w/ airflow

I have an airflow task where I try and load a file into an s3 bucket. I have airflow running on a Ec2 instance. Im running AF version 2.4.3 I have done

pip install 'apache-airflow[amazon]'

I start up my AF server, log in and go to the Admin section to add a connection. I open a new connection and I dont have an option for s3.

My only Amazon options are:
Amazon Elastic MapReduce
Amazon Redshift
Amazon Web services.

what else am I missing?

Tags:

3

Answers


  1. 0

    You need to define aws connection under "Amazon Web Services Connection"

    for more details see here

    Login or Signup to reply.
  2. 0

    You should define the connection within your DAG.

    You should also use a secure settings.ini file to save your secrets, and then call those variables from your DAG.

    See this answer for a complete guide: Airflow s3 connection using UI

    Login or Signup to reply.
  3. 0

    I am using Airflow version 2.4.3

    I also did:

    pip install 'apache-airflow[amazon]'

    and also only had the options of the following in the UI:

    • Amazon Elastic MapReduce
    • Amazon Redshift
    • Amazon Web services

    Other tutorials (maybe older versions?) show that there should be Amazon S3 as an option.

    However, if you select Amazon Web Services, then add you configuration in the "Extra" field:

    {"aws_access_key_id": "<insert here>", "aws_secret_access_key": "<insert here>"}

    Then it will work.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search