Question 71664 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Copy files from s3 bucket to azure blob storage in airflow – Amazon web services

My task is to Copy files from s3 bucket to azure blob storage in airflow.
I found using azcopy i can move my data from s3 bucket to azure blob storage but the problem is i have to create dag for this which should trigger daily on specific time.

i have write and read permission for particular container of client machine (azure blob storage).

client has provided few details of blob storage

AccountName : "xxx"
Container Name : "xxx"
Blob SAS URL : "xxx"
Blob SAS Token : "xxx"
Storage account key : "xxx"

Can someone please help me to create airflow dag which can copy or move data from s3 bucket to azure blob storage

Tags:

2

Answers


  1. 0

    You can use rclone. You must set schedule time slices.
    (Windows:Task scheduler Linux:Cron)

    Login or Signup to reply.
  2. 0

    You can use Skyplane to copy data across clouds. To transfer from S3 to Azure blob storage you can call the command:

    skyplane cp -r s3://aws-bucket-name/ az://azure-bucket-name/
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search