Question 19829 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

automatically move object from one s3 bucket to another

I want to automatically move objects from first s3 bucket to second bucket. As and when a file is created or uploaded to first bucket, that should be moved across to the second bucket. There shouldn’t be any copy of the file on the source bucket after the transfer.

I have seen examples of aws s3 sync but that leaves a copy on the source bucket and it’s not automated.

aws mv command from cli will move the files across but how to automate the process. Creating a lambda notification and send the files to second bucket could solve but I am looking for a more automated simpler solution. Not sure if there is anything we could do with SQS? Is there anything we can set on the source bucket which would automatically send the object to the second? Appreciate any ideas

Tags:

2

Answers


  1. 0

    There is no "move" command in Amazon S3. Instead, it involves CopyObject() and DeleteObject(). Even the AWS CLI aws mv command does a Copy and Delete.

    The easiest way to achieve your object is:

    • Configure an Event on the source bucket to trigger an AWS Lambda function
    • Code the Lambda function to copy the object to the target bucket, then delete the source object

    Here’s some sample code:

    import boto3
import urllib

TARGET_BUCKET = 'my-target-bucket'

def lambda_handler(event, context):

    # Get incoming bucket and key
    source_bucket = event['Records'][0]['s3']['bucket']['name']
    source_key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])

    # Copy object to different bucket
    s3_resource = boto3.resource('s3')
    copy_source = {
        'Bucket': source_bucket,
        'Key': source_key
    }
    s3_resource.Bucket(TARGET_BUCKET).Object(source_key).copy(copy_source)

    # Delete the source object
    s3_resource.Bucket(TARGET_BUCKET).Object(source_key).delete()

    It will copy the object to the same path in the destination bucket and then delete the source object.

    The Lambda function should be assigned an IAM Role that has sufficient permission to GetObject from the source bucket and CopyObject + PutObject in the destination bucket.

    Login or Signup to reply.
  2. 0

    A followup question over this. How about an approach of setting up a S3 Replication to the destination S3 bucket. The file landing on the destination bucket can trigger a Lambda to cleanup the source bucket.
    Bottomline question: Is S3 replication more efficient compared to a S3 copy()?

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search