Question 19601 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

AWS S3: can I move object within a bucket by changing its key?

I have a javascript lambda function that is triggered by file upload, reads the content, validates, converts data records contained therein into an array and puts data into a DynamoDB table.
When everything is OK, the file (S3 object) is moved to another folder in the same bucket – the function copies the original object to another folder and deletes the original.
This action takes some time and generates "unnecessary" data transfer.
Is there a possibility to "modify" the object key, effectively moving it?

Tags:

2

Answers


  1. 0

    Amazon S3 does not have a ‘rename’ or ‘modify’ function. All objects are immutable and the Key cannot be changed — it is the unique identifier of the object.

    To ‘move’ an object, the correct process is to CopyObject() and then DeleteObject(). This creates a new object with the desired Key.

    Login or Signup to reply.
  2. 0

    Short Answer

    Unfortunately you won’t be able to get a performance boost, but you can abstract away the two operations using the S3 High-Level API

    Long Answer

    AWS S3 has two APIs.

    1. s3 which is a High-Level API
    2. s3api which is a Low-Level API

    When using the s3 mv api, you may issue a command (using CLI) to move or rename (if moved to the same destination, with a different name) a file. This API is constructed to mimic the feel and outcome of the Linux mv command.

    Syntax:

    aws s3 mv [S3ObjectSourcePath] [S3ObjectDestinationPath]
aws s3 mv [S3ObjectSourcePath] [LocalFilePath]
aws s3 mv [LocalFilePath] [S3ObjectSourcePath]

    This is a high level API, that basically does two operations:

    • s3:ObjectCreated:Copy
    • s3:ObjectRemoved:DeleteMarkerCreated, but abstracts it for you.

    This can be tested by creating an S3 Event Notification and running a Lambda function against those S3 Bucket events and setting up the event received to be logged in CloudWatch Logs. Then run the mv command and you’ll see the above two operations are going to be logged per mv command that is successfully executed.

    Examples:

    1. Moving a local file to S3

      aws s3 mv ~/hello.txt s3://my-bucket/

      Response:

      upload: ./hello.txt to s3://my-bucket/hello.txt

    2. Renaming a file on S3 (move to the same location)

      aws s3 mv s3://my-bucket/file.txt s3://my-bucket/file-moved.txt

      Response:

      move: s3://my-bucket/file.txt to s3://my-bucket/file-moved.txt

    3. Moving a file from S3 to S3

      aws s3 mv s3://my-bucket/file.txt s3://my-bucket/path/

      Response:

      move: s3://my-bucket/file.txt to s3://my-bucket/path/

    4. Moving an S3 object to a local file

      aws s3 mv s3://my-bucket/file.txt ~/file-moved-to-local.txt

      Response:

      download: s3://my-bucket/file.txt to ~/file-moved-to-local.txt

    5. Recursively moving S3 objects to a local directory

      # my-bucket contains two files: `file.txt` and `file2.txt`
aws s3 mv s3://my-bucket . --recursive

      Response:

      download: s3://mybucket/file.txt to file.txt
download: s3://mybucket/file2.txt to file2.txt

    6. Recursively moving local files to S3

      # `localFolder` is a local directory containing two files: `file.txt` and `file2.txt`
aws s3 mv ~/localFolder s3://my-bucket --recursive

      Response:

      upload: s3TestFolder/file.txt to s3://my-bucket/file.txt
upload: s3TestFolder/file2.txt to s3://my-bucket/file2.txt
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search