I’d like to perform bulk operations within S3 asynchronously. For some reason, I could not find a way of doing that in the new SDK2 client built by AWS. I currently use the S3AsyncClient‘s putObject
method for uploading files, but I’d like to do that in batches for performance improvements. The same applies to downloading files. I’ve read about multipart uploads but I did not understand if that’s what I’m looking for, since my files are relatively small and I don’t need a multipart upload. I How can I perform a multi-file upload?
Question posted in Amazon Web Sevices
The official Amazon Web Services documentation can be found here.
The official Amazon Web Services documentation can be found here.
2
Answers
I am not aware of a multi-file upload mode in Amazon S3.
Another way is to zip the files and upload one big file, where you will benefit from multipart uploads. Then you let a Lambda function, triggered by the upload of a zip in your bucket, unzip automatically the files for you.
Further description of alternatives and a code example for the zip & unzip strategy can be found here.
You could use
S3TransferManager
which is the implementation ofTransferManager
library adjusted for AWS Java SDK v2.S3TransferManager
provides bulk uploads by using multiple threads.Keep in mind that in case of S3 you are paying for each
PUT
request. Each file upload will constitute at least one PUT request. Even if you are using something likeuploadDirectory
from theS3TransferManager
, you will still pay for after each file.