This is related to, but different from How to use multiple AWS accounts from the command line?
I would like to sync one AWS bucket to another across two accounts. I have credentials for both accounts set up on my local machine, and I can, for example, do
$ aws s3 sync s3://my-first-bucket/ ./
# or
$ aws s3 sync ./ s3://my-other-bucket --profile other_account
But I’d like to be able to transfer from the default account directly to the bucket on the other account without needing to download and then re-upload. In other words, something like
aws s3 sync --profile default s3://my-first-bucket/ --profile other_account s3://my-other-bucket
Can this be accomplished?
2
Answers
AWS has a prescriptive guidance on how to do exactly that.
In a nutshell, doing what you want the way you want is not the way things are done in S3.
Basically, your model of doing things would involve two different S3 agents running with two sets of credentials. One has the permissions to read source data, another one has the permissions to write destination data, and they engage in a copying session, sending and receiving data and coordinating their progress.
It would not be technically impossible to program S3 like that, because that what you would be essentially doing if you just piped two aws commands into one another on your local machine. But it’s not been done this way.
Insted, copy and sync are uniform commands running with the a single set of credentials on the target account. These credentials should have the permissions to both read the source data and write the target data. These permissions are set on the bucket level and/or the object level.
If you want the S3 copies to run completely within S3, you’ll need to run the command on the target account and grant it permissions on the bucket belonging to the source account.
If you don’t want to deal with cross-accounts permissions, you’ll have to use some kind of an intermediate solution which would read the data from the source and write it to the target using two sets of credentials. There are lots of third-party solutions doing that, such as rclone.
To use a single set of credentials, you could add a Bucket Policy to one of the buckets, which would permit credentials from a different AWS Account to access the bucket.
Option 1: Push
PutObjectfor the credentials from the ‘source’ accountACL=bucket-owner-full-controlwhen performing the copy, or that the destination bucket has ACLs disabled. This ensures that the AWS Account owning the destination bucket has control of the objectsOption 2: Pull
GetObjectfor the credentials from the ‘destination’ account