Lambda S3 Sync Utility

Thanks to @John Rotenstein and his suggestion above I was able to figure this out. John, I wanted to post a complete answer here as I had to do a bit of cleanup from my initial approach as well. Still - your input was the primary missing piece, so thank you very much.

Setup a utility to copy any uploaded files from a source bucket in one organization to a destination bucket in another organization

Resources

Source S3 Bucket: arn:aws:s3:::s3-transfer-test-source-bucket
Source IAM role for lambda: arn:aws:iam::<source-acct-id>:role/s3-transfer-lambda-role
Source lambda Function: arn:aws:lambda:us-east-1:<source-acct-id>:function:s3-cross-account-sync

Destination S3 Bucket: arn:aws:s3:::s3-transfer-test-destination-bucket
Destination IAM role for source Lambda (assumable): arn:aws:iam::<source-acct-id>:role/s3-receiver-test-lambda-role
Destination IAM Policy: arn:aws:iam::<dest-acct-id>:policy/put-object-s3-cross-account-test-policy

Step 1: Create S3 Buckets

• Source S3 Bucket: arn:aws:s3:::s3-transfer-test-source-bucket
• Destination S3 Bucket: arn:aws:s3:::s3-transfer-test-destination-bucket

Step 3: Create IAM Role for Lambda (Source Account)

• IAM role for lambda: arn:aws:iam::<source-acct-id>:role/s3-transfer-lambda-role

Ensure the IAM role has the necessary permissions to read objects from the source S3 bucket and to log to CloudWatch Logs. The role should include two policies:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::s3-transfer-test-source-bucket",
                "arn:aws:s3:::s3-transfer-test-source-bucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]
}

Step 4: Create Lambda Function

• Create Source lambda: arn:aws:lambda:us-east-1:<source-acct-id>:function:s3-cross-account-sync
• Runtime: Python 3.8 (or your preferred Python runtime)
• Role: Select the IAM role created in step 3

Use the default code for now

import json

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Step 5: Configure Event Notifications on S3 Bucket (Source)

Name: S3TestPutObjectNotification
Event Types: s3:ObjectCreated:Put and s3:ObjectCreated:Post
Lambda Function: s3-cross-account-sync

Step 6: Test Current Config

Open CloudWatch
Select Live Tail
Filter by lambda function
Open S3 Bucket in a new tab
Upload file to S3
Watch CloudWatch for logs

If you see them, move on to next steps.

Step 7: Configure IAM Policy on Destination Account

Create Policy: arn:aws:iam::<dest-acct-id>:policy/put-object-s3-cross-account-test-policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::s3-transfer-test-destination-bucket",
                "arn:aws:s3:::s3-transfer-test-destination-bucket/*"
            ]
        }
    ]
}

Step 8: Configure IAM Role with Trusted Relationship on Destination Account

In IAM create a new role, and select "AWS Account as the trusted entity type. Select "Another AWS Account" then put the source organization ID into the box. Leave the two check boxes un-marked. Click Next. Click Next again. Enter a name and then click Create Role.

Role: arn:aws:iam::<dest-acct-id>:role/test-cross-org-transfer-role

Open the role, navigate to the Trust relationships, and edit the trusted entities with the correct Principle. (Conform to least-privilege security practices)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<source-acct-id>:role/s3-transfer-lambda-role"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

Step 9: Configure IAM Policy on Destination Account

Add inline policy to test-cross-org-transfer-role that allows the role to put objects into destination bucket, and get objects from source bucket.

name: test-cross-org-transfer-policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": [
                "arn:aws:s3:::s3-transfer-test-destination-bucket",
                "arn:aws:s3:::s3-transfer-test-destination-bucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::s3-transfer-test-source-bucket",
                "arn:aws:s3:::s3-transfer-test-source-bucket/*"
            ]
        }
    ]
}

Step 10: Update the IAM policy in the Source Account

arn:aws:iam::<source-acct-id>:role/s3-transfer-lambda-role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::s3-transfer-test-source-bucket",
                "arn:aws:s3:::s3-transfer-test-source-bucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<dest-acct-id>:role/test-cross-org-transfer-role"
        }
    ]
}

Step 11: Test Lambda with Assumed Role

Update the lambda code to the following

import boto3
import logging

# Set up logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    try:
        sts_client = boto3.client('sts')
        logger.info("Assuming destination role...")
        assumed_role = sts_client.assume_role(
            RoleArn="arn:aws:iam::<dest-acct-id>:role/test-cross-org-transfer-role",
            RoleSessionName="CrossAccountSession"
        )
        logger.info(f"Assumed Role ARN: {assumed_role['AssumedRoleUser']['Arn']}")
        logger.info("Role assumed successfully.")

        credentials = assumed_role['Credentials']
        s3 = boto3.client(
            's3',
            aws_access_key_id=credentials['AccessKeyId'],
            aws_secret_access_key=credentials['SecretAccessKey'],
            aws_session_token=credentials['SessionToken'],
        )

        destination_bucket = 's3-transfer-test-destination-bucket'

        # Attempt to directly upload a simple object
        test_key = 'test-object.txt'
        test_body = 'This is a test object.'
        logger.info(f"Directly uploading test object to {destination_bucket}...")
        s3.put_object(Bucket=destination_bucket, Key=test_key, Body=test_body)
        logger.info("Direct upload successful.")

    except Exception as e:
        logger.error(f"Error during direct upload operation: {e}")
        raise e

Test this Lambda function. It should output something like this...

Function Logs
START RequestId: 36ea5e82-330a-4e79-86f9-a755220dcb2d Version: $LATEST
[INFO] 2024-02-04T02:39:31.461Z 36ea5e82-330a-4e79-86f9-a755220dcb2d Assuming destination role...
[INFO] 2024-02-04T02:39:31.950Z 36ea5e82-330a-4e79-86f9-a755220dcb2d Assumed Role ARN: arn:aws:sts::<dest-acct-id>:assumed-role/test-cross-org-transfer-role/CrossAccountSession
[INFO] 2024-02-04T02:39:31.950Z 36ea5e82-330a-4e79-86f9-a755220dcb2d Role assumed successfully.
[INFO] 2024-02-04T02:39:32.861Z 36ea5e82-330a-4e79-86f9-a755220dcb2d Directly uploading test object to s3-transfer-test-destination-bucket...
[INFO] 2024-02-04T02:39:33.378Z 36ea5e82-330a-4e79-86f9-a755220dcb2d Direct upload successful.
END RequestId: 36ea5e82-330a-4e79-86f9-a755220dcb2d
REPORT RequestId: 36ea5e82-330a-4e79-86f9-a755220dcb2d Duration: 2076.14 ms Billed Duration: 2077 ms Memory Size: 128 MB Max Memory Used: 83 MB

You can verify by looking for a new .txt file in the destination bucket as well.

Step 12: Update Source Bucket Policy

We need to allow the assumed role to get objects from the source bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowGetObjectForAssumedRole",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<dest-acct-id>:role/test-cross-org-transfer-role"
            },
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::s3-transfer-test-source-bucket",
                "arn:aws:s3:::s3-transfer-test-source-bucket/*"
            ]
        }
    ]
}

Step 12: Update Lambda

This version performs the desired steps but still includes the logging for testing and debugging.

import boto3
import urllib.parse
import logging

# Initialize logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    sts_client = boto3.client('sts')
    destination_bucket = 's3-transfer-test-destination-bucket'
    role_to_assume_arn = 'arn:aws:iam::<dest-acct-id>:role/test-cross-org-transfer-role'

    try:
        for record in event['Records']:
            source_bucket = record['s3']['bucket']['name']
            key = urllib.parse.unquote_plus(record['s3']['object']['key'])
            logger.info(f"Detected new object: {key} in bucket {source_bucket}")

            # Try to get the object metadata from the source bucket with the original client
            s3_source = boto3.client('s3')
            try:
                response = s3_source.head_object(Bucket=source_bucket, Key=key)
                logger.info(f"Successfully retrieved metadata for {key} from {source_bucket} with source client")
            except Exception as e:
                logger.error(f"Error retrieving metadata for {key} from {source_bucket} with source client: {str(e)}")
                raise e

            logger.info("Assuming destination role...")
            assumed_role = sts_client.assume_role(RoleArn=role_to_assume_arn, RoleSessionName="S3CopySession")
            logger.info(f"Assumed role ARN: {assumed_role['AssumedRoleUser']['Arn']}")

            credentials = assumed_role['Credentials']
            s3_assumed = boto3.client('s3',
                                      aws_access_key_id=credentials['AccessKeyId'],
                                      aws_secret_access_key=credentials['SecretAccessKey'],
                                      aws_session_token=credentials['SessionToken'])

            # Try to get the object metadata from the source bucket with the assumed role client
            try:
                response_assumed = s3_assumed.head_object(Bucket=source_bucket, Key=key)
                logger.info(f"Successfully retrieved metadata for {key} from {source_bucket} with assumed role client")
            except Exception as e:
                logger.error(f"Error retrieving metadata for {key} from {source_bucket} with assumed role client: {str(e)}")
                raise e

            copy_source = {'Bucket': source_bucket, 'Key': key}
            logger.info(f"Copying {key} from {source_bucket} to {destination_bucket}")
            s3_assumed.copy_object(Bucket=destination_bucket, Key=key, CopySource=copy_source)
            logger.info(f"Successfully copied {key} to {destination_bucket}")

    except Exception as e:
        logger.error(f"Error during copy operation: {str(e)}")
        raise e

Step 13: Test S3 Upload

Drop a file in S3 source bucket, watch CloudWatch live tail - it should succeed. Check destination bucket for file to be sure!

Python Cleanup

For production, you'll want to streamline your code by removing excessive logging and checks that were primarily for debugging. Here's a cleaner version of the Lambda function that focuses on the essential operations, with minimal logging:

import boto3
import urllib.parse

def lambda_handler(event, context):
    # Configuration
    sts_client = boto3.client('sts')
    destination_bucket = 's3-transfer-test-destination-bucket'
    role_to_assume_arn = 'arn:aws:iam::<dest-acct-id>:role/test-cross-org-transfer-role'
    
    # Process each record from the event
    for record in event['Records']:
        source_bucket = record['s3']['bucket']['name']
        key = urllib.parse.unquote_plus(record['s3']['object']['key'])

        # Assume the destination role
        assumed_role = sts_client.assume_role(
            RoleArn=role_to_assume_arn,
            RoleSessionName="S3CopySession"
        )
        credentials = assumed_role['Credentials']

        # Create an S3 client with the assumed role's credentials
        s3_assumed = boto3.client(
            's3',
            aws_access_key_id=credentials['AccessKeyId'],
            aws_secret_access_key=credentials['SecretAccessKey'],
            aws_session_token=credentials['SessionToken']
        )

        # Copy object from source to destination bucket
        copy_source = {'Bucket': source_bucket, 'Key': key}
        s3_assumed.copy_object(
            Bucket=destination_bucket,
            Key=key,
            CopySource=copy_source
        )

Additional Recommendations for Production

1. Error Handling: While the streamlined code removes detailed logging and checks, ensure you have appropriate error handling in place. Consider using AWS Lambda Dead Letter Queues (DLQs) or SNS topics to capture and alert on failures.

2. Monitoring and Logging: Implement CloudWatch monitoring for the Lambda function to track invocations, errors, and performance metrics. Use CloudWatch Alarms to get notified of any operational issues.

3. Security: Regularly review the IAM roles and policies to ensure they follow the principle of least privilege. Rotate credentials if necessary and audit access regularly.

4. Performance Tuning: Based on the size and number of objects being transferred, you may need to adjust the Lambda function's memory and timeout settings for optimal performance.

5. Cost Management: Monitor the AWS bill for S3 and Lambda usage, especially if you're transferring large volumes of data. Consider implementing cost controls and alerts.

6. Testing: Before going live, thoroughly test the function in a staging environment to ensure it performs as expected under various conditions.

By following these guidelines and using the streamlined Lambda function, you should have a robust solution for copying objects between S3 buckets across accounts.