I have some EC2 instances in a private subnet (as part of a larger EKS cluster) in Account A. There is a containerized application running on these EC2 instances that reaches out to S3 for some data to process. The S3 bucket is in Account B. Currently the application just uses plain Access Key and Secret Key (since its a legacy app we don’t want to modify).
Currently I have a NAT gateway configured that allows the EC2 instances to access the internet. When the applications (within the Kubernetes pods) reaches out for files in S3, it goes through the NAT gateway, over the internet.
This incurs a significant cost for data transfer processed via the NAT gateway.
I’ve heard that VPC Gateway endpoints for S3 can help alleviate the problem by allowing access to go through the Amazon network, without using the internet. This can save money when it comes to data transfer costs. However, most blog posts seem to assume that the EC2 instances and the S3 buckets exist in the same account and region. In my case, they are in different AWS accounts.
I’ve also heard that VPC Interface Endpoints are an option. The documentation mentions they can be used across regions (but doesn’t mention across accounts). More importantly, there is a cost associated with the data transfer, unlike with VPC Gateway endpoints.
Is this the appropriate tool to use for this case or is there an alternate way that I can access an S3 bucket from a separate AWS account while avoiding too many data transfer costs?
My question is similar to this one, but that question focuses on public IP addresses, and in my case all my EC2 instances are in a private subnet.
2
Answers
An S3 vpc gateway endpoint is definitely what you want to be using. It doesn’t matter if the S3 bucket you want to access is in another account, it will work. The gateway endpoint gives connectivity to the S3 service, so it doesn’t matter where the bucket you want to access lives.
Do note that the gateway endpoint should be created in the same region as that where the S3 bucket lives.
I am assuming that you are wanting to reduce the "per GB data processed" costs of NAT Gateway. (Note, this is different to "Data Transfer", which is a term normally associated with sending traffic from AWS to the Internet).
Please note that there is no charge for sending traffic "between AWS Accounts". AWS Accounts are merely constructs for billing and security and do not impact architecture. Rather, charges arise from transferring data in/out of VPCs and between Regions.
In your scenario, resources in Amazon EKS in a VPC are communicating with S3 through NAT Gateway. Therefore, options are:
Option 1: Use a NAT Instance
A NAT Instance is just an Amazon EC2 instance that is acting as a NAT. It is charged purely per-hour. However, the size of the EC2 instance could restrict the network bandwidth available.
Option 2: Use a VPC Endpoint for S3
You can create a Gateway endpoint for Amazon S3 – Amazon Virtual Private Cloud, which creates a direct link between the VPC and the Amazon S3 service in the same region. This means requests to S3 do not go via the NAT Gateway.
The documentation says:
This would be the better option for the scenario you have described.