Here’s my setup:
- using AWS Organizations to segregate different application stacks; each Organizational Unit has separate AWS accounts to separate Dev and Prod.
- one "Shared services" AWS account with a CodeArtifact repository containing private Python packages, including common packages for use by multiple OUs.
This works: I’ve been able to set up cross-org access to CodeArtifact and am able to pull down a package from CodeArtifact in Account Shared Services
to say an EC2 instance in Account A
.
I am setting up an AWS Glue job in Account A
, and I want it to use some packages from my CodeArtifact repo.
This works: If I run the Glue job outside my VPC, no problem, I’m able to access CodeArtifact.
This doesn’t work: However, I want to run Glue within my VPC in Account A
(to access other resources running in the VPC). It looks like when I do that, I lose the ability to connect to CodeArtifact.
This seems to be because when Glue runs within a VPC, AWS automatically creates an elastic network interface without any public IPs (described in more detail here). AWS suggests using a NAT gateway to get around this.
Now, if my CodeArtifact repo were also within Account A
(where my Glue job is), I think I could create a VPC endpoint for it that I could use to connect from Glue from within the VPC. Is there some way to do something similar given its in a different account? Everything I’m talking about here is all within AWS, I don’t want to create a NAT gateway just to connect to an AWS service from another AWS service.
2
Answers
My question stemmed from an unclear understanding of the CodeArtifact service and VPC endpoints. Specifically: when a VPC endpoint is connected to an AWS service, it is simply a pathway for resources in a private subnet in the VPC to connect to the service without having access to the Internet. The service itself is global and is not account-specific.
In other words, to solve for the scenario described in my question I was able to simply create a CodeArtifact VPC endpoint (I only needed the
repository
endpoint, not theapi
one) withinAccount A
. This gave the Glue job running within theAccount A
VPC access to the global CodeArtifact service, and so long as I had the necessary permissions set for cross-account access on the CodeArtifact domain owned by theAccount Shared Services
account, I was able to download packages for use in my Glue job.Many AWS Services struggle with cross-account things. Drop AWS Support a ticket and they will tell you the details, it’s the only guaranteed way of getting a correct answer on the topic as they often don’t publish these kinds of nuances in any of the publicly available documentation.