I have tried everything here, here, here, and virtually every SO article from a Google of the error.
I have a private ECR image that I am trying to pull with an ECS service in a public subnet.
Error when trying to create an ECS service:
Resourceinitializationerror: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "https://api.ecr.us-west-2.amazonaws.com/": dial tcp: lookup api.ecr.us-west-2.amazonaws.com: i/o timeout
Task definition:
{
"family": "chat-app-frontend",
"containerDefinitions": [
{
"name": "frontend",
"image": "576765093341.dkr.ecr.us-west-2.amazonaws.com/frontend:latest",
"cpu": 0,
"portMappings": [
{
"name": "frontend-80-tcp",
"containerPort": 80,
"hostPort": 80,
"protocol": "tcp",
"appProtocol": "http"
}
],
"essential": true,
"environment": [],
"mountPoints": [],
"volumesFrom": [],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-create-group": "true",
"awslogs-group": "/ecs/chat-app-frontend",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "ecs"
}
}
}
],
"taskRoleArn": "arn:aws:iam::576765093341:role/ecsTaskExecutionRole",
"executionRoleArn": "arn:aws:iam::576765093341:role/ecsTaskExecutionRole",
"networkMode": "awsvpc",
"requiresCompatibilities": [
"FARGATE"
],
"cpu": "1024",
"memory": "3072",
"runtimePlatform": {
"cpuArchitecture": "X86_64",
"operatingSystemFamily": "LINUX"
},
"tags": [
{
"key": "ecs:taskDefinition:createdFrom",
"value": "ecs-console-v2"
}
]
}
ECS task execution Role.
ecs-extra-services-access:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:CreateLogGroup"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"ssm:GetParameters",
"secretsmanager:GetSecretValue",
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds",
"secretsmanager:ListSecrets"
],
"Resource": [
"arn:aws:secretsmanager:us-west-2:576765093341:secret:prod/ecr-private-registry",
"arn:aws:kms:us-west-2:576765093341:key/807cbd08-a0ce-4948-b681-a49c7553003a"
]
}
]
}
VPC Endpoints
These are attached to the public subnets.
Endpoint Policy for com.amazonaws.us-west-2.secretsmanager
{
"Statement": [
{
"Sid": "AccessSpecificAccount",
"Principal": {
"AWS": "*"
},
"Action": "secretsmanager:*",
"Effect": "Allow",
"Resource": "*"
}
]
}
Security Group
Applies to ECS task and all endpoints
Other configs
- Task definition public IP create enabled
- Subnets public IP create enabled
- VPC DNS resolution and hostnames enabled
- Created repositories docker images with the flag
--endpoint-url https://api.ecr.us-west-2.amazonaws.com
- Tried using ECR private registry permissions but they are "Not allowed" with no specified reason.
I also want to note that there are no logs being generated which makes me suspicious the entire taskExecutionRole (my extra permissions policy) is not being applied somehow.
3
Answers
I launched the task in the default VPC with zero config (default security group) and it worked. Something wrong with my VPC config.
I want to add that I was able to create a custom VPC and get it working but it would always fail if I created the VPC with this module.
https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest
I had a similar issue and found out that ECS requires that your service has a public_ip assigned if it’s in a public subnet.
If you don’t want your service to have a public IP, you can create your service in a private subnet instead