skip to Main Content

I have tried everything here, here, here, and virtually every SO article from a Google of the error.

I have a private ECR image that I am trying to pull with an ECS service in a public subnet.

Error when trying to create an ECS service:

Resourceinitializationerror: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "https://api.ecr.us-west-2.amazonaws.com/": dial tcp: lookup api.ecr.us-west-2.amazonaws.com: i/o timeout

Task definition:

{
    "family": "chat-app-frontend",
    "containerDefinitions": [
        {
            "name": "frontend",
            "image": "576765093341.dkr.ecr.us-west-2.amazonaws.com/frontend:latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "frontend-80-tcp",
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [],
            "mountPoints": [],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/chat-app-frontend",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "ecs"
                }
            }
        }
    ],
    "taskRoleArn": "arn:aws:iam::576765093341:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::576765093341:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "1024",
    "memory": "3072",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    },
    "tags": [
        {
            "key": "ecs:taskDefinition:createdFrom",
            "value": "ecs-console-v2"
        }
    ]
}

ECS task execution Role.

enter image description here

ecs-extra-services-access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:CreateLogGroup"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "ssm:GetParameters",
                "secretsmanager:GetSecretValue",
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds",
                "secretsmanager:ListSecrets"
            ],
            "Resource": [
                "arn:aws:secretsmanager:us-west-2:576765093341:secret:prod/ecr-private-registry",
                "arn:aws:kms:us-west-2:576765093341:key/807cbd08-a0ce-4948-b681-a49c7553003a"
            ]
        }
    ]
}

VPC Endpoints

These are attached to the public subnets.

enter image description here
enter image description here

Endpoint Policy for com.amazonaws.us-west-2.secretsmanager

{
    "Statement": [
        {
            "Sid": "AccessSpecificAccount",
            "Principal": {
                "AWS": "*"
            },
            "Action": "secretsmanager:*",
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Security Group

Applies to ECS task and all endpoints
enter image description here

Other configs

  • Task definition public IP create enabled
  • Subnets public IP create enabled
  • VPC DNS resolution and hostnames enabled
  • Created repositories docker images with the flag --endpoint-url https://api.ecr.us-west-2.amazonaws.com
  • Tried using ECR private registry permissions but they are "Not allowed" with no specified reason.

I also want to note that there are no logs being generated which makes me suspicious the entire taskExecutionRole (my extra permissions policy) is not being applied somehow.

3

Answers


  1. Chosen as BEST ANSWER

    I launched the task in the default VPC with zero config (default security group) and it worked. Something wrong with my VPC config.


  2. I want to add that I was able to create a custom VPC and get it working but it would always fail if I created the VPC with this module.

    https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest

    Login or Signup to reply.
  3. I had a similar issue and found out that ECS requires that your service has a public_ip assigned if it’s in a public subnet.

    If you don’t want your service to have a public IP, you can create your service in a private subnet instead

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search