Amazon web services – How to create aws cluster with infrastructure as ec2 and asg using aws cli

ok after a lot of debugging and tutorials (none of which are complete) I've established a working model of steps.

Pre-requisite: Run and test local Dockerfile for single nginx hello world example

Folder directory

/deployScripts 
  ..scripts here...
/project_name/
  DockerFile etc...

Disclaimer This tutorial also is not complete because you are going to have to supply the variables as below.

Also you will need to configure your current aws user login to have the permission for the following commands. This won't be too hard to discover because every command you follow below will give an error message to say you need that permission. You can simply go to your configured login permission and add them one by one. Eventually your permission will look something like this (not complete).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:CreateRepository",
...ADD YOUR PERMISSION HERE...
            ],
            "Resource": "*"
        }
    ]
}

0. cd deployScripts

1. Create-Repository.sh

aws ecr create-repository --repository-name "${project_name}" > /dev/null

2. Push.sh

cd deployScripts
aws ecr get-login-password --region ${region}|docker login --username AWS --password-stdin ${imageFullTAG}
docker build -t ${project_name} ../${project_name}
docker tag ${project_name} ${imageFullTAG}
docker push ${imageFullTAG}

3. create-cluster.sh

aws ecs create-cluster --cluster-name $project_name

4. Create-security-groups.sh

#!/bin/sh

securityGroupID=$(aws ec2 create-security-group --group-name $securityGroupName --description "Security group for docker" --vpc-id $vpcID | jq -r '.GroupId')

# windows. Use alternative command for linux
myip=$(curl ifcfg.me)

# ssh from my ip
aws ec2 authorize-security-group-ingress 
    --group-id $securityGroupID 
    --protocol tcp 
    --port 22 
    --cidr $myip/32

## https from anywhere
aws ec2 authorize-security-group-ingress 
    --group-id $securityGroupID 
    --protocol tcp 
    --port 443 
    --cidr 0.0.0.0/0

## http from anywhere
aws ec2 authorize-security-group-ingress 
    --group-id $securityGroupID 
    --protocol tcp 
    --port 80 
    --cidr 0.0.0.0/0

5. Run EC2 instance and associate with cluster

user-data.template

#!/bin/bash
echo ECS_CLUSTER=$project_name >> /etc/ecs/ecs.config
echo ECS_AVAILABLE_LOGGING_DRIVERS='["json-file","awslogs"]' >> /etc/ecs/ecs.config
echo "ECS_CONTAINER_INSTANCE_TAGS={"ECS_CLUSTER": "$project_name"}" >> /etc/ecs/ecs.config

Run-instance-associate-with-cluster.sh

#!/bin/sh

amiID=$(aws --profile default ssm get-parameter --name ' /aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id' | jq -r '.Parameter.Value')

vpcID=$(aws ec2 describe-vpcs | jq -r '.Vpcs[].VpcId')
# subnets=$(aws --profile default ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-0eb5674ddf242dea3" | jq -r '[.Subnets[] | select(.Tags[]?.Value) .SubnetId] | join(" ")')

subnetID=$(aws --profile default ec2 describe-subnets --filters "Name=vpc-id,Values=${vpcID}" | jq -r '[.Subnets[] | select(.MapPublicIpOnLaunch==true) .SubnetId][0]')


# Create user-data.txt file
envsubst < ./user-data.template > ./user-data.txt

instanceID=$(aws ec2 run-instances 
  --profile default 
  --image-id $amiID 
  --count 1 
  --instance-type t2.micro 
  --iam-instance-profile Name=ecsInstanceRole 
  --key-name $keypair 
  --security-group-ids $securityGroupID 
  --subnet-id $subnetID 
  --user-data file://user-data.txt 
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=nginx-instance}]' 
  | jq -r '.Instances[] | .InstanceId')

echo "Created instace for cluster: instanceID=${instanceID}"

6. create-log-group.sh

aws logs create-log-group --log-group-name $project_name

7. Create Task definition

Create both roles which both have permission "AmazonEC2ContainerServiceforEC2Role":

• ecsTaskExecutionRole
• ecs-task-role

task-definition.template

{
  "family": "$project_name",
  "requiresCompatibilities": [
    "EC2"
  ],
  "executionRoleArn": "arn:aws:iam::$accountID:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::$accountID:role/ecs-task-role",
  "networkMode": "bridge",
  "containerDefinitions": [
    {
      "name": "$project_name",
      "image": "$imageFullTAG",
      "cpu": 128,
      "memoryReservation": 256,
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "$project_name",
          "awslogs-region": "$region",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "portMappings": [
        {
          "hostPort": 80,
          "containerPort": 80,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "healthCheck": {
        "command": [
          "CMD-SHELL",
          "curl -f http://localhost/ || exit 1"
        ],
        "interval": 10,
        "retries": 3,
        "startPeriod": 0,
        "timeout": 5
      }
    }
  ]
}

Create-task-definition.sh

#!/bin/sh
envsubst < ./task-definition.template > ./task-definition.json
export taskDefinitionArn=$(aws ecs register-task-definition --cli-input-json file://task-definition.json | jq -r '.taskDefinition.taskDefinitionArn')
echo "Created Task Definition: ${taskDefinitionArn}"

8. create-service.sh

#!/bin/sh
serviceArn=$(aws ecs create-service 
--cluster $project_name 
--service-name $project_name 
--task-definition $taskDefinitionArn 
--desired-count 1 
--launch-type EC2 | jq -r '.service .serviceArn')

echo "Created serviceArn=${serviceArn}"

Visit the Public IPv4 DNS of the in AWS > EC2 > 'nginx-instance'

done!

Conclusion. The original answer as not to use aws cli commands because it isn't typically done this way. Alternatives are to use the AWS console or terraform or potentially ecs-cli, but how do you really learn the low level details if you do that? A lot of the terminology and steps are hidden. My steps above maybe able to refine a little more but I had lots of hurdles to get that to work.

Notes: The main problems I had were following tutorials that weren't complete. For example including the image tag in the taskdefinition.json I added the shortcut name and that continously failed for me. I had to add the full image tag as it is pushed onto aws.