Question 420168 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – AWS EC2 user data fails on boot but runs OK manually

On an AWS EC2 Linux 2023 instance, the user data fails with a timeout error on boot.

If I manually run the same AWS CLI command on the instance, it executes with no errors.

There are no networking or other environment changes between the boot fail and the manual success.

I tried:

  • adding a 5 and 60 second delay in the boot user data
  • different availability zone

Linux: Amazon Linux 2023 AMI 2023.6.20241212.0 x86_64 HVM kernel-6.1

AWS CLI: aws-cli/2.22.26 Python/3.12.6 Linux/6.1.119-129.201.amzn2023.x86_64 exe/x86_64.amzn.2023

The launch template is set to Metadata – V1 and V2 (token optional).

User data (updated to include the suggested ping test):

#!/bin/bash
echo "pulling instance ID from instance meta data"
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
echo ""
echo "instance ID: "$INSTANCE_ID
echo ""
# wait for network
echo "Wait for ping response from ec2.us-east-2.amazonaws.com"
echo ""
printf -v date '%(%Y-%m-%d %H:%M:%S)Tn' -1
echo $date
while ! ping -c 1 ec2.us-east-2.amazonaws.com &> /dev/null; do
    echo "Waiting for ping response from ec2.us-east-2.amazonaws.com..."
    printf -v date '%(%Y-%m-%d %H:%M:%S)Tn' -1
    echo $date
    sleep 1
done
echo ""
echo "associating elastic IP eipalloc-redacted to instance"
aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id=eipalloc-redacted --allow-reassociation
echo ""
echo "elastic IP address associated to instance"
echo ""
echo ""

Boot run fail log:

2024-12-29 18:12:13,616 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ec2.us-east-2.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.22.26 md/awscrt#0.23.4 ua/2.0 os/linux#6.1.119-129.201.amzn2023.x86_64 md/arch#x86_64 lang/python#3.12.6 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/distrib#amzn.2023 md/prompt#off md/command#ec2.associate-address', 'X-Amz-Date': b'20241229T181213Z', 'X-Amz-Security-Token': redacted, 'Authorization': b'AWS4-HMAC-SHA256 Credential=redacted/us-east-2/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=redacted', 'Content-Length': '137'}>
2024-12-29 18:12:13,623 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.22.26/dist/awscli/botocore/cacert.pem
2024-12-29 18:12:13,623 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): ec2.us-east-2.amazonaws.com:443
2024-12-29 18:13:15,154 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
  File "urllib3/connection.py", line 174, in _new_conn
  File "urllib3/util/connection.py", line 95, in create_connection
  File "urllib3/util/connection.py", line 85, in create_connection
TimeoutError: timed out

Manual test success log:

2024-12-29 20:43:11,351 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ec2.us-east-2.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.22.26 md/awscrt#0.23.4 ua/2.0 os/linux#6.1.119-129.201.amzn2023.x86_64 md/arch#x86_64 lang/python#3.12.6 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/distrib#amzn.2023 md/prompt#off md/command#ec2.associate-address', 'X-Amz-Date': b'20241229T204311Z', 'X-Amz-Security-Token': redacted', 'Authorization': redacted/us-east-2/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=redacted', 'Content-Length': '137'}>
2024-12-29 20:43:11,352 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.22.26/dist/awscli/botocore/cacert.pem
2024-12-29 20:43:11,352 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): ec2.us-east-2.amazonaws.com:443
2024-12-29 20:43:12,365 - MainThread - urllib3.connectionpool - DEBUG - https://ec2.us-east-2.amazonaws.com:443 "POST / HTTP/1.1" 200 278
2024-12-29 20:43:12,366 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'redacted', 'Cache-Control': 'no-cache, no-store', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Content-Type': 'text/xml;charset=UTF-8', 'Content-Length': '278', 'Date': 'Sun, 29 Dec 2024 20:43:11 GMT', 'Server': 'AmazonEC2'}
2024-12-29 20:43:12,366 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>n<AssociateAddressResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/"><requestId>redacted</requestId><return>true</return><associationId>eipassoc-redacted</associationId></AssociateAddressResponse>'
2024-12-29 20:43:12,366 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ec2.AssociateAddress: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fae76a03e90>>
2024-12-29 20:43:12,367 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-12-29 20:43:12,367 - MainThread - botocore.hooks - DEBUG - Event after-call.ec2.AssociateAddress: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at redacted>>
2024-12-29 20:43:12,367 - MainThread - awscli.formatter - DEBUG - RequestId: redacted
{
    "AssociationId": "eipassoc-redacted"
}

EC2 Instance IAM role policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeAddresses",
                "ec2:AllocateAddress",
                "ec2:DescribeInstances",
                "ec2:AssociateAddress",
                "ec2:DescribeAddressesAttribute",
                "ec2:DescribeAddressTransfers",
                "ec2:DisableAddressTransfer",
                "ec2:DisassociateAddress",
                "ec2:EnableAddressTransfer"
            ],
            "Resource": "*"
        }
    ]
}

The full cloud-init-output logs are too large to paste in as text.

boot fail cloud-init-output.log:

https://docs.google.com/document/d/1IucFCiUGAozDCbNfc4sdeL32vOn_gVUtH4xap34b_jQ/edit?usp=sharing

manual run success cloud-init-output.log:

https://docs.google.com/document/d/1xD0AQRDd7PtX1k5tXr-Peqej6Hkm2Vt5EqsMqI61Djk/edit?usp=sharing

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    By utilizing the ping test suggested by @jarmod and @Naresh, I was able to establish that the instance was unable to ping ec2.us-east-2.amazonaws.com.

    The problem was that the EC2 Launch Template was not set to auto-assign a public IP address.

    The 'auto-assign public IP' EC2 Launch Template option is located in: Network settings : Advanced network configuration : Auto-assign public IP

    To enable EC2 user data to associate an Elastic IP address, set the 'Auto-assign public IP' option to: Enable


  2. 0

    1)Make sure that the EC2 instance has an IAM role attached with the necessary permissions to execute the ec2:AssociateAddress API call.

    2)Also,include a test in the script to verify that the network is operational before attempting the command.
    Example bash script to include:

    while ! ping -c 1 ec2.us-east-2.amazonaws.com &> /dev/null; do
    echo "Waiting for network..."
    sleep 10
done
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search