AWS Load Balancer with ECS. How do I troubleshoot health checks failing? – Nginx

I am deploying a multi container Flask python app (with gunicorn) to ECS with Docker to my ECS cluster that uses a single t2.small EC2 instance. My app runs on port 8000 and runs fine, I can use my app perfectly when using my EC2 DNS: http://ec2-xx-xxx-xxx-xx.us-east-2.compute.amazonaws.com:8000

I now want to use my own custom domain instead from GoDaddy. I’m using Route 53 for the nameserver registration, and plan to use an alias that points to my instance via a load balancer (Application Load Balancer).

Before setting up the alias, I want to first check my ALB is is successfully allowing me to access my app on port 8000 via HTTP (port 80) using the target group. My ECS service creates fine and I can see my web app running in the logs, but when I put the ALB DNS into my browser I get: 502 Bad Gateway.

I’ve checked my Target Group and it seems that the registration of my EC2 instance is failing on port 8000 due to "Health checks fail". I can’t find any further details on the cause of failure, ‘Health status details’ just says ‘Health checks failed’ and describe-target-health returns Target.FailedHealthChecks.

I’ve tried to troubleshoot myself following these steps: https://www.youtube.com/watch?v=cmRZleI18Yg

When I SSH into my EC2 instance on cmd and run telnet 80, I get a ‘Connection refused’ error rather than HTTP 200 response. When I try the same using my load balancer DNS, It connects successfully on PORT 80. My current thinking is that for some reason, my EC2 instance is not listening on port 80. I have no idea why and have tried the following already:

Ensured correct set-up of security groups and NACL

Yes, EC2 security group set-up to allow all traffic on port 80 and have added rule for my ALB security group on all ports. NACL accepting all inbound and outbound traffic.

**Ensure no firewall on EC2 blocking HTTP **

Have run sudo service iptables status and got the following:

Redirecting to /bin/systemctl status iptables.service
● iptables.service - IPv4 firewall with iptables
   Loaded: loaded (/usr/lib/systemd/system/iptables.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

which seems to suggest no firewalls in place?

Confirm web server is actually running on EC2 instance

I’m very open to suggestions here, but I assume it’s running from a) being able to see logs in ECS and b) I could use the app successfully when using http://ec2-xx-xxx-xxx-xx.us-east-2.compute.amazonaws.com:8000. Is there anything else I can do to validate this?

Logs from ECS:

Dockerfile

FROM python:3.7.5-slim-buster

RUN apt-get update && apt-get install -qq -y 
  build-essential libpq-dev --no-install-recommends

RUN apt-get install libcurl4-openssl-dev libssl-dev -y

ENV INSTALL_PATH /canopact
RUN mkdir -p $INSTALL_PATH

WORKDIR $INSTALL_PATH

COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

COPY . .
RUN pip install --editable .

CMD gunicorn -c "python:config.gunicorn" "canopact.app:create_app()"

Gunicorn.py

# -*- coding: utf-8 -*-

bind = '0.0.0.0:8000'
keepalive = 120
accesslog = '-'
access_log_format = '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s" in %(D)sµs'

I’m still using the flask config SERVER_NAME as: "ec2-xx-xxx-xxx-xx.us-east-2.compute.amazonaws.com:8000", but I have tried changing it to "0.0.0.0:8000" and getting the same result of healthchecks failing

Questions

I’m really really unsure on what else I can do to troubleshoot this. Am I correct to assume that my healthchecks are failing because of the Connection refused when trying to connect to EC2 instance on PORT 80, or should I be investigating something else? Feels like every stackoverflow post suggests trying the above 3 steps to troubleshoot Connection refused, so I’m not sure what else to try.

I’ve seen some posts suggesting use of NGINX, but I thought ALB could be used independently on NGINX?