Question 107992 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – 502 Bad Gateway error on fastapi app hosted on EC2 instance + ELB

I have a FastAPI app that is hosted on EC2 instance with ELB for securing the endpoints using SSL.

The app is running using a docker-compose.yml file

version: '3.8'

services:

  fastapi:
    build: .
    ports:
      - 8000:8000
    command: uvicorn app.main:app --host 0.0.0.0 --reload
    volumes:
      - .:/kwept
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
      - redis

  worker:
    build: .
    command: celery worker --app=app.celery_worker.celery --loglevel=info --logfile=app/logs/celery.log
    volumes:
      - .:/kwept
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
      - fastapi
      - redis

  redis:
    image: redis:6-alpine
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data

volumes:
  redis_data:

Till Friday evening, the elb endpoint was working absolutely fine and I could use it. But since today morning, I have suddenly started getting a 502 Bad Gateway error. I had made no changes in the code or the settings on AWS.

The ELB listener settings on AWS:

enter image description here

The target group that is connected to the EC2 instance

enter image description here

When I log into the EC2 instance & check the logs of the docker container that is running the fastapi app, I see the following:

enter image description here

These logs show that the app is starting correctly

I have not configured any health checks specifically. I just have the default settings

enter image description here

Output of netstat -ntlp

enter image description here

I have the logs on the ELB:

http 2022-07-21T06:47:12.458060Z app/dianee-tools-elb/de7eb044e99165db 162.142.125.221:44698 172.31.31.173:443 -1 -1 -1 502 - 41 277 "GET http://18.197.14.70:80/ HTTP/1.1" "-" - - arn:aws:elasticloadbalancing:eu-central-1:xxxxxxxxxx:targetgroup/dianee-tools/da8a30452001c361 "Root=1-62d8f670-711975100c6d9d4038d73544" "-" "-" 0 2022-07-21T06:47:12.457000Z "forward" "-" "-" "172.31.31.173:443" "-" "-" "-"
http 2022-07-21T06:47:12.655734Z app/dianee-tools-elb/de7eb044e99165db 162.142.125.221:43836 172.31.31.173:443 -1 -1 -1 502 - 158 277 "GET http://18.197.14.70:80/ HTTP/1.1" "Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)" - - arn:aws:elasticloadbalancing:eu-central-1:xxxxxxxxxx:targetgroup/dianee-tools/da8a30452001c361 "Root=1-62d8f670-5ceb74c8530832f859038ef6" "-" "-" 0 2022-07-21T06:47:12.654000Z "forward" "-" "-" "172.31.31.173:443" "-" "-" "-"
http 2022-07-21T06:47:12.949509Z app/dianee-tools-elb/de7eb044e99165db 162.142.125.221:48556 - -1 -1 -1 400 - 0 272 "- http://dianee-tools-elb-yyyyyy.eu-central-1.elb.amazonaws.com:80- -" "-" - - - "-" "-" "-" - 2022-07-21T06:47:12.852000Z "-" "-" "-" "-" "-" "-" "-"
Tags:

3

Answers


  1. 0

    what is "502 Bad Gateway"?

    The HyperText Transfer Protocol (HTTP) 502 Bad Gateway server error response code indicates that the server, while acting as a gateway or proxy, received an invalid response from the upstream server.

    HTTP protocols

    • http – port number: 80

    • https – port number: 443

    From docker-compose.yml file you are exposing port "8000" which will not work.

    Possible solutions

    • using NGINX

    install the NGINX and add the server config

    server {
    listen 80;
    listen 443 ssl;
    # ssl on;
    # ssl_certificate /etc/nginx/ssl/server.crt;
    # ssl_certificate_key /etc/nginx/ssl/server.key;
    # server_name <DOMAIN/IP>;
    location / {
        proxy_pass http://127.0.0.1:8000;
    }
}

    • Changing the port to 80 or 443 in the docker-compose.yml file

    My suggestion is to use the nginx.

    Login or Signup to reply.
  2. 0

    I see you are using EC2 launch type. I’ll suggest ssh into the container and try curling the localhost on port 8080, it should return your application page. After that check the same on the instance as well since you have made the container mapping to port 8080. If this also works, try modifying the target group port to 8080 which is the port on which your application works. If the same setup is working on other resources, it could be you are using redirection. If this doesn’t help fetch the full logs using – https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-logs-collector.html

    If your application is working on port 8000. You need to modify the target group to perform health check there. Once the Target Group port will change to 8000 the health check should go through

    Login or Signup to reply.
  3. 0

    Make sure you’ve set Keep-Alive parameter of you webserver (in your case uvicorn) to something more than the default value of AWS ALB, which is 60s. Doing it this way you will make sure the service doesn’t close the HTTP Keep-Alive connection before the ALB.

    For uvicorn it will be: uvicorn app.main:app --host 0.0.0.0 --timeout-keep-alive=65

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search