Question 313277 in Docker Question posted in
The official documentation can be found here.

AKS CrashLoopBackOff for an image working without problem in docker host: logs: error: container containerd://x is not valid for pod flask-app

I have a Flask app that should run on Apache2 server. I dockerized the app and it is working as expected on docker host in my test VM.

Once deployed on AKS the POD is stuck in state CrashLoopBackOff.
I am using a gitlab agent to deploy into AKS with gitlab-ci, the image is stored in gitlab private registry and the image was successifully pulled into AKS.

Here are the files/logs of my apps/AKS pod

Dockerfile

FROM debian:bullseye-slim


RUN apt-get update && apt-get install -y apache2 
    libapache2-mod-wsgi-py3 
    python3 
    python3-pip 
    && apt-get clean 
    && apt-get autoremove 
    && python3 -m pip install flask 
    && rm -rf /var/lib/apt/lists/*

COPY ./requirements.txt /var/www/webApp/requirements.txt
RUN pip install -r /var/www/webApp/requirements.txt

COPY ./webApp.conf /etc/apache2/sites-available/webApp.conf
COPY ./ /var/www/webApp/

RUN  a2dissite 000-default.conf
RUN  a2ensite webApp.conf


RUN ln -sf /proc/self/fd/1 /var/log/apache2/access.log && 
    ln -sf /proc/self/fd/1 /var/log/apache2/error.log

EXPOSE 8080

WORKDIR /var/www/webApp

CMD /usr/sbin/apache2ctl -D FOREGROUND

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: flask-app
  name: flask-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask-app
  template:
    metadata:
      labels:
        app: flask-app
    spec:
      containers:
      - image: registry.gitlab.com/sast6353642/sast-01-pull/flask-appache:0a654662 
        name: flask-appache
        ports:
        - containerPort: 8080
        #securityContext:
            #privileged: false
        imagePullPolicy: Always
        #resources:
            #limits:
                #memory: 100Mi
                #cpu: 300m
      imagePullSecrets:
      - name: gitlab
      restartPolicy: Always

kubectl get pods

NAME                         READY   STATUS             RESTARTS        AGE
flask-app-6d74447889-bghtx   0/1     CrashLoopBackOff   6 (2m27s ago)   8m18s

kubectl describe pods flask-app-6d74447889-bghtx

Name:             flask-app-6d74447889-bghtx
Namespace:        default
Priority:         0
Service Account:  default
Node:             aks-agentpool-16891536-vmss000000/10.224.0.5
Start Time:       Tue, 02 Jan 2024 20:20:49 +0000
Labels:           app=flask-app
                  pod-template-hash=6d74447889
Annotations:      cni.projectcalico.org/containerID: 966c590b4ce639fa829feba8e597d63395af53ea231ab005f8d547e1721f8233
                  cni.projectcalico.org/podIP: 10.244.1.17/32
                  cni.projectcalico.org/podIPs: 10.244.1.17/32
Status:           Running
IP:               10.244.1.17
IPs:
  IP:           10.244.1.17
Controlled By:  ReplicaSet/flask-app-6d74447889
Containers:
  flask-appache:
    Container ID:   containerd://5cdc5e1294267fce47bda95cb81bd09654a78fcf21dfe4bbaece786d195ea778
    Image:          registry.gitlab.com/sast6353642/sast-01-pull/flask-appache:0a654662
    Image ID:       registry.gitlab.com/sast6353642/sast-01-pull/flask-appache@sha256:e798e4036dc13d72ef36c866d678fe295e3eac4b0058cd076330654bc56af317
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 02 Jan 2024 20:22:20 +0000
      Finished:     Tue, 02 Jan 2024 20:22:20 +0000
    Ready:          False
    Restart Count:  4
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sj7s8 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-sj7s8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  2m57s                  default-scheduler  Successfully assigned default/flask-app-6d74447889-bghtx to aks-agentpool-16891536-vmss000000
  Normal   Pulled     2m57s                  kubelet            Successfully pulled image "registry.gitlab.com/sast6353642/sast-01-pull/flask-appache:0a654662" in 864.830288ms (864.838488ms including waiting)
  Normal   Pulled     2m55s                  kubelet            Successfully pulled image "registry.gitlab.com/sast6353642/sast-01-pull/flask-appache:0a654662" in 827.635558ms (827.908837ms including waiting)
  Normal   Pulled     2m38s                  kubelet            Successfully pulled image "registry.gitlab.com/sast6353642/sast-01-pull/flask-appache:0a654662" in 875.237209ms (875.244689ms including waiting)
  Normal   Created    2m10s (x4 over 2m57s)  kubelet            Created container flask-appache
  Normal   Started    2m10s (x4 over 2m56s)  kubelet            Started container flask-appache
  Normal   Pulled     2m10s                  kubelet            Successfully pulled image "registry.gitlab.com/sast6353642/sast-01-pull/flask-appache:0a654662" in 827.338313ms (827.349353ms including waiting)
  Warning  BackOff    100s (x7 over 2m55s)   kubelet            Back-off restarting failed container flask-appache in pod flask-app-6d74447889-bghtx_default(1177ffe3-61ff-418f-a37d-3b6d857c603b)
  Normal   Pulling    88s (x5 over 2m57s)    kubelet            Pulling image "registry.gitlab.com/sast6353642/sast-01-pull/flask-appache:0a654662"
  Normal   Pulled     87s                    kubelet            Successfully pulled image "registry.gitlab.com/sast6353642/sast-01-pull/flask-appache:0a654662" in 862.977469ms (862.990149ms including waiting)

kubectl logs flask-app-6d74447889-bghtx -c containerd://5cdc5e1294267fce47bda95cb81bd09654a78fcf21dfe4bbaece786d195ea778

error: container containerd://c4c557b4e6f92ac803c7e9c42624e59e39941e3d731015e12c4bea193dd5db3b is not valid for pod flask-app-6d74447889-bghtx

Logs in diagnostics settings in Azure

exec /usr/local/bin/python: exec format error

**app.py **

from website import create_app


app = create_app()

if __name__ == "__main__":
    app.run(debug=True)

Can anyone help with that please ?

I tried making changes into the Dockerfile (changing the port exposed from 80 to 8080) but nothing happened. I searched from the log container not valid for pod and did not find anything related to that.

EDIT

My .gitlab-ci.yml

stages:
- build
- deploy

variables:
  DOCKER_SOURCE_IMAGE: flask-appache:$CI_COMMIT_SHORT_SHA
  DOCKER_TARGET_IMAGE: $CI_REGISTRY_TAG/$DOCKER_SOURCE_IMAGE
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2 
  CI_REGISTRY_TAG: registry.gitlab.com/sast6353642/sast-01-pull 
  CI_REGISTRY: registry.gitlab.com

pull-push:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  before_script:  
    - docker login ${CI_REGISTRY} -u ${USERNAME_REGISTRY} -p ${TOKEN_REGISTRY}    

  script: 
    - docker build . -t ${DOCKER_SOURCE_IMAGE}
    - docker tag ${DOCKER_SOURCE_IMAGE} ${DOCKER_TARGET_IMAGE}
  
    - docker push ${DOCKER_TARGET_IMAGE}
    - docker logout ${CI_REGISTRY}   

deploy:
  stage: deploy
  image: 
    name: bitnami/kubectl:latest
    entrypoint: ['']
  script:
    - kubectl config use-context k8s-flask/k8s-connection:k8s-agent-connection  
    - kubectl get pods
    - kubectl apply -f .
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I resolved the issue. The cpu architecture of node pools (ubuntu) in AKS is ARM, I had to specify FROM --platform=linux/arm in my Dockerfile so the base image would be based on ARM instead on AMD64 so it can work in an ARM based host.

    Exemple: FROM --platform=linux/arm python:3.10-alpine

    If the image is built in an Amd64 based host with the --platform=linux/arm prefix for base image in the Dockerfile this will not work => Qemu has to be installed so it can provide other system architecture platforms.

    Before installing Qemu

    debian@debian11:~/simple-flask$ docker buildx ls 

NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT                              PLATFORMS default * docker   default default         running v0.11.7-0.20230525183624-798ad6b0ce9f linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

    Installing Qemu on Debian/Ubuntu

    sudo apt-get install -y qemu qemu-user-static

    You can see ARM now in platforms

    debian@debian11:~/simple-flask$ docker buildx ls


NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT                              PLATFORMS
default * docker
  default default         running v0.11.7-0.20230525183624-798ad6b0ce9f linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

    If you build the image using CI/CD as gitlab-ci you just need to specify the --platform=linux/arm in you Dockerfile Before the image name.

    Hope this helps !


  2. 0

    In order to run your dockized flask application on AKS, you can refer to below example.

    Here I will be using a simple flask app that says This is a Hello World application using the below code. You can replace it with your own.

    from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    return "<p>This is a Hello World application</p>"

if __name__ == "__main__":
    app.run(host='0.0.0.0', debug=True)

    Then containerize the app using docker. In this case, Flask application only requires the flask framework to run. Therefore, in the working directory, I will create a new file named ‘Dockerfile’ and use the following code:

    FROM python:3.10-alpine
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
EXPOSE 5000
CMD ["python", "/app/app.py"]

    and then build it using docker build -t bravinwasike/flask-kubernetes .

    Output:
    enter image description here

    After creating the Docker image, ensure you push the Docker image into your Azure container repository using the following commands:

    az login
az acr login --name <youracrname>
docker tag <the_name_you_want_to_show_in_ACR> <youracrname>.azurecr.io/<the_image_name_which_you_built>

docker push <youracrname>.azurecr.io/<the_name_you_want_to_show_in_ACR>

    enter image description here

    next we will deploy the containerized Flask application to the created AKS cluster
    enter image description here

    Create a deployment yaml file that refers to the image that I just pushed in my acr
    enter image description here

    Deployment and service yaml file combined

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-app-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      containers:
      - name: test-app
        image: bravinwasike/flask-kubernetes:latest
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 5000
---

apiVersion: v1
kind: Service
metadata:
  name: flask-app-service
spec:
  selector:
    app: flask-app-deployment
  ports:
  - port: 6000
    targetPort: 5000
  type: LoadBalancer

    then apply the yaml

     kubectl apply -f <whatever_you_have_named_your_file>.yaml

    enter image description here

    Done.. Now when you do kubectl get pods your pods are up
    enter image description here

    Your flask app is up

    enter image description here

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search