Question 267158 in Azure Question posted in
The official documentation to get you started can be found here.

Azure – Kyverno admission controller pod stuck on init (aks)

I’m trying to install Kyverno using the guide here:

https://kyverno.io/docs/installation/
https://nirmata.com/2023/10/13/simplify-image-verification-with-kyverno-and-azure-ad-workload-identity-on-aks/

However after successfully installing the helm chart the admission controller pods are perpetually hung up on init

kyverno-admission-controller-6cc87c69c7-d5fgm   0/1     Init:0/1   0          115s
kyverno-admission-controller-6cc87c69c7-mkdx2   0/1     Init:0/1   0          115s
kyverno-admission-controller-6cc87c69c7-zjlqr   0/1     Init:0/1   0          115s
kyverno-background-controller-bc589cf8c-kqw94   1/1     Running    0          115s
kyverno-background-controller-bc589cf8c-mxwrc   1/1     Running    0          115s
kyverno-cleanup-controller-5c9fdbbc5c-4gwxb     0/1     Running    0          115s
kyverno-cleanup-controller-5c9fdbbc5c-nmnr6     0/1     Running    0          115s
kyverno-reports-controller-6cc9fd6979-6s5qv     1/1     Running    0          115s
kyverno-reports-controller-6cc9fd6979-tbvx2     1/1     Running    0          115s

The official documentation says I have to add an annotation on AKS workload due to the Admission enforcer fighting with Kyverno and removing the webhooks.

https://kyverno.io/docs/installation/platform-notes/#notes-for-aks-users
https://learn.microsoft.com/en-us/azure/aks/faq#can-admission-controller-webhooks-impact-kube-system-and-internal-aks-namespaces

Wiped the board clean and reinstalled, but this time with a values. yaml specifically containing the annotation to be passed through

config:
  webhookAnnotations:    
admissions.enforcer/disabled: "true"
admissionController:
  rbac:
    serviceAccount:
      annotations: 
        azure.workload.identity/client-id: ****
  podLabels:
    azure.workload.identity/use: "true"

and running

helm install kyverno kyverno/kyverno 
--namespace kyverno 
--set admissionController.replicas=3 
--set backgroundController.replicas=2 
--set cleanupController.replicas=2 
--set reportsController.replicas=2 
-f values.yaml

Stil the same issue, any idea what im missing or where to dig deeper to get these pods functional?

EDIT:

Getting logs from the init container and it looks like its having trouble communiticating with the kubernetes api and referencing objects within its own namespace

W1017 01:35:47.048643       1 reflector.go:533] k8s.io/[email protected]/tools/cache/reflector.go:231: failed to list *v1.ConfigMap: Get "https://10.0.0.1:443/api/v1/namespaces/kyverno/configmaps?fieldSelector=metadata.name%3Dkyverno-metrics&limit=500&resourceVersion=0": EOF
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Networking Issue

    Checked the container logs of the pods and saw that it could not reference its own config map within its namespace due to an inability for it to communicate with kubernetes api server.

    I caught it in the firewall with the SNI TLS exception, the target subnet was not in any of the route tables so that internal traffic was attempting to egress and hit the firewall, hence the SNI error that prevented the admission controller pods from running


  2. 0

    I followed the exact steps in your document and I was able to successfully deploy kyverno on my AKS cluster.
    I can see you followed the production approach so I followed the same to verify if I am getting the same error.

    Here I am using an AKS cluster named Kyverno-Prod of below config:
    K8s version– 1.26.6
    Type– production with 2 node pools
    Node sizes– Standard_D8ds_v5

    My exact steps–
    Logged into the cluster using az aks get-credentials --resource-group myresourcegroup --name Kyverno-Prod

    Validated my nodes are up and running using kubectl get nodes.

    Checked if I have any available chart using helm list
    Added the kyverno repo using helm repo add kyverno https://kyverno.github.io/kyverno/

    Once added, updated helm and listed all the versions available for kyverno

    enter image description here

    Installed kyverno version 1.10.3 using below command on a dedicated namespace called kyverno

    helm install kyverno kyverno/kyverno -n kyverno --create-namespace 
--set admissionController.replicas=3 
--set backgroundController.replicas=2 
--set cleanupController.replicas=2 
--set reportsController.replicas=2

    It got installed as below

    NAME: kyverno
LAST DEPLOYED: Tue Oct 17 06:04:59 2023
NAMESPACE: kyverno
STATUS: deployed
REVISION: 1
NOTES:
Chart version: 3.0.5
Kyverno version: v1.10.3

Thank you for installing kyverno! Your release is named kyverno

    enter image description here

    And it got successfully deployed with the admission-controller up and running.

    enter image description here

    If your error persists, I recommend you wipe out the setup completely and start fresh.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search