skip to Main Content

Following the example here https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects, I have created installed metrics-server and modified it as follows:

    spec:
      containers:
      - command:
        - metrics-server
        - --secure-port=8443
        - --kubelet-insecure-tls=true
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        image: docker.io/bitnami/metrics-server:0.3.7-debian-10-r89
        imagePullPolicy: IfNotPresent
        name: metrics-server
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP
        resources: {}

My nodes are listed when queried:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
{"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/nodes"},"items":[{"metadata":{"name":"eo-test-metrics-35lks","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/eo-test-metrics-35lks","creationTimestamp":"2020-11-04T04:05:58Z"},"timestamp":"2020-11-04T04:05:28Z","window":"30s","usage":{"cpu":"770120208n","memory":"934476Ki"}},{"metadata":{"name":"eo-test-metrics-35lkp","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/eo-test-metrics-35lkp","creationTimestamp":"2020-11-04T04:05:58Z"},"timestamp":"2020-11-04T04:05:25Z","window":"30s","usage":{"cpu":"483763591n","memory":"850756Ki"}}]}

But, the HPA targets remain ‘unknown’:

kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache <unknown>/50% 1 10 1 31m

Running top nodes works but top pods does not

kubectl top nodes
NAME                    CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
eo-test-metrics-35lkp   93m          4%     934Mi           30%
eo-test-metrics-35lks   166m         8%     1053Mi          33%

Top pods return error error: Metrics not available:

 kubectl top pods
W1104 11:19:27.854485   62798 top_pod.go:266] Metrics not available for pod default/php-apache-d4cf67d68-blt2c, age: 13h1m51.854463s
error: Metrics not available for pod default/php-apache-d4cf67d68-blt2c, age: 13h1m51.854463s

This is on Kubernetes server version v1.19.3 and metrics server version 0.3.6

The logs from metrics-server

E1104 21:21:56.496129       1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-d4cf67d68-blt2c: no metrics known for pod
E1104 21:22:10.945091       1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-d4cf67d68-blt2c: no metrics known for pod
E1104 21:22:26.496814       1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-d4cf67d68-blt2c: no metrics known for pod

2

Answers


  1. Chosen as BEST ANSWER

    The issue is resolved when Docker 19.03 is used on Kubernetes version 1.19 in relation to this upstream issue: https://github.com/kubernetes/kubernetes/issues/94281


  2. Use this version of metrics server

    git clone https://github.com/kodekloudhub/kubernetes-metrics-server
    kubectl apply -f kubernetes-metrics-server
    

    Then follow the same instructions as mentioned in this link

    It will increase the number of pods as the load goes up

    BUT I noticed the autoscaler does not scale down the deployment upon the load stopping. It might take some time until it scales down again

    Note: This version of metrics server could only be used in a dev or learning environment.

    I used Katakoda environment for testing the answer.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search