Following the example here https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects, I have created installed metrics-server and modified it as follows:
spec:
containers:
- command:
- metrics-server
- --secure-port=8443
- --kubelet-insecure-tls=true
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
image: docker.io/bitnami/metrics-server:0.3.7-debian-10-r89
imagePullPolicy: IfNotPresent
name: metrics-server
ports:
- containerPort: 8443
name: https
protocol: TCP
resources: {}
My nodes are listed when queried:
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
{"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/nodes"},"items":[{"metadata":{"name":"eo-test-metrics-35lks","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/eo-test-metrics-35lks","creationTimestamp":"2020-11-04T04:05:58Z"},"timestamp":"2020-11-04T04:05:28Z","window":"30s","usage":{"cpu":"770120208n","memory":"934476Ki"}},{"metadata":{"name":"eo-test-metrics-35lkp","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/eo-test-metrics-35lkp","creationTimestamp":"2020-11-04T04:05:58Z"},"timestamp":"2020-11-04T04:05:25Z","window":"30s","usage":{"cpu":"483763591n","memory":"850756Ki"}}]}
But, the HPA targets remain ‘unknown’:
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache <unknown>/50% 1 10 1 31m
Running top nodes works but top pods does not
kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
eo-test-metrics-35lkp 93m 4% 934Mi 30%
eo-test-metrics-35lks 166m 8% 1053Mi 33%
Top pods return error error: Metrics not available
:
kubectl top pods
W1104 11:19:27.854485 62798 top_pod.go:266] Metrics not available for pod default/php-apache-d4cf67d68-blt2c, age: 13h1m51.854463s
error: Metrics not available for pod default/php-apache-d4cf67d68-blt2c, age: 13h1m51.854463s
This is on Kubernetes server version v1.19.3
and metrics server version 0.3.6
The logs from metrics-server
E1104 21:21:56.496129 1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-d4cf67d68-blt2c: no metrics known for pod
E1104 21:22:10.945091 1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-d4cf67d68-blt2c: no metrics known for pod
E1104 21:22:26.496814 1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-d4cf67d68-blt2c: no metrics known for pod
2
Answers
The issue is resolved when Docker 19.03 is used on Kubernetes version 1.19 in relation to this upstream issue: https://github.com/kubernetes/kubernetes/issues/94281
Use this version of metrics server
Then follow the same instructions as mentioned in this link
It will increase the number of pods as the load goes up
BUT I noticed the autoscaler does not scale down the deployment upon the load stopping. It might take some time until it scales down again
Note: This version of metrics server could only be used in a dev or learning environment.
I used Katakoda environment for testing the answer.