Question posted in
The official documentation can be found here.

Redis – Resource Quota applied before LimitRanger in Kubernetes for Pod without specified limits

While using Kubernetes v1.16.8 both the ResourceQuota and LimitRanger are enabled by default and I did not have to add them in my admission plugin in kube-apiserver.
In my case, I use the following LimitRanger 

apiVersion: v1
items:
- apiVersion: v1
  kind: LimitRange
  metadata:
    name: mem-limit-range
    namespace: test
  spec:
    limits:
    - default:
        memory: 512Mi
      defaultRequest:
        memory: 256Mi
      type: Container

and it adds the default limit for memory usage in a new Pod without specified limits, as expected.
The Pod’s definition is as simple as possible:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod-ctr
    image: redis

When I get the created pod described it has acquired the value for limit from the LimitRanger.
Everything is fine!

The problem occurs when i try to enforce a resourcequota for the namespace.
The ResourceQuota looks like this:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-cpu-demo
spec:
  hard:
    limits.cpu: "2"
    limits.memory: 2Gi

When I delete and recreate the pod it will not be created.
The resourcequota will result in the following error:

Error from server (Forbidden): error when creating “test-pod.yml”: pods “test-pod” is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu

In other words, the resourcequota is applied before LimitRanger so it does not let me create pods without a specified limit.

Is there a way to enforce LimitRanger first and then the ResourceQuota?
How do you apply them to your namespaces?

I would like to have developers that do not specify limits in the pod definition to be able to acquire the defaults while enforcing the resource quota as well.

Tags:

2

Answers


  1. 0

    TL;DR:

    Error from server (Forbidden): error when creating “test-pod.yml”: pods “test-pod” is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu

    • You didn’t set a default limit for CPU, according to ResourceQuota Docs:

      If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation.

    • This is why the pod is not being created. Add a cpu-limit.yaml:

    apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-limit-range
  namespace: test
spec:
  limits:
  - default:
      cpu: 1
    defaultRequest:
      cpu: 0.5
    type: Container

    • The limitRanger injects the defaults at container runtime, and yes, it injects the default values prior to the ResourceQuota validation.

    • Other minor issue that I found, is that not all your yamls contains the namespace: test line under metadata, that’s important to assign the resources to the right namespace, I fixed it on the example below.

    Reproduction:

    • Created namespace, applied first the mem-limit and quota, as you mentioned:
    $ kubectl create namespace test
namespace/test created

$ cat mem-limit.yaml 
apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
  namespace: test
spec:
  limits:
  - default:
      memory: 512Mi
    defaultRequest:
      memory: 256Mi
    type: Container

$ cat quota.yaml 
apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-cpu-demo
  namespace: test
spec:
  hard:
    limits.cpu: "2"
    limits.memory: 2Gi

$ kubectl apply -f mem-limit.yaml 
limitrange/mem-limit-range created

$ kubectl apply -f quota.yaml 
resourcequota/mem-cpu-demo created

$ kubectl describe resourcequota -n test
Name:          mem-cpu-demo
Namespace:     test
Resource       Used  Hard
--------       ----  ----
limits.cpu     0     2
limits.memory  0     2Gi

$ kubectl describe limits -n test
Name:       mem-limit-range
Namespace:  test
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   memory    -    -    256Mi            512Mi          -
    • Now if I try to create the pod:
    $ cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: test
spec:
  containers:
  - name: test-pod-ctr
    image: redis

$ kubectl apply -f pod.yaml 
Error from server (Forbidden): error when creating "pod.yaml": pods "test-pod" is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu
    • Same error you faced, because there is no default limits for CPU set. We’ll create and apply it:
    $ cat cpu-limit.yaml 
apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-limit-range
  namespace: test
spec:
  limits:
  - default:
      cpu: 1
    defaultRequest:
      cpu: 0.5
    type: Container

$ kubectl apply -f cpu-limit.yaml 
limitrange/cpu-limit-range created

$ kubectl describe limits cpu-limit-range -n test
Name:       cpu-limit-range
Namespace:  test
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   cpu       -    -    500m             1              -
    • Now with the cpu limitRange in action, let’s create the pod and inspect it:
    $ kubectl apply -f pod.yaml 
pod/test-pod created

$ kubectl describe pod test-pod -n test
Name:         test-pod
Namespace:    test
Status:       Running
...{{Suppressed output}}...

    Limits:
      cpu:     1
      memory:  512Mi
    Requests:
      cpu:        500m
      memory:     256Mi
    • Our pod was created with the enforced limitRange.

    If you have any question let me know in the comments.

    Login or Signup to reply.
  2. 0

    The error clearly defines how you are supposed to handle the issue.

    Error from server (Forbidden): error when creating "test-pod.yml": pods "test-pod" is forbidden: failed quota: mem-cpu-demo: must specify limits.cpu

    Your LimitRange object defines default memory, but not CPU. Your quota restricts both memory and CPU. So you must request for CPU and memory in your Pod manifest. The LimitRange takes care of your default memory, but there is no default CPU request. So in that case, either you must add CPU request in Pod manifest or add default CPU request in your LimitRange.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search