Question 242519 in Azure Question posted in
The official documentation to get you started can be found here.

Azure – How do I trigger AKS cluster auto scaling when the node volume limit is reached

In Azure Kubernetes Service, the number of Azure Disks you can attach to a Node depends on the VM size. Eg. https://learn.microsoft.com/en-us/azure/virtual-machines/dv4-dsv4-series. See the data disks column.

I have pods that attach a disk using the disk.csi.azure.com provisioner.
This works as long as there are less then 4 concurrent pods like that.
The fifth stalls and errors out, because the VM size we use only supports 4 disks.

How can I schedule pods with the disk limit in mind, such that the auto scaler scales up the cluster when the disk limit is reached on all the nodes?

Based on this I hoped AKS would expose this limit as an extended resource, but there is no hint of it in the capacity status of the nodes.
Affinities seem to be binary rules for putting pods on the node; I see no way to model "at most x pods that do y can go to a node".

Edit for context: The pods are CI jobs generated by gitlab-runner. I want to use persistent volumes as build areas to avoid having to clone the entire project from scratch in every job. From the point of view of Kubenernetes this looks like dynamically generated Pods, each of which requests to mount one of a previously defined set of Persistent Volume Claims. The PVC:s are fungible (they cache the same clone), but the configuration forces us to request one by some specific name for each pod.

Tags:

2

Answers


  1. 0

    You will probably need to create a custom solution because AKS does not come with this capabilities by default.

    In order to manage storage and disc attachment limits in the Azure Kubernetes Service, you can make use of Kubernetes Persistent Volumes and Persistent Volume Claims. These tools will help you manage storage in a more dynamic and abstracted manner.

    1. Understand persistent volumes and claims based on persistent volumes.

    Persistent Volume: A PV is a storage unit in the cluster that has either been provided statically using Storage Classes or by an administrator. The same as nodes and pods, PVs are resources in the cluster.

    Continuous Volume Claim: A PVC is a user’s request for storage. It is comparable to a pod asking for CPU and memory. PVCs are utilised by pods and utilise PV resources.

    1. Establish Storage Classes

    If you use dynamic storage provisioning, create storage classes that outline the different "classes" of storage you want to provide. The type of underlying storage characteristics may be specified by a storage class.

    Example of a Storage Class for AKS:

    kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
provisioner: kubernetes.io/azure-disk
parameters:
  storageaccounttype: Standard_LRS

    Apply the Storage Class to your cluster:

    kubectl apply -f azure-standard-storage-class.yaml
    1. Determine Persistent Volume Claims:

    To request the storage you require, include PVCs in your Kubernetes pod setups. Your pods are connected to the underlying PVs by PVCs.

    Example PVC :

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 10Gi
    1. Apply the PVC to your cluster:
          kubectl apply -f my-pvc.yaml

    4. Attach PVCs to Pods:

    Refer to the PVCs you’ve defined in your pod setups. It connects the pod to the storage that was mentioned in the PVC.

    apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: my-container
      image: nginx
      volumeMounts:
        - name: my-volume
          mountPath: /data
  volumes:
    - name: my-volume
      persistentVolumeClaim:
        claimName: my-pvc

    Apply the pod to your cluster:

    kubectl apply -f my-pod.yaml

    Here, the PVC called "my-pvc" is being used by the pod "my-pod" to request and access storage. The storage is mounted inside the pod at the path ‘/data’.

    5. Scaling Pods :

    Now, if you choose dynamic provisioning, Kubernetes will automatically provision extra PVs and connect them to the new pods through PVCs when you need to scale your application by adding more pods. As a result, you may manage storage dynamically without being concerned about node-specific disc attachment restrictions.

    PODs auto scaling automatically when load increase.

    enter image description here

    Load Increase:

    enter image description here

    Login or Signup to reply.
  2. 0

    You can think about using S3 buckets instead of attached disks, they may be even better suited for high (unknown) number of pods requiring storage with pod-specific naming conventions.

    Another possibility is to try out a topology spread constraint of the topologyKey "node", e.g.

    kind: Pod
apiVersion: v1
metadata:
  name: mypod
  labels:
    foo: bar
spec:
  topologySpreadConstraints:
  - maxSkew: 4
    topologyKey: node
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        foo: bar
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.1

    https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/#example-one-topologyspreadconstraint

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search