Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Ubuntu – GPU-enabled Kubernetes container is not scheduled

I have installed Nvidia’s GPU operator and have my GPU-enabled node automatically labelled (what I treat as important, long list of other labels is there as well):

nvidia.com/gpu.count=1

Node is seemingly schedule able

Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Tue, 10 Sep 2024 15:05:17 +0000   Tue, 10 Sep 2024 15:05:17 +0000   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Tue, 10 Sep 2024 16:26:50 +0000   Tue, 10 Sep 2024 15:05:04 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Tue, 10 Sep 2024 16:26:50 +0000   Tue, 10 Sep 2024 15:05:04 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Tue, 10 Sep 2024 16:26:50 +0000   Tue, 10 Sep 2024 15:05:04 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Tue, 10 Sep 2024 16:26:50 +0000   Tue, 10 Sep 2024 15:05:04 +0000   KubeletReady                 kubelet is posting ready status

Node also reports as ready in "kubectl get nodes".
However when I’m looking at demo workload, I see

`Warning  FailedScheduling  11s (x17 over 79m)  default-scheduler  0/6 nodes are available: 3 Insufficient nvidia.com/gpu, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/6 nodes are available: 3 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.`

I have even tried to manually add label node with nvidia.com/gpu=1 no luck so far.
I have followed guide from Nvidia https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html.
Deviations from automatic deployment was that I have installed driver (v550) manually as Nvdia hasn’t generated images for Ubuntu 24.
I see an output in nvidia-smi, which essentially should be true as node being labelled by operator.
Kubernetes v1.31.0
Anything else I am missing?

Tried manually label node and re-create pod.
Expectations are to see pod scheduled

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Well, it's embarrassing that I for some reason overlooked failing nvidia-operator-validator pod. Would anybody believe "I bet it was running"? Anyway looking at pod logs or description does not give any information. But going onto worker node where container is scheduled (one with GPU) and running sudo crictl ps -a shows container driver-validation with increasing fail counter. Those logs are actually useful and besides of (in my case) successfully executing nvidia-smi gives an answer:

    ` time="2024-09-12T15:33:34Z" level=info msg="creating symlinks under /dev/char that correspond to NVIDIA character devices" time="2024-09-12T15:33:34Z" level=info msg="Error: error validating driver installation: error creating symlink creator: failed to load NVIDIA kernel modules: failed to load module nvidia: exit status 1; output=modprobe: ERROR: ../libkmod/libkmod-module.c:968 kmod_module_insert_module() could not find module by name='nvidia_current_updates'nmodprobe: ERROR: could not insert 'nvidia_current_updates': Unknown symbol in module, or unknown parameter (see dmesg)nnnFailed to create symlinks under /dev/char that point to all possible NVIDIA character devices.nThe existence of these symlinks is required to address the following bug:nn    https://github.com/NVIDIA/gpu-operator/issues/430nnThis bug impacts container runtimes configured with systemd cgroup management enabled.nTo disable the symlink creation, set the following envvar in ClusterPolicy:nn    validator:n      driver:n        env:n        - name: DISABLE_DEV_CHAR_SYMLINK_CREATIONn          value: "true"" `

    I wasn't savvy enough to understand where would I put that ClusterPolicy, but reinstalling gpu-operator with helm install --wait gpu-operator-1 -n gpu-operator --create-namespace nvidia/gpu-operator --set driver.enabled=false --set validator.driver.env[0].name=DISABLE_DEV_CHAR_SYMLINK_CREATION --set-string validator.driver.env[0].value=true saved the day.


  2. 0

    In our case we using Airflow Kubernetes pod operators with the following affinity:

    affinity = k8s.V1Affinity(
    node_affinity=k8s.V1NodeAffinity(
        preferred_during_scheduling_ignored_during_execution=[
            k8s.V1PreferredSchedulingTerm(
                weight=1,
                preference=k8s.V1NodeSelectorTerm(
                    match_expressions=[
                        k8s.V1NodeSelectorRequirement(
                            key="nvidia.com/gpu.present", operator="In", values=["true"]
                        )
                    ]
                ),
            )
        ]
    ),
)

    Hence, our nodes needed this label:

    nvidia.com/gpu.present=true
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search