skip to Main Content

I was training models last night on my Ubuntu workstation, and then woke up this morning and saw this message:

Failed to initialize NVML: Driver/library version mismatch

Apparently the NVIDIA system driver automatically updated itself, and now I need to reboot the machine to use my GPUs… How do I prevent automatic updates from NVIDIA?

2

Answers


  1. I think I have had the same issue. It is because of so-called unattended upgrades on Ubuntu.

    Solution 1: check the changed packages and revert the updates

    Check the apt history logs

    less /var/log/apt/history.log
    

    Then you can see what packages have changed. Use apt or aptitude to revert the changes.

    Solution 2: disable unattended upgrades

    Use this guide to disable unattended upgrades. Please consider if this solution works for you as you have to install security updates manually after this change.

    Solution 3: hold specific packages

    Use this guide on how to hold certain packages. Read the apt history as mentioned above to determine which packages you have to put on hold. Probably CUDA related packages such as nvidia-cuda-toolkit. Hard to say since some information is missing from your post. You can see all nvidia related packages like this

    dpkg -l *nvidia*
    

    I hope at least one of my solutions works for you 🙂

    P.S. you have to change the title. NVIDIA isn’t upgrading anything on your system by itself. Ubuntu is the one causing your trouble 😉

    Login or Signup to reply.
  2. If this is because of unattended upgrades (likely), then you should add the NVIDIA drivers to it’s blacklist. Add nvidia- and libnvidia- to /etc/apt/apt.conf.d/50unattended-upgrades like so:

    Unattended-Upgrade::Package-Blacklist {
            "nvidia-";
            "libnvidia-";
            ...
    }
    

    Consider adding additional lines if you see any other NVIDIA driver names in the output of apt list --installed | grep nv

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search