I was training models last night on my Ubuntu workstation, and then woke up this morning and saw this message:
Failed to initialize NVML: Driver/library version mismatch
Apparently the NVIDIA system driver automatically updated itself, and now I need to reboot the machine to use my GPUs… How do I prevent automatic updates from NVIDIA?
2
Answers
I think I have had the same issue. It is because of so-called unattended upgrades on Ubuntu.
Solution 1: check the changed packages and revert the updates
Check the apt history logs
Then you can see what packages have changed. Use
apt
oraptitude
to revert the changes.Solution 2: disable unattended upgrades
Use this guide to disable unattended upgrades. Please consider if this solution works for you as you have to install security updates manually after this change.
Solution 3: hold specific packages
Use this guide on how to hold certain packages. Read the apt history as mentioned above to determine which packages you have to put on hold. Probably CUDA related packages such as
nvidia-cuda-toolkit
. Hard to say since some information is missing from your post. You can see all nvidia related packages like thisI hope at least one of my solutions works for you 🙂
P.S. you have to change the title. NVIDIA isn’t upgrading anything on your system by itself. Ubuntu is the one causing your trouble 😉
If this is because of unattended upgrades (likely), then you should add the NVIDIA drivers to it’s blacklist. Add
nvidia-
andlibnvidia-
to/etc/apt/apt.conf.d/50unattended-upgrades
like so:Consider adding additional lines if you see any other NVIDIA driver names in the output of
apt list --installed | grep nv