Question 170588 in Azure Question posted in
The official documentation to get you started can be found here.

Can Azure VM use CuDNN library 8.2.4 if source was compiled with 8.6.0?

I have tried to fit a Keras model on a notebook in Microsoft Azure Machine Learning Studio GPU machine. I have received an error similar to what was described here:

2023-04-27 09:56:21.098249: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.2.4 but source was compiled with: 8.6.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2023-04-27 09:56:21.099011: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at pooling_ops_common.cc:412 : UNIMPLEMENTED: DNN library is not found.
2023-04-27 09:56:21.099050: I tensorflow/core/common_runtime/executor.cc:1197] [/job:localhost/replica:0/task:0/device:GPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): UNIMPLEMENTED: DNN library is not found.
     [[{{node model_2/max_pooling1d_6/MaxPool}}]]
2023-04-27 09:56:21.100704: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.2.4 but source was compiled with: 8.6.0.  CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
2023-04-27 09:56:21.101366: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at pooling_ops_common.cc:412 : UNIMPLEMENTED: DNN library is not found.

What is the solution for Azures’ machines?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I have a friend suggsted the following:

    In any notebook, you run:

    !conda create -n cuda_env python=3.8 numpy scipy pandas scikit-learn matplotlib jupyter ipykernel cudatoolkit=10.1 -c anaconda -y
!pip install tensorflow-gpu==2.4.1
!pip install keras==2.4.3
!python -m ipykernel install --user --name cuda_env --display-name "Python (CUDA)"

    This creates a kernel named Python (CUDA) you can later choose.


  2. 0

    This was a royal pain in the arse to fix – I don’t know why Microsoft haven’t fixed/bumped the cuDNN version from 6.1. The included conda environment with tensorflow doesn’t work.

    Essentially, we need to manually install an older version of tensorflow, or a newer version of cuDNN. As no version of tensorflow is compatable with cuDNN 6.1 we are forced to upgrade cuDNN.

    The solution that works is as follows:

    1. At time of writing – you want cuDNN version 6.8 (for TF 1.12.x) – get cuDNN link from here with your client computer, but stop the link so you can get one with an auth key

    From nVidia website
    enter image description here

    1. Enter the link into the export URL line below
    2. Copy and paste this into your running compute terminal
    3. Wait 5 minutes ☕️
    export URL="PASTE-LINK-HERE"
# ==== DOWNLOAD CUDDN ==== 
curl $URL -o ./cudnn-linux-x86_64-8.6.0.163_cuda11-archive.tar.xz 
sudo tar -xvf ./cudnn-linux-x86_64-8.6.0.163_cuda11-archive.tar.xz
# ==== INSTALL CUDDN ==== 
sudo cp ./cudnn-*-archive/include/cudnn*.h /usr/local/cuda/include 
sudo cp -P ./cudnn-*-archive/lib/libcudnn* /usr/local/cuda/lib64 
sudo chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*
# ==== CONFIGURE DYNAMIC RUNTIME BINDINGS ==== 
sudo ldconfig
# ==== INSTALL CONDA ENV ==== 
conda create -n "tfgpu" python=3.10 -y
conda activate tfgpu
conda install -c conda-forge cudatoolkit=11.8.0 ipykernel -y
python3 -m pip install nvidia-cudnn-cu11==8.6.0.163 tensorflow==2.12.*
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
source $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
python3 -m ipykernel install --user --name tfgpu --display-name "Python (tf-cudnn8.6)"
# ==== VERIFY ==== 
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

    Testing this on the tensorflow mnist example:

    enter image description here

    I hope this helps!

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search