I’m new to rapids ai libraries. I’ve an existing conda environment yaml file where I’m using python 3.8.5, tensorflow 2.7.0, opencv-python-headless 4.5.5.62, numpy 1.22.2, pandas 1.4.1, pandas-profiling 3.1.0, seaborn 0.11.2, matplotlib 3.5.1, jupyterlab 3.2.9.
I’ve added below 2 channels to the file:
- rapidsai
- nvidia
And the below packages:
- cudf=22.02
- cudatoolkit=11.5
The installation is going on for hours and while trying to find incompatible packages, it seems to be in some sort of loop as I keep seeing below message multiple times in the terminal:
Found conflicts! Looking for incompatible packages.
Is there any known issue/limitations that I should be aware of?
Since we don’t get interactive shell on GPU h/w easily, I’m trying the conda environment update on non-GPU machine and once installed, I’ll try cudf package on GPU machine.
EDIT1:
This is what I have as working without tensorflow and tensorflow-hub
name: cudf-env
channels:
- default
- rmg
- rapidsai
- nvidia
- numba
- conda-forge
- anaconda
dependencies:
- glibc=2.19
- libgcc-ng=11.2.0
- python=3.8.5
- cudf=22.02
- cudatoolkit=11.2
- pytest=6.1.2
- pandas=1.3.5
- numpy=1.21.5
- requests=2.25.0
- scikit-learn=0.24.2
- dill=0.3.4
- tqdm=4.62.3
- ruamel.yaml=0.17.19
- yappi=1.3.3
- black=22.1.0
- pillow=9.0.1
- jupyterlab=3.2.9
- matplotlib=3.5.1
- seaborn=0.11.2
- plotly=5.6.0
- pandas-profiling=3.1.0
- black=22.1.0
# - pip
# - pip:
# - tensorflow==2.7.0
# - tensorflow-hub==0.12.0
# - opencv-python-headless==4.5.5.62
# - opencv-contrib-python-headless==4.5.5.62
Now, if I uncomment the pip section, the anaconda crashes while creating the environment. Since pip may not be supported with cudf, I tried following as well, the conda create env
hangs while solving the environment (strangely, it’s not resolving from conda-forge channel):
name: cudf-env
channels:
- default
- rmg
- rapidsai
- nvidia
- numba
- conda-forge
- anaconda
dependencies:
- glibc=2.19
- libgcc-ng=11.2.0
- python=3.8.5
- cudf=22.02
- cudatoolkit=11.2
- pytest=6.1.2
- pandas=1.3.5
- numpy=1.21.5
- requests=2.25.0
- scikit-learn=0.24.2
- dill=0.3.4
- tqdm=4.62.3
- ruamel.yaml=0.17.19
- yappi=1.3.3
- black=22.1.0
- pillow=9.0.1
- jupyterlab=3.2.9
- matplotlib=3.5.1
- seaborn=0.11.2
- plotly=5.6.0
- pandas-profiling=3.1.0
- black=22.1.0
- tensorflow
- tensorflow-hub
My system details are following:
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
$ uname -r
3.10.0-1127.10.1.el7.x86_64
EDIT2: I forgot to mention that if I comment out glibc, cudf and cudatoolkit, the tensorflow installation through pip works fine.
2
Answers
I was finally able to complete the installation of cudf along with all the required packages after I reimaged my machine to Ubuntu20.04.
And I dropped below 2 packages as with Ubuntu20.04 I don't need those -
Challenges updating conda environments can be tricky to untangle, but in this case the issue (in terms of cuDF) is likely the pinning of pandas to 1.4.1.
cuDF does not yet support
pandas=1.4.1
. The cuDF nightly packages currently supportpandas >=1.0,<1.4.0dev0
(everything from 1.0 up to a dev build of 1.4.0). For the moment, if you switch topandas=1.3.5
things will work (assuming the other packages are compatible — Tensorflow likely will require CUDA Toolkit 11.2, not 11.5, and some others might not be either). You can also let conda solve for the appropriate pandas version.