We are working on an AI project which amongst others calculates the position of a human body lying in a bed. External supporters provided us a code package which does this job and calculates the deviation between the real position of the body (detected by a camera) and the predicted position.
That code package has been developed on Windows and now needs to be ported to Ubuntu 20.04. Until now, in the code package for Ubuntu, we just changed some path names (backslash to slash) to make it work.
If we now run – the exactly same – code with the exactly same training and test data under Windows 10 and Ubuntu we observe the following strange behaviour:
The deviation between real body positions and predicted body positions is 25mm (average value) in Windows, which is a good result for our purposes and excepted as a match. In Ubuntu the deviation is 150mm (average value), which is a not usable result.
We also ran the Windows code package in a Windows 10 VM (Virtualbox) on the respective Ubuntu machine. The result is close to the result we get when running the code under ‘plain‘ Ubuntu (not usable).
When running under Ubuntu, there is no difference whether we utilize CPU or GPU.
The code is written in Python 3 and executed under Python 3.7.12 in an Anaconda 3 virtual environment.
We assume that Linux/Ubuntu does not fully support a certain CPU instruction set. but at the end we do not have an explanation for this behaviour. Does anybody have an idea?
Our CPU:
Intel Core I7 (Caby Lake)
Our env.yaml for conda:
name: cosy-bunk-3.7
channels:
- conda-forge
- anaconda
- defaults
- pytorch
dependencies:
- python=3.7
- numpy
- matplotlib
- pandas
- ipykernel
- torchvision
- pytorch
- cudatoolkit=10.2
- torchaudio
- plotly
- scikit-learn
- pip
- pip:
- optuna
- dash
Output of ‘conda list‘ under Ubuntu:
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
alembic 1.8.1 pypi_0 pypi
alsa-lib 1.2.6.1 h7f98852_0 conda-forge
attr 2.5.1 h166bdaf_0 conda-forge
attrs 22.1.0 pypi_0 pypi
autopage 0.5.1 pypi_0 pypi
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
blas 1.1 openblas conda-forge
brotli 1.0.9 pypi_0 pypi
brotli-bin 1.0.9 h166bdaf_7 conda-forge
brotlipy 0.7.0 py37h540881e_1004 conda-forge
ca-certificates 2022.6.15 ha878542_0 conda-forge
certifi 2022.6.15 py37h89c1867_0 conda-forge
cffi 1.15.1 py37h43b0acd_0 conda-forge
charset-normalizer 2.1.0 pyhd8ed1ab_0 conda-forge
click 8.1.3 pypi_0 pypi
cliff 3.10.1 pypi_0 pypi
cmaes 0.8.2 pypi_0 pypi
cmd2 2.4.2 pypi_0 pypi
colorlog 6.6.0 pypi_0 pypi
cryptography 37.0.4 py37h38fbfac_0 conda-forge
cudatoolkit 10.2.89 h713d32c_10 conda-forge
cudnn 7.6.5.32 h01f27c4_1 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
dash 2.6.1 pypi_0 pypi
dash-core-components 2.0.0 pypi_0 pypi
dash-html-components 2.0.0 pypi_0 pypi
dash-table 5.0.0 pypi_0 pypi
dbus 1.13.6 h5008d03_3 conda-forge
debugpy 1.6.0 py37hd23a5d3_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
expat 2.4.8 h27087fc_0 conda-forge
fftw 3.3.10 nompi_ha7695d1_103 conda-forge
flask 2.2.2 pypi_0 pypi
flask-compress 1.12 pypi_0 pypi
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.0 h8e229c2_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.34.4 py37h540881e_0 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
gettext 0.19.8.1 h73d1719_1008 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
glib 2.72.1 h6239696_0 conda-forge
glib-tools 2.72.1 h6239696_0 conda-forge
greenlet 1.1.2 pypi_0 pypi
gst-plugins-base 1.20.3 hf6a322e_0 conda-forge
gstreamer 1.20.3 hd4edc92_0 conda-forge
icu 70.1 h27087fc_0 conda-forge
idna 3.3 pyhd8ed1ab_0 conda-forge
importlib-metadata 4.12.0 pypi_0 pypi
importlib-resources 5.9.0 pypi_0 pypi
ipykernel 6.15.1 pyh210e3f2_0 conda-forge
ipython 7.33.0 py37h89c1867_0 conda-forge
itsdangerous 2.1.2 pypi_0 pypi
jack 1.9.18 h8c3723f_1002 conda-forge
jedi 0.18.1 pyhd8ed1ab_2 conda-forge
jinja2 3.1.2 pypi_0 pypi
joblib 1.1.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h166bdaf_2 conda-forge
jupyter_client 7.3.4 pyhd8ed1ab_0 conda-forge
jupyter_core 4.11.1 py37h89c1867_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 py37h7cecad7_0 conda-forge
krb5 1.19.3 h3790be6_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libblas 3.9.0 15_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_7 conda-forge
libbrotlidec 1.0.9 h166bdaf_7 conda-forge
libbrotlienc 1.0.9 h166bdaf_7 conda-forge
libcap 2.64 ha37c62d_0 conda-forge
libcblas 3.9.0 15_linux64_openblas conda-forge
libclang 14.0.6 default_h2e3cab8_0 conda-forge
libclang13 14.0.6 default_h3a83d3e_0 conda-forge
libcups 2.3.3 hf5a7f15_1 conda-forge
libdb 6.2.32 h9c3ff4c_0 conda-forge
libdeflate 1.13 h166bdaf_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libevent 2.1.10 h9b69904_4 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libflac 1.3.4 h27087fc_0 conda-forge
libgcc-ng 12.1.0 h8d9b700_16 conda-forge
libgfortran-ng 12.1.0 h69a702a_16 conda-forge
libgfortran5 12.1.0 hdcd56e2_16 conda-forge
libglib 2.72.1 h2d90d5f_0 conda-forge
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 15_linux64_openblas conda-forge
libllvm14 14.0.6 he0ac6c6_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libogg 1.3.4 h7f98852_1 conda-forge
libopenblas 0.3.20 pthreads_h78a6416_1 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpng 1.6.37 h753d276_3 conda-forge
libpq 14.4 hd77ab85_0 conda-forge
libprotobuf 3.20.1 h6239696_0 conda-forge
libsndfile 1.0.31 h9c3ff4c_1 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libstdcxx-ng 12.1.0 ha89aaad_16 conda-forge
libtiff 4.4.0 h0e0dad5_3 conda-forge
libtool 2.4.6 h9c3ff4c_1008 conda-forge
libudev1 249 h166bdaf_4 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp 1.2.3 h522a892_1 conda-forge
libwebp-base 1.2.3 h166bdaf_2 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge
libxml2 2.9.14 h22db469_3 conda-forge
libzlib 1.2.12 h166bdaf_2 conda-forge
llvm-openmp 14.0.4 he0ac6c6_0 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
magma 2.5.4 h5da55e3_2 conda-forge
mako 1.2.1 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
matplotlib 3.5.2 py37h89c1867_1 conda-forge
matplotlib-base 3.5.2 py37hc347a89_1 conda-forge
matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge
mkl 2022.1.0 h84fe81f_915 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mysql-common 8.0.30 haf5c9bc_0 conda-forge
mysql-libs 8.0.30 h28c427c_0 conda-forge
nccl 2.13.4.1 h1a5f58c_0 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
nest-asyncio 1.5.5 pyhd8ed1ab_0 conda-forge
ninja 1.11.0 h924138e_0 conda-forge
nspr 4.32 h9c3ff4c_1 conda-forge
nss 3.78 h2350873_0 conda-forge
numpy 1.21.6 py37h976b520_0 conda-forge
openblas 0.3.20 pthreads_h320a7e8_1 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1q h166bdaf_0 conda-forge
optuna 2.10.1 pypi_0 pypi
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.5 py37he8f5f7f_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
pbr 5.9.0 pypi_0 pypi
pcre 8.45 h9c3ff4c_0 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 9.2.0 py37h44f0d7a_0 conda-forge
pip 22.2.2 pyhd8ed1ab_0 conda-forge
plotly 5.9.0 pyhd8ed1ab_0 conda-forge
ply 3.11 py_1 conda-forge
portaudio 19.6.0 h57a0ea0_5 conda-forge
prettytable 3.3.0 pypi_0 pypi
prompt-toolkit 3.0.30 pyha770c72_0 conda-forge
psutil 5.9.1 py37h540881e_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pulseaudio 14.0 h7f54b18_8 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.12.0 pyhd8ed1ab_0 conda-forge
pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyperclip 1.8.2 pypi_0 pypi
pyqt 5.15.7 py37hf30b843_0 conda-forge
pyqt5-sip 12.11.0 py37hd23a5d3_0 conda-forge
pysocks 1.7.1 py37h89c1867_5 conda-forge
python 3.7.12 hb7a2778_100_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.7 2_cp37m conda-forge
pytorch 1.12.0 cuda102py37haad9b4f_202 conda-forge
pytorch-mutex 1.0 cuda pytorch
pytz 2022.1 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 pypi_0 pypi
pyzmq 23.2.0 py37h0c0c2a8_0 conda-forge
qt-main 5.15.4 ha5833f6_2 conda-forge
readline 8.1.2 h0f457ee_0 conda-forge
requests 2.28.1 pyhd8ed1ab_0 conda-forge
scikit-learn 1.0.2 py37hf9e9bfc_0 conda-forge
scipy 1.7.3 py37hf2a6cf1_0 conda-forge
setuptools 59.8.0 py37h89c1867_1 conda-forge
sip 6.6.2 py37hd23a5d3_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sleef 3.5.1 h9b69904_2 conda-forge
sqlalchemy 1.4.40 pypi_0 pypi
sqlite 3.39.2 h4ff8645_0 conda-forge
stevedore 3.5.0 pypi_0 pypi
tbb 2021.5.0 h924138e_1 conda-forge
tenacity 8.0.1 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
torchaudio 0.12.0 py37_cu102 pytorch
torchvision 0.13.0 cuda102py37h9785060_0 conda-forge
tornado 6.2 py37h540881e_0 conda-forge
tqdm 4.64.0 pypi_0 pypi
traitlets 5.3.0 pyhd8ed1ab_0 conda-forge
typing-extensions 4.3.0 hd8ed1ab_0 conda-forge
typing_extensions 4.3.0 pyha770c72_0 conda-forge
unicodedata2 14.0.0 py37h540881e_1 conda-forge
urllib3 1.26.11 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
werkzeug 2.2.2 pypi_0 pypi
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
xcb-util 0.4.0 h166bdaf_0 conda-forge
xcb-util-image 0.4.0 h166bdaf_0 conda-forge
xcb-util-keysyms 0.4.0 h166bdaf_0 conda-forge
xcb-util-renderutil 0.3.9 h166bdaf_0 conda-forge
xcb-util-wm 0.4.1 h166bdaf_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.2.5 h516909a_1 conda-forge
zeromq 4.3.4 h9c3ff4c_1 conda-forge
zipp 3.8.1 pypi_0 pypi
zlib 1.2.12 h166bdaf_2 conda-forge
zstd 1.5.2 h8a70e8d_3 conda-forge
Output of ‘conda list‘ under Windows:
# Name Version Build Channel
alembic 1.8.1 pypi_0 pypi
attrs 22.1.0 pypi_0 pypi
autopage 0.5.1 pypi_0 pypi
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
blas 1.0 mkl anaconda
brotli 1.0.9 pypi_0 pypi
brotli-bin 1.0.9 h8ffe710_7 conda-forge
ca-certificates 2022.6.15 h5b45459_0 conda-forge
certifi 2022.6.15 py37h03978a9_0 conda-forge
cffi 1.15.1 py37hd8e9650_0 conda-forge
click 8.1.3 pypi_0 pypi
cliff 3.10.1 pypi_0 pypi
cmaes 0.8.2 pypi_0 pypi
cmd2 2.4.2 pypi_0 pypi
colorama 0.4.5 pyhd8ed1ab_0 conda-forge
colorlog 6.6.0 pypi_0 pypi
cudatoolkit 11.3.1 h280eb24_10 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
dash 2.6.1 pypi_0 pypi
dash-core-components 2.0.0 pypi_0 pypi
dash-html-components 2.0.0 pypi_0 pypi
dash-table 5.0.0 pypi_0 pypi
debugpy 1.6.0 py37hf2a7229_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
flask 2.2.2 pypi_0 pypi
flask-compress 1.12 pypi_0 pypi
fonttools 4.34.4 py37hcc03f2d_0 conda-forge
freetype 2.10.4 h546665d_1 conda-forge
future 0.18.2 py37h03978a9_5 conda-forge
greenlet 1.1.2 pypi_0 pypi
icu 58.2 vc14hc45fdbb_0 [vc14] anaconda
importlib-metadata 4.12.0 pypi_0 pypi
importlib-resources 5.9.0 pypi_0 pypi
intel-openmp 2022.1.0 h57928b3_3787 conda-forge
ipykernel 6.15.1 pyh025b116_0 conda-forge
ipython 7.33.0 py37h03978a9_0 conda-forge
itsdangerous 2.1.2 pypi_0 pypi
jedi 0.18.1 pyhd8ed1ab_2 conda-forge
jinja2 3.1.2 pypi_0 pypi
joblib 1.1.0 pyhd8ed1ab_0 conda-forge
jpeg 9b vc14h4d7706e_1 [vc14] anaconda
jupyter_client 7.3.4 pyhd8ed1ab_0 conda-forge
jupyter_core 4.11.1 py37h03978a9_0 conda-forge
kiwisolver 1.4.4 py37h8c56517_0 conda-forge
libblas 3.9.0 12_win64_mkl conda-forge
libbrotlicommon 1.0.9 h8ffe710_7 conda-forge
libbrotlidec 1.0.9 h8ffe710_7 conda-forge
libbrotlienc 1.0.9 h8ffe710_7 conda-forge
libcblas 3.9.0 12_win64_mkl conda-forge
liblapack 3.9.0 12_win64_mkl conda-forge
libpng 1.6.37 h1d00b33_2 conda-forge
libsodium 1.0.18 h8d14728_1 conda-forge
libsqlite 3.39.2 h8ffe710_1 conda-forge
libtiff 4.0.8 vc14h04e2a1e_10 [vc14] anaconda
libuv 1.44.2 h8ffe710_0 conda-forge
libwebp 1.2.4 h8ffe710_0 conda-forge
libwebp-base 1.2.4 h8ffe710_0 conda-forge
m2w64-gcc-libgfortran 5.3.0 6 conda-forge
m2w64-gcc-libs 5.3.0 7 conda-forge
m2w64-gcc-libs-core 5.3.0 7 conda-forge
m2w64-gmp 6.1.0 2 conda-forge
m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge
mako 1.2.1 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
matplotlib 3.5.3 py37h03978a9_0 conda-forge
matplotlib-base 3.5.3 py37h54234da_0 conda-forge
matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge
mkl 2021.4.0 h0e2418a_729 conda-forge
mkl-service 2.4.0 py37h75fcce0_0 conda-forge
msys2-conda-epoch 20160418 1 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
nest-asyncio 1.5.5 pyhd8ed1ab_0 conda-forge
ninja 1.11.0 h2d74725_0 conda-forge
numpy 1.21.6 py37h2830a78_0 conda-forge
openssl 1.1.1q h8ffe710_0 conda-forge
optuna 2.10.1 pypi_0 pypi
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.5 py37h9386db6_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
pbr 5.10.0 pypi_0 pypi
pickleshare 0.7.5 py_1003 conda-forge
pillow 9.2.0 py37hdc2b20a_1
pip 22.2.2 pyhd8ed1ab_0 conda-forge
plotly 5.10.0 pyhd8ed1ab_0 conda-forge
prettytable 3.3.0 pypi_0 pypi
prompt-toolkit 3.0.30 pyha770c72_0 conda-forge
psutil 5.9.1 py37hcc03f2d_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.12.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyperclip 1.8.2 pypi_0 pypi
pyqt 5.9.2 py37h6538335_4 conda-forge
pyreadline3 3.4.1 pypi_0 pypi
python 3.7.12 h7840368_100_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.7 2_cp37m conda-forge
pytorch 1.10.2 cpu_py37h907fbb5_0 anaconda
pytz 2022.2 pyhd8ed1ab_0 conda-forge
pywin32 303 py37hcc03f2d_0 conda-forge
pyyaml 6.0 pypi_0 pypi
pyzmq 23.2.0 py37hcce574b_0 conda-forge
qt 5.9.7 vc14h73c81de_0 [vc14] anaconda
scikit-learn 1.0.2 py37hcabfae0_0 conda-forge
scipy 1.7.3 py37hb6553fb_0 conda-forge
setuptools 59.8.0 py37h03978a9_1 conda-forge
sip 4.19.8 py37h6538335_1000 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sqlalchemy 1.4.40 pypi_0 pypi
sqlite 3.39.2 h8ffe710_1 conda-forge
stevedore 3.5.0 pypi_0 pypi
tbb 2021.5.0 h2d74725_1 conda-forge
tenacity 8.0.1 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tk 8.6.7 vc14hb68737d_1 [vc14] anaconda
torchaudio 0.10.2 py37_cu113 pytorch
torchvision 0.11.3 py37_cu113 pytorch
tornado 6.2 py37hcc03f2d_0 conda-forge
tqdm 4.64.0 pypi_0 pypi
traitlets 5.3.0 pyhd8ed1ab_0 conda-forge
typing-extensions 4.3.0 hd8ed1ab_0 conda-forge
typing_extensions 4.3.0 pyha770c72_0 conda-forge
ucrt 10.0.20348.0 h57928b3_0 conda-forge
unicodedata2 14.0.0 py37hcc03f2d_1 conda-forge
vc 14.2 hb210afc_6 conda-forge
vs2015_runtime 14.29.30037 h902a5da_6 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
werkzeug 2.2.2 pypi_0 pypi
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
zeromq 4.3.4 h0e60522_1 conda-forge
zipp 3.8.1 pypi_0 pypi
zlib 1.2.11 vc14h1cdd9ab_1 [vc14] anaconda
3
Answers
After deeper research we discovered the error in the prepairing of the data caused by the different handling of os.listdir() in python. The order is not the same on windows and linux machines.
I guess there is some version issues in the conda env. When you use python 3 the conda env manages all the compatible packages to your env. So when you use python 3.7 the packages are updated. You will have different versions for each package . i can suggest you can either use pip or docker containers.
If you want more consistency in your environment solves, then add specificity to your YAML. I would recommend specifying every package that is important to you up through minor version. That is what is recommended for Conda YAMLs amongst the Snakemake community.
Also, the channel order doesn’t make sense, and is potentially a source of inconsistency. If using PyTorch, then the
pytorchchannel should absolutely be top priority. Also,anacondais a subset ofdefaults, so that’s unnecessary. I’d change the channels to:And the
nodefaultsat the end is a directive to ignore any system- or user-level configured channels, ensuring only the YAML channels are considered.