I am installing a package whose dependency needs to import numpy
inside its setup.py
. It also needs Cython
to correctly build this dependency. This dependency is scikit-learn==0.21.2
. Here is the setup.py
of my own package called mypkgname
:
from setuptools import find_packages, setup
import Cython # to check that Cython is indeed installed
import numpy # to check that numpy is indeed installed
setup(
name="mypkgname",
version="0.1.0",
packages=find_packages("src", exclude=["tests"]),
package_dir={"": "src"},
install_requires=[
"scikit-learn==0.21.2"
],
)
To make sure that numpy
and Cython
are available inside mypkgname’s setup.py
when pip
installs mypkgname
, I set up the pyproject.toml
like this:
[build-system]
requires = ["setuptools>=40.8.0", "Cython", "numpy>=1.11.0,<=1.22.4", "wheel"]
build-backend = "setuptools.build_meta"
After running pip install -e .
, the import numpy; import Cython
in mypkgname’s setup.py
work, but the import Cython
inside the scikit-learn==0.21.2
install does not:
File "/home/vvvvv/.pyenv/versions/3.8.12/envs/withingswpm04-38/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1016, in get_subpackage
config = self._get_configuration_from_setup_py(
File "/home/vvvvv/.pyenv/versions/3.8.12/envs/withingswpm04-38/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 958, in _get_configuration_from_setup_py
config = setup_module.configuration(*args)
File "sklearn/utils/setup.py", line 8, in configuration
from Cython import Tempita
ModuleNotFoundError: No module named 'Cython'
error: subprocess-exited-with-error
I don’t understand why Cython
is available for the install of my own mypkgname, but not for the install_requires
packages of mypkgname. As if Cython
was uninstalled(?) before installing the install_requires
packages. But by checking the logs with pip install -v
, it does not seem it is the case.
I tried installing setuptools
and Cython
beforehand:
pip install setuptools Cython
pip install -e .
and it works. Indeed, it seems that scikit-learn==0.21.2
needs these 2 packages already installed to be properly installed. However, the scikit-learn version I am trying to install does not specify any Cython
build requirements inside a pyproject.toml
. Here is a link to the setup.py
of the scikit-learn package.
If I just install setuptools
, it still fails with the same error as in the first example (ModuleNotFoundError: No module named 'Cython'
).
Notes:
- I am forced for some out-of-context reasons to use this specific version of scikit-learn.
- I am using Python 3.8.12, pip 23.0.1 and Ubuntu 22.04. I tried this using pyenv and virtualenv with the same results. I also tried with
pip install -v -U --no-cache-dir
.
How can I tell mypkgname that I need numpy
, Cython
and setuptools
installed from the beginning of the pip install
until the very end? I want the numpy
, Cython
and setuptools
packages to be available in the install of mypkgname
and in every install of the install_requires
packages.
Since it will be a package deployed on pypi, I don’t want people to have anything other than pip
and maybe setuptools
already installed when running pip install mypkgname
.
2
Answers
Nowadays, pip lets builds (from sdist to wheel) happen in isolated environments. In each build environment, the build requirements declared in
[build-system]
are installed. Build environments are not reused. Each dependency that requires a build step will get its own ephemeral build environment.My guess is that one dependency (direct or indirect) of your project does not declare its build requirements correctly (maybe this dependency does not even have a
pyproject.toml
file of its own at all).Your project declares a direct dependency on
scikit-learn==0.21.2
. You also mention that you are using Python 3.8. At the following link we can see that PyPI does not have any built distributions (wheels) ofscikit-learn==0.21.2
for Python 3.8:There is a source distribution (sdist,
.tar.gz
) though. And we can see at the following link that this sdist does not contain anypyproject.toml
file in order to declare its build dependencies:Possible work arounds:
scikit-learn==0.21.2
has wheels for Python 3.5 to 3.7, you could agree to use one of those Python interpreters.scikit-learn
for which PyPI possesses wheels compatible with Python 3.8.scikit-learn==0.21.2
compatible with Python 3.8.python -m pip install --no-build-isolation --editable .
.For building packages pip uses build Isolation. I.e. it installs build dependencies into a separate virtual environment, build a package and remove the isolating venv. So build dependencies (in your case Cython and numpy) are removed along with the isolating venv.
You can disable isolation but better and more correct way is to declare your dependencies two times — as build dependencies and runtime dependencies:
When building your package
pip
/setuptools
include the list of dependencies into the wheel and they will be automatically installed in the target environment along with your package.