I am following the tutorial here about kserve https://github.com/kserve/modelmesh-serving/blob/main/docs/quickstart.md
Is this my docker&k8s issue? I have spent hours trying to debug but to no avail.
I am getting the following error:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m20s default-scheduler Successfully assigned modelmesh-serving/modelmesh-serving-mlserver-0.x-77cc8fd548-xdgvr to minikube
Normal Pulling 4m18s kubelet Pulling image "kserve/modelmesh:v0.9.0-rc0"
Normal Pulled 3m18s kubelet Successfully pulled image "kserve/modelmesh:v0.9.0-rc0" in 59.419620166s
Normal Created 3m18s kubelet Created container mm
Normal Started 3m17s kubelet Started container mm
Normal Pulling 3m17s kubelet Pulling image "seldonio/mlserver:0.5.2"
Warning Failed 68s kubelet Failed to pull image "seldonio/mlserver:0.5.2": rpc error: code = Unknown desc = context deadline exceeded
Warning Failed 68s kubelet Error: ErrImagePull
Normal Pulling 68s kubelet Pulling image "kserve/modelmesh-runtime-adapter:v0.9.0-rc0"
3
Answers
The image you try to pull exists and is available here. However, from what I see it’s very big (
2.74 GB
). That probably means that you experienced a timeout while pull the image.Depending on your Kubernetes cluster and use case you can do the following:
--runtime-request-timeout
in yourkubelet
docker pull seldonio/mlserver:0.5.2
As mentioned by @Rafał Leszko, The image “seldonio/mlserver:0.5.2” you are trying to pull is very large which possibly throws the error: ErrImagePull: context deadline error. You can still pull the image with
docker pull
after which the scheduling succeedes.When the timeout is exceeded, kubelet will cancel the request, throw out an error. The possible workaround is by setting or increasing the parameter
--runtime-request-timeout duration
via the config file(in /var/lib/kubelet) by adjusting the timeout and then pull the image by running the commanddocker pull imagename
.See kubelet-config-file for more information.
this will most likely have to do with your image being too big.
One thing I would do is in your deployment file,
in the spec for the conatainer, add a resources with a request and a limit,eg