skip to Main Content

I am following the tutorial here about kserve https://github.com/kserve/modelmesh-serving/blob/main/docs/quickstart.md

Is this my docker&k8s issue? I have spent hours trying to debug but to no avail.

I am getting the following error:


Events:
  Type     Reason     Age    From               Message
  ----     ------     ----   ----               -------
  Normal   Scheduled  4m20s  default-scheduler  Successfully assigned modelmesh-serving/modelmesh-serving-mlserver-0.x-77cc8fd548-xdgvr to minikube
  Normal   Pulling    4m18s  kubelet            Pulling image "kserve/modelmesh:v0.9.0-rc0"
  Normal   Pulled     3m18s  kubelet            Successfully pulled image "kserve/modelmesh:v0.9.0-rc0" in 59.419620166s
  Normal   Created    3m18s  kubelet            Created container mm
  Normal   Started    3m17s  kubelet            Started container mm
  Normal   Pulling    3m17s  kubelet            Pulling image "seldonio/mlserver:0.5.2"
  Warning  Failed     68s    kubelet            Failed to pull image "seldonio/mlserver:0.5.2": rpc error: code = Unknown desc = context deadline exceeded
  Warning  Failed     68s    kubelet            Error: ErrImagePull
  Normal   Pulling    68s    kubelet            Pulling image "kserve/modelmesh-runtime-adapter:v0.9.0-rc0"

3

Answers


  1. The image you try to pull exists and is available here. However, from what I see it’s very big (2.74 GB). That probably means that you experienced a timeout while pull the image.

    Depending on your Kubernetes cluster and use case you can do the following:

    • Try a few times and it’ll pull the image at some point
    • Increase the --runtime-request-timeout in your kubelet
    • Execute docker pull seldonio/mlserver:0.5.2
    Login or Signup to reply.
  2. As mentioned by @Rafał Leszko, The image “seldonio/mlserver:0.5.2” you are trying to pull is very large which possibly throws the error: ErrImagePull: context deadline error. You can still pull the image with docker pull after which the scheduling succeedes.

    When the timeout is exceeded, kubelet will cancel the request, throw out an error. The possible workaround is by setting or increasing the parameter --runtime-request-timeout duration via the config file(in /var/lib/kubelet) by adjusting the timeout and then pull the image by running the command docker pull imagename.

    See kubelet-config-file for more information.

    Login or Signup to reply.
  3. this will most likely have to do with your image being too big.

    One thing I would do is in your deployment file,

    in the spec for the conatainer, add a resources with a request and a limit,eg

     resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search