When building a container (through docker build, docker run or docker-compose) using NVIDIA containers, I get the following error, somewhat randomly. When I usually start building the container, it works the first time. But next few times usually don’t work.
------
> [internal] load metadata for nvcr.io/nvidia/l4t-base:r32.5.0:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0:
failed to create LLB definition: failed to authorize:
rpc error: code = Unknown desc = failed to fetch anonymous token: unexpected status: 401 Unauthorized
My current workaround is to use a slightly different image temporarily whilst I’m adjusting other parts of the Dockerfile.
4
Answers
You have to
docker login nvcr.io
Most of this is documented in the setup docs, unfortunately
nvcr.io
doesn't have very good SEO or UX, so search results and most attempts don't show this page. This is, confusingly, different to ngc catalog /docker login ngc.nvidia.com
.curl -O https://ngc.nvidia.com/downloads/ngccli_mac.zip && unzip ngccli_mac.zip && chmod u+x ngc
mv ngc /usr/local/bin/ngc
ngc config set
docker login nvcr.io
Tips
Make sure you login to the right container registry. It's in the name of the docker image you are using. e.g.
nvcr.io/nvidia/tensorrt:22.01-py3
usesnvcr.io
.Explanation
With the benefit of hindsight, I understood
failed to fetch anonymous token: unexpected status: 401 Unauthorized
. I was not authenticated (or more correctly, I was authenticated to Docker Hub and NGC Catalog instead of nvcr.io). It was giving me an anonymous token, which hit the rate limit. I needed to create an account to get higher limits.Updating Docker from my version (4.5.0) to the latest version (4.9.0) helped me.
If you happen to get this error and it has worked before, you probably have run into a download limit from nvcr. To circumvent it, it suffices to just restart docker.
For Docker desktop click on the docker icon in the taskbar and click restart.
Or on Mac your can use cmd + r.
Restarting docker server solved the issue