I am trying to create a k8s pod with a docker container image from a private insecure registry. With the latest K8s, I get ErrImagePull as it complains of http vs https for the insecure registry.
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7s default-scheduler Successfully assigned imagename to xxxx
Normal Pulling 7s kubelet Pulling image "registry:5000/imagename:v1”
Warning Failed 6s kubelet Failed to pull image "registry:5000/imagename:v1”: rpc error: code = Unknown desc = failed to pull and unpack image "registry:5000/imagename:v1”: failed to resolve reference "registry:5000/imagename:v1”: failed to do request: Head "https://registry:5000/v2/imagename/manifests/v1”: http: server gave HTTP response to HTTPS client
Warning Failed 6s kubelet Error: ErrImagePull
Normal BackOff 6s kubelet Back-off pulling image "registry:5000/imagename:v1”
Warning Failed 6s kubelet Error: ImagePullBackOff
Before the CRI changes for K8s (i.e. https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/), this has worked for me when I used to have insecure registry configuration in /etc/docker/daemon.json, however with the new changes in K8s, I am trying to understand what is the right configuration needed here.
On the same node, I am able to pull the image from the insecure registry successfully with “docker pull imagename” (since I have /etc/docker/daemon.json configuration for the insecure registry), and I have also verified with containerd command “ctr -i pull —plain-http imagename”.
What configuration is needed for this to work in a pod.yaml for me to pull this image via “kubectl create -f pod.yaml”. It’s just a simple pod.yaml with the image, nothing fancy.
I saw a post on creating secret key for private registry (https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/), but that requires registry authentication token to create a key. I just tried using /etc/docker/daemon.json to create a regcred, but when I used it in imagePullSecrets in pod.yaml, k8s was still complaining of the same http vs https error.
My /etc/docker/daemon.json
{
"insecure-registries": ["registry:5000"]
}
I have a new install of K8s, and containerd is the CRI.
Thank you for your help.
4
Answers
I guess you now would have to configure containerd rather than docker to support your insecure registry. This is done in the
/etc/containerd/config.toml. A config example can be found here:Adding insecure registry in containerd
I faced a similar problem recently about not being able to pull images from an insecure private docker registry using containerd only. I will post my solution here in case it works for your question too. Steps below show the details of how I solved it on Ubuntu Server 20.04 LTS:
insecure private docker registry running at 17.5.20.23:5000
The file
/etc/containerd/config.tomlgets created automatically when you install docker using.debpackages in ubuntu looks as follows:In my first few attempts I was editing this file (which is created automatically) by simply adding the appropriate lines mentioned at Adding insecure registry in containerd at the end of the file and restarting containerd. This made the file look as follows:
This did not work for me. To know why, I checked the configurations with which containerd was running (after
/etc/containerd/config.tomlwas edited) using:The output of the above command is shown below:
In the above output I noticed that the configurations I was trying to add by editing the
/etc/containerd/config.tomlwere actually not there. So somehow containerd was not accepting the added configurations. To fix this I decided to start from scratch by generating a full configuration file and editing it appropriately (according to instructions at Adding insecure registry in containerd).First took a backup of the current containerd configuration file:
Then generated a fresh full configuration file:
This generated a file that looked as follows:
Then edited the above file to look as follows (the edited lines have been appended with the comment ‘# edited line’):
Then I restarted containerd
Finally I tried pulling an image from the private registry using
crictlwhich pulled it successfully:In my case, I simply added
[[registry]]field into/etc/containers/registries.conffile simply because I was usingcrioand restart crio
Please refer
https://github.com/cri-o/cri-o/blob/main/docs/crio.conf.5.md
to make it more simple
find this line in /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".registry.configs]
add these 6 lines below that line
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry-ip:5000"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry-ip:5000".tls]
then search this line
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
and add the below lines below that line
after this i restarted containerd
sudo systemctl restart containerd