TL:DR – Sorry if this is basic stuff, I’m learning Kubernetes.
I’m trying to create a Kubernetes deployment in IBM Cloud and it’s failing. The deployment works fine on my local minikube, but fails in IBM Cloud. Am I required to use the IBM Cloud namespace for deployment or should I be able to pull a Docker image into IBM Cloud from Docker public offerings?
Long version
I’m working through this course on IBM Cloud Kubernetes service (https://courses.cognitiveclass.ai/courses/course-v1:CognitiveClass+CO0401EN+v1/info)
In the prereqs, I need to deploy Redis for a guestbook database.
Those steps are:
1. Clone the repo
$ clone https://github.com/IBM/guestbook.git
-
cd to /guestbook/v2
-
Create the Redis controllers and services for both the master and the slave:
$ kubectl create -f redis-master-deployment.yaml
The problem is that when I run:
$ kubectl create -f redis-master-deployment.yaml
The deployment fails:
$ kubectl get deployments -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
redis-master 0/1 1 0 10m redis-master redis:3.2.9 app=redis,role=master
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redis-master-577bc6fbb-b4trd 0/1 ImagePullBackOff 0 14m 172.30.113.18 10.241.0.11 <none> <none>
I’m guessing here, but it looks like the IBM Cloud Kubernetes service cannot contact Docker to get the image.
$ kubectl describe pods redis-master-577bc6fbb-b4trd
Name: redis-master-577bc6fbb-b4trd
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: 10.241.0.11/10.241.0.11
Start Time: Thu, 05 Mar 2020 12:01:45 -0500
Labels: app=redis
pod-template-hash=577bc6fbb
role=master
Annotations: kubernetes.io/psp: ibm-privileged-psp
Status: Pending
IP: 172.30.113.18
Controlled By: ReplicaSet/redis-master-577bc6fbb
Containers:
redis-master:
Container ID:
Image: redis:3.2.9
Image ID:
Port: 6379/TCP
Host Port: 0/TCP
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-rxmp2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-rxmp2:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-rxmp2
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 600s
node.kubernetes.io/unreachable:NoExecute for 600s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/redis-master-577bc6fbb-b4trd to 10.241.0.11
Warning Failed 13m (x2 over 14m) kubelet, 10.241.0.11 Failed to pull image "redis:3.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/redis:3.2.9": failed to resolve reference "docker.io/library/redis:3.2.9": failed to do request: Head https://registry-1.docker.io/v2/library/redis/manifests/3.2.9: dial tcp 34.197.189.129:443: i/o timeout
Warning Failed 12m kubelet, 10.241.0.11 Failed to pull image "redis:3.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/redis:3.2.9": failed to resolve reference "docker.io/library/redis:3.2.9": failed to do request: Head https://registry-1.docker.io/v2/library/redis/manifests/3.2.9: dial tcp 3.224.75.242:443: i/o timeout
Normal Pulling 11m (x4 over 14m) kubelet, 10.241.0.11 Pulling image "redis:3.2.9"
Warning Failed 11m kubelet, 10.241.0.11 Failed to pull image "redis:3.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/redis:3.2.9": failed to resolve reference "docker.io/library/redis:3.2.9": failed to do request: Head https://registry-1.docker.io/v2/library/redis/manifests/3.2.9: dial tcp 54.210.105.17:443: i/o timeout
Normal BackOff 10m (x6 over 14m) kubelet, 10.241.0.11 Back-off pulling image "redis:3.2.9"
Warning Failed 9m21s (x5 over 14m) kubelet, 10.241.0.11 Error: ErrImagePull
Warning Failed 4m38s (x28 over 14m) kubelet, 10.241.0.11 Error: ImagePullBackOff
I then went to my minikube to test and ran the same command
$ kubectl create -f ./redis-master-deployment.yaml
deployment.apps/redis-master created
$ kubectl get deployments -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
redis-master 1/1 1 1 14s redis-master redis:3.2.9 app=redis,role=master
So I just need a point in the direction of where I’m going wrong here.
2
Answers
Found the issue thanks to @bhpratt
The subnets that the worker nodes are on do not have public IPs. They do have a proxy resource but that has to be enabled on the subnet itself.
IBM Cloud Subnet public gateway pic
That was turned off. Flip the switch and the deployment created successfully.
Thanks for the additional info. The answer is your worker nodes need public access in order to pull from public docker. Three solutions:
https://cloud.ibm.com/docs/containers?topic=containers-plan_clusters#gateway
update the YAML to pull from the container registry, and deploy the
containers. Your cluster can pull images from your IBM Cloud registry from the private network.