skip to Main Content

TL:DR – Sorry if this is basic stuff, I’m learning Kubernetes.
I’m trying to create a Kubernetes deployment in IBM Cloud and it’s failing. The deployment works fine on my local minikube, but fails in IBM Cloud. Am I required to use the IBM Cloud namespace for deployment or should I be able to pull a Docker image into IBM Cloud from Docker public offerings?

Long version

I’m working through this course on IBM Cloud Kubernetes service (https://courses.cognitiveclass.ai/courses/course-v1:CognitiveClass+CO0401EN+v1/info)

In the prereqs, I need to deploy Redis for a guestbook database.

Those steps are:
1. Clone the repo

$  clone https://github.com/IBM/guestbook.git
  1. cd to /guestbook/v2

  2. Create the Redis controllers and services for both the master and the slave:

    $ kubectl create -f redis-master-deployment.yaml

The problem is that when I run:

$ kubectl create -f redis-master-deployment.yaml

The deployment fails:

$ kubectl get deployments -o wide
NAME           READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS     IMAGES        SELECTOR
redis-master   0/1     1            0           10m   redis-master   redis:3.2.9   app=redis,role=master

$ kubectl get pods -o wide
NAME                           READY   STATUS             RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
redis-master-577bc6fbb-b4trd   0/1     ImagePullBackOff   0          14m   172.30.113.18   10.241.0.11   <none>           <none>

I’m guessing here, but it looks like the IBM Cloud Kubernetes service cannot contact Docker to get the image.

$ kubectl describe pods redis-master-577bc6fbb-b4trd
Name:               redis-master-577bc6fbb-b4trd
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               10.241.0.11/10.241.0.11
Start Time:         Thu, 05 Mar 2020 12:01:45 -0500
Labels:             app=redis
                    pod-template-hash=577bc6fbb
                    role=master
Annotations:        kubernetes.io/psp: ibm-privileged-psp
Status:             Pending
IP:                 172.30.113.18
Controlled By:      ReplicaSet/redis-master-577bc6fbb
Containers:
  redis-master:
    Container ID:   
    Image:          redis:3.2.9
    Image ID:       
    Port:           6379/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rxmp2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-rxmp2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-rxmp2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 600s
                 node.kubernetes.io/unreachable:NoExecute for 600s
Events:
  Type     Reason     Age                   From                  Message
  ----     ------     ----                  ----                  -------
  Normal   Scheduled  <unknown>             default-scheduler     Successfully assigned default/redis-master-577bc6fbb-b4trd to 10.241.0.11
  Warning  Failed     13m (x2 over 14m)     kubelet, 10.241.0.11  Failed to pull image "redis:3.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/redis:3.2.9": failed to resolve reference "docker.io/library/redis:3.2.9": failed to do request: Head https://registry-1.docker.io/v2/library/redis/manifests/3.2.9: dial tcp 34.197.189.129:443: i/o timeout
  Warning  Failed     12m                   kubelet, 10.241.0.11  Failed to pull image "redis:3.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/redis:3.2.9": failed to resolve reference "docker.io/library/redis:3.2.9": failed to do request: Head https://registry-1.docker.io/v2/library/redis/manifests/3.2.9: dial tcp 3.224.75.242:443: i/o timeout
  Normal   Pulling    11m (x4 over 14m)     kubelet, 10.241.0.11  Pulling image "redis:3.2.9"
  Warning  Failed     11m                   kubelet, 10.241.0.11  Failed to pull image "redis:3.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/redis:3.2.9": failed to resolve reference "docker.io/library/redis:3.2.9": failed to do request: Head https://registry-1.docker.io/v2/library/redis/manifests/3.2.9: dial tcp 54.210.105.17:443: i/o timeout
  Normal   BackOff    10m (x6 over 14m)     kubelet, 10.241.0.11  Back-off pulling image "redis:3.2.9"
  Warning  Failed     9m21s (x5 over 14m)   kubelet, 10.241.0.11  Error: ErrImagePull
  Warning  Failed     4m38s (x28 over 14m)  kubelet, 10.241.0.11  Error: ImagePullBackOff

I then went to my minikube to test and ran the same command

$ kubectl create -f ./redis-master-deployment.yaml 
deployment.apps/redis-master created


$ kubectl get deployments -o wide
NAME           READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS     IMAGES        SELECTOR
redis-master   1/1     1            1           14s   redis-master   redis:3.2.9   app=redis,role=master

So I just need a point in the direction of where I’m going wrong here.

2

Answers


  1. Chosen as BEST ANSWER

    Found the issue thanks to @bhpratt

    The subnets that the worker nodes are on do not have public IPs. They do have a proxy resource but that has to be enabled on the subnet itself.

    IBM Cloud Subnet public gateway pic

    That was turned off. Flip the switch and the deployment created successfully.

    $ kubectl get deployments -o wide
    NAME           READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS     IMAGES        SELECTOR
    redis-master   1/1     1            1           4m50s   redis-master   redis:3.2.9   app=redis,role=master
    

  2. Thanks for the additional info. The answer is your worker nodes need public access in order to pull from public docker. Three solutions:

    1. Create worker nodes with public and private IPs
    2. Create a gateway enabled cluster:
      https://cloud.ibm.com/docs/containers?topic=containers-plan_clusters#gateway
    3. Push the docker image to your IBM Cloud container registry,
      update the YAML to pull from the container registry, and deploy the
      containers. Your cluster can pull images from your IBM Cloud registry from the private network.
    4. If this is a cluster in a Virtual Private Cloud (VPC), then create a public gateway for your subnet(s) so the worker nodes can access public endpoints, like Docker. https://cloud.ibm.com/docs/vpc?topic=vpc-public-gateways#pg-getting-started
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search