Case
I am following tutorial: Deploy a Spring Boot Java app to Kubernetes on Google Kubernetes Engine . I came to the step when I want to deploy my application (docker image of it) to kubernetes. But after I create the deployment the pods are not starting because there is problem with pulling the image (Error: ErrImagePull).
I create the deployment like that:
kubectl create deployment questy-java --image=us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1
After running the command:
kubectl describe pod questy-java-54dbd6ccd4-5cb94
I am getting event information:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 36s default-scheduler Successfully assigned default/questy-java-54dbd6ccd4-tmdkl to gke-questy-java-cluster-default-pool-ca7ad417-lxkw
Normal Pulling 23s (x2 over 35s) kubelet Pulling image "us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1"
Warning Failed 23s (x2 over 35s) kubelet Failed to pull image "us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1": rpc error: code = Unknown desc = failed to pull and unpack image "us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1": failed to resolve reference "us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden
Warning Failed 23s (x2 over 35s) kubelet Error: ErrImagePull
Normal BackOff 8s (x2 over 35s) kubelet Back-off pulling image "us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1"
Warning Failed 8s (x2 over 35s) kubelet Error: ImagePullBackOff
The root couse seems to be mentioned in here:
failed to authorize: failed to fetch oauth token: unexpected status:
403 Forbidden
Anyhow the tutorial did not mention any additional steps to mantain authentication or authorisation.
What I have tried:
- I have checked the VPC networks dashboard to see if Private Google Access is enabled and it looks fine:
- I have followed instructions to install gke-gcloud-auth-plugin from: https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke. Recreating the deployment after this change did not help.
- Created new repository in the Artifact Registry and tried to deploy my image there
- build an entirely new Docker container and tried to deploy it in new repository
- Tried to deploy questy-java image directly from Google Cloud Console:
- Run the command:
gcloud auth configure-docker us-central1-docker.pkg.dev
- Checked the path throughly. I have copied and paste the path to the image directly from Console:
I have also tried to run the commands:
docker pull
us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1
It was successfull.
Question
What could be the reason of this error? What is the most elegant way to fix the issue?
Edit
I have checked the path to the image directly from Console and it is
us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1
2
Answers
I have followed the tips from @Hemanth Kumar and finaly what solved my problem was adding a Artifact Registry Reader role for specific repository to the service account user which runs in GKE.
I did it via Google Cloud Console:
The Error: ErrImagePull : failed to fetch oauth token: unexpected status: 403 Forbidden is due to authorization issue where the Artifact registry is not having the necessary permissions to pull the image.
Refer to this Troubleshooting error 4xx issues Authentication and authorization errors when connecting to GKE clusters. Set the environment variables to print the access token and Verify that your access token is valid by following the steps mentioned in the Troubleshooting error 4xx issues.
Also as per this Troubleshoot doc you can check the following:
correct. The path must include the registry hostname, Google Cloud
project ID, repository, and image. For example:
us-east1-docker.pkg.dev/my-project/my-rep
permissions to read from the repository. you must grant the Artifact
Registry Reader role to the runtime service account then only
this will help to pull the image.
Grant permissions to the account that interacts with the
repository.
that the image used by a container cannot be loaded from the image
registry.