GCP Kubernetes not using service account for pulling docker images

I’m using the latest version of google-kubernetes (1.22.8-gke.202) in a Kubernetes managed cluster. I also have a custom service account that has access to the "Artifact Registry Reader" scope that should grant it permission to pull private images from the repository – calling this custom-service-account.

I’ve validated that the nodes themselves have the custom-service-account service account linked to them within Compute Engine. Kubernetes is setup with a service account that is linked to the IAM service account with the same name through workload identity. However, when I try to spawn a pod that pulls from my private repo it fails indefinitely.

Events:
  Type     Reason             Age                   From                Message
  ----     ------             ----                  ----                -------
  Warning  FailedScheduling   21m (x3 over 24m)     default-scheduler   0/2 nodes are available: 2 node(s) were unschedulable.
  Warning  FailedScheduling   19m                   default-scheduler   no nodes available to schedule pods
  Normal   NotTriggerScaleUp  18m (x25 over 24m)    cluster-autoscaler  pod didn't trigger scale-up: 1 node(s) had taint {reserved-pool: true}, that the pod didn't tolerate
  Normal   Scheduled          18m                   default-scheduler   Successfully assigned default/test-service-a-deployment-5757fc5797-b54gx to gke-personal-XXXX--personal-XXXX--ac9a05b6-16sb
  Normal   Pulling            17m (x4 over 18m)     kubelet             Pulling image "us-central1-docker.pkg.dev/personal-XXXX/my-test-repo/my-test-repo-business-logic:latest"
  Warning  Failed             17m (x4 over 18m)     kubelet             Failed to pull image "us-central1-docker.pkg.dev/personal-XXXX/my-test-repo/my-test-repo-business-logic:latest": rpc error: code = Unknown desc = failed to pull and unpack image "us-central1-docker.pkg.dev/personal-XXXX/my-test-repo/my-test-repo-business-logic:latest": failed to resolve reference "us-central1-docker.pkg.dev/personal-XXXX/my-test-repo/my-test-repo-business-logic:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
  Warning  Failed             17m (x4 over 18m)     kubelet             Error: ErrImagePull
  Warning  Failed             16m (x6 over 18m)     kubelet             Error: ImagePullBackOff
  Normal   BackOff            3m27s (x65 over 18m)  kubelet             Back-off pulling image "us-central1-docker.pkg.dev/personal-XXXX/my-test-repo/my-test-repo-business-logic:latest"

I’ve also ssh’ed into the nodes themselves and at least by default with a regular docker pull or crictl pull see this same error.

So, the specific questions I have:

• How is GCP injecting the service account credentials into Kubernetes/Docker worker that tries to launch the images? Is it expected that the regular docker command doesn’t seem to have these credentials?
• Do I need to manually bootstrap some additional authentication for Kubernetes aside from just inheriting the service account on the pods?

EDIT: Result of here

> gcloud container clusters describe personal-XXXX-gke --zone us-central1-a --format="value(workloadIdentityConfig.workloadPool)"
personal-XXXX.svc.id.goog

> gcloud container node-pools describe personal-XXXX-gke-node-pool --cluster personal-XXXX-gke --format="value(config.workloadMetadataConfig.mode)" --zone us-central1-a
GKE_METADATA

> kubectl describe serviceaccount --namespace default be-service-account
Name:                be-service-account
Namespace:           default
Labels:              <none>
Annotations:         iam.gke.io/gcp-service-account: [email protected]
Image pull secrets:  <none>
Mountable secrets:   be-service-account-token-jmss9
Tokens:              be-service-account-token-jmss9
Events:              <none>

> gcloud iam service-accounts get-iam-policy [email protected]
bindings:
- members:
  - serviceAccount:personal-XXXX.svc.id.goog[default/be-service-account]
  role: roles/iam.workloadIdentityUser
etag: BwXjqJ9DC6A=
version: 1