skip to Main Content

I want to configure native Kubernetes cluster using Terraform script. I tried this Terraform script:

terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.13.1"
    }
    kubectl = {
      source = "gavinbunney/kubectl"
      version = "1.14.0"
    }
    helm = {
      source = "hashicorp/helm"
      version = "2.6.0"
    }
  }
}

provider "kubectl" {
  # run kubectl cluster-info to get expoint and port
  host = "https://192.168.1.139:6443/"
  token = "eyJhbGciOiJSUzI1NiIsImt....."
  insecure = "true"
}

provider "kubernetes" {
  # run kubectl cluster-info to get expoint and port
  host = "https://192.168.1.139:6443/"
  token = "eyJhbGciOiJSUzI1NiIsImt....."
  insecure = "true"
}

resource "kubernetes_namespace" "example" {
  metadata {
    annotations = {
      name = "example-annotation"
    }

    labels = {
      mylabel = "label-value"
    }

    name = "terraform-example-namespace"
  }
}

ref: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs
https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs

I tried to create a user from this tutorial: https://killercoda.com/kimwuestkamp/scenario/k8s1.24-serviceaccount-secret-changes

kubectl create sa cicd

kubectl get sa,secret


cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: cicd
spec:
  serviceAccount: cicd
  containers:
  - image: nginx
    name: cicd
EOF




kubectl exec cicd -- cat /run/secrets/kubernetes.io/serviceaccount/token && echo

kubectl exec cicd cat /run/secrets/kubernetes.io/serviceaccount/token && echo


kubectl create token cicd

kubectl create token cicd --duration=999999h



cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: cicd
  annotations:
    kubernetes.io/service-account.name: "cicd"
EOF



kubectl get sa,secret

kubectl describe secret cicd

kubectl describe sa cicd

kubectl get sa cicd -oyaml

kubectl get sa,secret

When I run the Terraform script I get error:

kubernetes_namespace.example: Creating...
╷
│ Error: namespaces is forbidden: User "system:serviceaccount:default:cicd" cannot create resource "namespaces" in API group "" at the cluster scope
│
│   with kubernetes_namespace.example,
│   on main.tf line 36, in resource "kubernetes_namespace" "example":
│   36: resource "kubernetes_namespace" "example" {

Can you advise what user configuration I’m missing?

Can you advise what is the proper way to implement this script and provision HELM chart into native Kubernetes.

2

Answers


  1. Error: namespaces is forbidden: User "system:serviceaccount:default:cicd" cannot create resource "namespaces" in API group "" at the cluster scope

    The service account cicd in namespace default is lacked of permissions. You can first assign cluster-admin permissions to ensure your pipeline is functioning, then trim the permissions gradually according to your use case. Apply the following spec before your pipeline starts:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: <of your own>
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
    - kind: ServiceAccount
      name: cicd
      namespace: default
    
    Login or Signup to reply.
  2. You need implement RBAC (Role Back Access Control) by attaching ClusterRole and ClusterRoleBinding with your ServiceAccount. SA only contains token for authentication with the k8s api-server. Authorization is enabled via RBAC Here’s an example:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: <cluster-role-name>
    rules:
    - apiGroups: [""]
      resources: ["namespace"]
      verbs: ["create"] # specify other verbs e.g get, list, delete, watch etc.
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: <cluster-role-binding-name>
    subjects:
    - kind: ServiceAccount
      name: <service-account-name>
      namespace: default # specify other namespace
    roleRef:
      kind: ClusterRole
      name: <cluster-role-name>
      apiGroup: rbac.authorization.k8s.io
    

    Check out the official k8s document.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search