skip to Main Content

im currently doing an end of studies internship of my bachelor degree and my project is doing a CI/CD pipeline so m kinda new to devops so i wish someone can give me some of his precious time to help me out .
after setting up 3 ubuntu server instances on vmware using NAT network
heres a list of the commands ive executed to setup a kubernetes cluster with 1 master and 2 workers using kubeadm

apt install docker.io -y
sudo swapoff -a
nano /etc/fstab
#ive commented the swap line here
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable --now kubelet
#then on my master node ive executed the kubeadm init and did the export config thing and joined the worker nodes using the cmd the master gave me 
sudo kubeadm init

the first 10 mins everything works fine i joined the nodes installed calico etc then whenever i try to execute "kubectl get nodes" i get the error
The connection to the server 192.168.149.141:6443 was refused - did you specify the right host or port?

ive tried restarting docker and kubelet service it solves the issue for another 10 mins before it crashes again
also another thing ive noticed some pods keeps restartign as u can see CrashLoopBackOff and its not performance issue because when i installed the metrics the Cpu and ram usages were bellow 40% on all nodes
kubectl get po -n kube-system

heres the logs of kubelet :
https://imgur.com/a/sWH3oUb

my kube-apiserver.yaml

  GNU nano 6.2                                                                                   kube-apiserver.yaml                                                                                             apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.149.141:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.149.141
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.k8s.io/kube-apiserver:v1.29.3
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.149.141
      httpGet:
        host: 192.168.149.141
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.149.141
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.149.141
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}

Im up to provide more info

2

Answers


  1. Chosen as BEST ANSWER

    After doing more researches and inspectations found out that kubernetes removed support for docker as container runtime so i reintialized my cluster using cri-o and everything is working fine followed this blog for installation :

    https://kubernetes.io/blog/2023/10/10/cri-o-community-package-infrastructure/


  2. You can try following steps to narrow down which components are causing the issue:

    1. Login into one of the Node(192.168.149.141), try connect to the API-Server from localhost:6443

      This should help you verify if the API Server is not working properly, or its network related.

    2. Open a nc -v -l 50000 process on Node(192.168.149.141), and connect to the Node via nc -v 192.168.149.141 50000, and try to send a message to see if the remote Node receives them.

      If its a network related issue, this will help you determine if its a NAT(VMWare) related or Router.

    3. You may wanna tail at the Api-server’s log while you are trying to connect, see if it can provide any useful information.

    If you are out of patience, try use Rancher(or RKE) to deploy the cluster instead of kubeadmin, its much more friendly to beginners.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search