skip to Main Content

Dear friendly developer,

I am trying to register a Gitlab Kubernetes Agent inside a Minikube with a self hosted Gitlab instance. The Gitlab instance is a dockerized Omnibus installation. It does not have any exposed ports. Instead I chose to use a nginx within the same docker network to proxy_pass requests to Gitlab.

When I deploy the agent to the cluster and the container is running, it logs theses errors:

{"level":"warn","time":"2022-02-26T00:12:59.647Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01FWSNZ31HRVTAAD5J5700BBXH"}
{"level":"error","time":"2022-02-26T00:13:28.271Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01FWSP040J2CRGF5WFHMEX1ACC"}

Visiting http://gitlab.local/api/v4/internal/kubernetes/agent_info results in

{
  "message": "KAS JWT authentication invalid"
}

The agent successfully connects to Gitlab when I expose the gitlab ports directly to localhost (and change the agent’s kubernetes config accordingly). That is why I am quite sure that it has to be a problem with my nginx websocket configuration.

I have triple checked that the token inside the kubernetes secret for the agent matches the base64 registration token generated by Gitlab.

This is an excerpt of my docker-compose file for gitlab:

services:
  gitlab:
    image: gitlab/gitlab-ee:latest
    container_name: gitlab
    restart: always
    hostname: gitlab.local
    networks:
      - ci-cd
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab.local'
        registry_external_url 'http://gitlab.local:5050'
        registry['enable'] = true
        registry['env'] = {
          "REGISTRY_HTTP_RELATIVEURLS" => true
        }
        gitlab_kas['enable'] = true
        gitlab_kas['gitlab_address'] = 'http://gitlab.local'
    volumes:
      - $GITLAB_HOME/etc:/etc/gitlab:rw
      - $GITLAB_HOME/opt:/var/opt/gitlab:rw
      - $GITLAB_HOME/log:/var/log/gitlab:rw
    shm_size: "512m"
    ulimits:
      sigpending: 62793
      nproc: 131072
      nofile: 60000
      core: 0
    sysctls:
      net.core.somaxconn: 1024

The default API path that gitlab uses for the agent websocket connection is:
/-/kubernetes-agent/

This is my nginx configuration:

upstream gitlab_container {
 server gitlab;
}

upstream gitlab_registry_container {
 server gitlab:5050;
}

map $http_upgrade $connection_upgrade {
 default upgrade;
 `` close;
}

server {
 listen 80;
 listen [::]:80;

 server_name gitlab.local;

 location / {
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-NginX-Proxy true;
  proxy_set_header Host $host;
  proxy_pass http://gitlab_container;
  proxy_ssl_session_reuse off;
  proxy_redirect off;
  proxy_cache_bypass $http_upgrade;
 }

 location /-/kubernetes-agent/ {
  proxy_pass http://gitlab;
  proxy_http_version 1.1;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-NginX-Proxy true;
  proxy_set_header Host $host;
  proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
  proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
  proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
  proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;
  proxy_cache_bypass $http_upgrade;
 }
}

server {
 listen 5050;
 listen [::]:5050;

 server_name gitlab.local;

 location / {
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-NginX-Proxy true;
  proxy_set_header Host $host;
  proxy_pass http://gitlab_registry_container;
  proxy_redirect off;
  proxy_ssl_session_reuse off;
  proxy_cache_bypass $http_upgrade;
 }
}

This is the kubernetes configuration for my agent:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab-agent
  namespace: gitlab-kubernetes-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitlab-agent
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        app: gitlab-agent
    spec:
      hostAliases:
        - ip: ${INTERNAL_HOST_IP}
          hostnames:
            - "gitlab.local"
      containers:
        - args:
            - --token-file=/config/token
            - --kas-address
            - ws://gitlab.local/-/kubernetes-agent/
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          image: registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:stable
          livenessProbe:
            httpGet:
              path: /liveness
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 20
          name: agent
          readinessProbe:
            httpGet:
              path: /readiness
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          volumeMounts:
            - mountPath: /config
              name: token-volume
      serviceAccountName: gitlab-agent
      volumes:
        - name: token-volume
          secret:
            secretName: ${GITLAB_AGENT_TOKEN_NAME}

The handshake and the protocol upgrade seems to be working fine, as my nginx log shows

172.19.0.1 - - [26/Feb/2022:00:29:32 +0000] "GET /-/kubernetes-agent/ HTTP/1.1" 101 3450 "-" "gitlab-agent/v14.8.1/86d5bf7" "-" 

I guess that somehow the registration token gets lost when passing through the reverse proxy. Sadly, I cannot find any technical documentation on how the authentication works in detail.

Any clue as to what I am missing is highly appreciated!

3

Answers


  1. Chosen as BEST ANSWER

    I don't know exactly what I did different then the previous 10 times, but suddenly the agent connected successfully with the configuration shown above. I suppose it was any of these lines inside my nginx configuration for gitlab:

      proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
      proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
      proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
      proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
    

    Those I added last. But I cannot guarantee that was the breaking change. Good luck to everyone with similar issues reading this post.


  2. Another option:

    See GitLab 14.10 (April 2022)

    The agent server for Kubernetes enabled by default in the Helm chart

    The first step for using the agent for Kubernetes in self-managed instances is
    to enable the agent server, a backend service for the agent for Kubernetes.

    In GitLab 14.8 we enabled the agent server for Omnibus based installations.

    The feature has matured in the past few months, so we are now making the agent server enabled by default in the GitLab Helm chart as well to simplify setup for GitLab administrators.
    Besides being enabled by default, the agent server accepts various configuration options to customize it according to your needs.

    See Documentation and Issue.

    That might be easier than setting it up through NGiNX.


    This is confirmed with See GitLab 15.1 (June 2022)

    Agent server for Kubernetes enabled by default in the Helm chart

    The first step for using the agent for Kubernetes in self-managed instances is to enable the agent server, a backend service for the agent for Kubernetes.
    In GitLab 14.8, we enabled the agent server for Omnibus based installations.

    The feature has matured in the past few months, so we are now making the agent server enabled by default in the GitLab Helm chart as well to simplify setup for GitLab administrators.
    Besides being enabled by default, the agent server accepts various configuration options to customize it according to your needs.

    See Documentation and Issue.


    See GitLab 15.1 (June 2022)

    GitLab agent for Kubernetes supports proxied connections

    Many users require a proxy to connect Kubernetes clusters to GitLab. Previously, the default installation method for the GitLab agent for Kubernetes did not support proxied connections.

    Now, you can use the HTTP_PROXY environment variable in the agentk Helm package to support proxied connections.

    See Documentation and Issue.

    Login or Signup to reply.
  3. I had exactly same error although probably had a different reason of it. My new gitlab server, by my error, was using an alternate DNS which had an entry for gitlab’s external URL poining to my OLD gitlab server.
    Everything else in the network (kubernetes with agent I was trying to install) were using correct DNS and effectively were pointing to the correct new gitlab server, but the gitlab server itself wasn’t. I’ve discovered it by looking at the tcpdump and noticing the traffic on 443 between my old and new gitlab servers (so advice is to trace the https traffic on your gitlab server). Took me 2 days 🙁 These messages should be a bit more elaborative (if they could give IPs and ports for this connection I would figure out my error in 2 minutes).
    Hope this helps next people with similar problem to pin point the issue.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search