Dear friendly developer,
I am trying to register a Gitlab Kubernetes Agent inside a Minikube with a self hosted Gitlab instance. The Gitlab instance is a dockerized Omnibus installation. It does not have any exposed ports. Instead I chose to use a nginx within the same docker network to proxy_pass requests to Gitlab.
When I deploy the agent to the cluster and the container is running, it logs theses errors:
{"level":"warn","time":"2022-02-26T00:12:59.647Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01FWSNZ31HRVTAAD5J5700BBXH"}
{"level":"error","time":"2022-02-26T00:13:28.271Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01FWSP040J2CRGF5WFHMEX1ACC"}
Visiting http://gitlab.local/api/v4/internal/kubernetes/agent_info
results in
{
"message": "KAS JWT authentication invalid"
}
The agent successfully connects to Gitlab when I expose the gitlab ports directly to localhost (and change the agent’s kubernetes config accordingly). That is why I am quite sure that it has to be a problem with my nginx websocket configuration.
I have triple checked that the token inside the kubernetes secret for the agent matches the base64 registration token generated by Gitlab.
This is an excerpt of my docker-compose file for gitlab:
services:
gitlab:
image: gitlab/gitlab-ee:latest
container_name: gitlab
restart: always
hostname: gitlab.local
networks:
- ci-cd
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://gitlab.local'
registry_external_url 'http://gitlab.local:5050'
registry['enable'] = true
registry['env'] = {
"REGISTRY_HTTP_RELATIVEURLS" => true
}
gitlab_kas['enable'] = true
gitlab_kas['gitlab_address'] = 'http://gitlab.local'
volumes:
- $GITLAB_HOME/etc:/etc/gitlab:rw
- $GITLAB_HOME/opt:/var/opt/gitlab:rw
- $GITLAB_HOME/log:/var/log/gitlab:rw
shm_size: "512m"
ulimits:
sigpending: 62793
nproc: 131072
nofile: 60000
core: 0
sysctls:
net.core.somaxconn: 1024
The default API path that gitlab uses for the agent websocket connection is:
/-/kubernetes-agent/
This is my nginx configuration:
upstream gitlab_container {
server gitlab;
}
upstream gitlab_registry_container {
server gitlab:5050;
}
map $http_upgrade $connection_upgrade {
default upgrade;
`` close;
}
server {
listen 80;
listen [::]:80;
server_name gitlab.local;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Host $host;
proxy_pass http://gitlab_container;
proxy_ssl_session_reuse off;
proxy_redirect off;
proxy_cache_bypass $http_upgrade;
}
location /-/kubernetes-agent/ {
proxy_pass http://gitlab;
proxy_http_version 1.1;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Host $host;
proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_cache_bypass $http_upgrade;
}
}
server {
listen 5050;
listen [::]:5050;
server_name gitlab.local;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Host $host;
proxy_pass http://gitlab_registry_container;
proxy_redirect off;
proxy_ssl_session_reuse off;
proxy_cache_bypass $http_upgrade;
}
}
This is the kubernetes configuration for my agent:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: gitlab-agent
namespace: gitlab-kubernetes-agent
spec:
replicas: 1
selector:
matchLabels:
app: gitlab-agent
strategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "8080"
prometheus.io/scrape: "true"
labels:
app: gitlab-agent
spec:
hostAliases:
- ip: ${INTERNAL_HOST_IP}
hostnames:
- "gitlab.local"
containers:
- args:
- --token-file=/config/token
- --kas-address
- ws://gitlab.local/-/kubernetes-agent/
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
image: registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:stable
livenessProbe:
httpGet:
path: /liveness
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
name: agent
readinessProbe:
httpGet:
path: /readiness
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
volumeMounts:
- mountPath: /config
name: token-volume
serviceAccountName: gitlab-agent
volumes:
- name: token-volume
secret:
secretName: ${GITLAB_AGENT_TOKEN_NAME}
The handshake and the protocol upgrade seems to be working fine, as my nginx log shows
172.19.0.1 - - [26/Feb/2022:00:29:32 +0000] "GET /-/kubernetes-agent/ HTTP/1.1" 101 3450 "-" "gitlab-agent/v14.8.1/86d5bf7" "-"
I guess that somehow the registration token gets lost when passing through the reverse proxy. Sadly, I cannot find any technical documentation on how the authentication works in detail.
Any clue as to what I am missing is highly appreciated!
3
Answers
I don't know exactly what I did different then the previous 10 times, but suddenly the agent connected successfully with the configuration shown above. I suppose it was any of these lines inside my nginx configuration for gitlab:
Those I added last. But I cannot guarantee that was the breaking change. Good luck to everyone with similar issues reading this post.
Another option:
See GitLab 14.10 (April 2022)
That might be easier than setting it up through NGiNX.
This is confirmed with See GitLab 15.1 (June 2022)
See GitLab 15.1 (June 2022)
I had exactly same error although probably had a different reason of it. My new gitlab server, by my error, was using an alternate DNS which had an entry for gitlab’s external URL poining to my OLD gitlab server.
Everything else in the network (kubernetes with agent I was trying to install) were using correct DNS and effectively were pointing to the correct new gitlab server, but the gitlab server itself wasn’t. I’ve discovered it by looking at the tcpdump and noticing the traffic on 443 between my old and new gitlab servers (so advice is to trace the https traffic on your gitlab server). Took me 2 days 🙁 These messages should be a bit more elaborative (if they could give IPs and ports for this connection I would figure out my error in 2 minutes).
Hope this helps next people with similar problem to pin point the issue.