Question 219822 in Nginx Question posted in
The official documentation can be found here.

Nginx metrics not appearing on prometheus

I have a single node kubernetes cluster deployed by microk8s.
I’m using the default Nginx ingress controller and the prometheus addon.
I’m trying to collect the nginx metrics on prometheus so that I monitor it later on grafana.
The problem is that I’m not getting the metrics on Prometheus after creating the serviceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-prom-stack-nginx-ingress-microk8s
  namespace: observability
  annotations:
    meta.helm.sh/release-name: kube-prom-stack
    meta.helm.sh/release-namespace: observability
  labels:
    app.kubernetes.io/component: metrics
    app.kubernetes.io/instance: kube-prom-stack
    app.kubernetes.io/name: nginx-ingress-microk8s
    app.kubernetes.io/part-of: nginx-ingress-microk8s
    release: kube-prom-stack
spec:
  selector:
    matchLabels:
      microk8s-application: nginx-ingress-microk8s
  endpoints:
    - honorLabels: true
      enableHttp2: true
      port: health
      path: /metrics
      interval: 5s
      scheme: http
  namespaceSelector:
    matchNames:
      - ingress
      - observability
      - kube-system
      - default

Prometheus seems to detect the scrape pool but can’t find any endpoint associated with it.

prometheus targets

Nginx seems to be exporting the metrics correctly.
Nginx metrics

The serviceMonitor seems to have the following prometheus job:

- job_name: serviceMonitor/observability/kube-prom-stack-nginx-ingress-microk8s/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_microk8s_application, __meta_kubernetes_service_labelpresent_microk8s_application]
    separator: ;
    regex: (nginx-ingress-microk8s);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: health
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: health
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - ingress
      - observability
      - kube-system
      - default
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Finally I found out that when deploying the Nginx ingress controller, microk8s doesn't create a service associated to it. I had to do it manually so that it could be discovered by the prometheus ServiceMonitor. Here is the created service:

    apiVersion: v1
kind: Service
metadata:
  name: kube-prom-stack-nginx-ingress-microk8s
  namespace: ingress
  labels:
    microk8s-application: nginx-ingress-microk8s
    release: kube-prom-stack
spec:
  ports:
    - port: 10254
      targetPort: 10254
      name: prometheus
      protocol: TCP
  selector:
    microk8s-application: nginx-ingress-microk8s

    enter image description here


  2. 0

    As to your ServiceMonitor‘s definition, check the following 3 parts for your Nginx ingress controller service(endpoints):

    1. Is it in the namespaces listed below?
      - ingress
  - observability
  - kube-system
  - default
    1. Is it labelld with the following label?
      microk8s-application: nginx-ingress-microk8s
    1. Does the following port name exist in its yaml definition?
      port: health
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search