Vincenzo
February 3, 2023
214 views
0 votes
2 Answers

I have an AKS cluster with a Node.js server connecting to a Neo4j-standalone instance all deployed with Helm.
I installed an ingress-nginx controller, referenced a default Let's Encrypt certificate and habilitated TPC ports with Terraform as

resource "helm_release" "nginx" {
  name      = "ingress-nginx"
  repository = "ingress-nginx"
  # repository = "https://kubernetes.github.io/ingress-nginx"
  chart     = "ingress-nginx/ingress-nginx"
  namespace = "default"


  set {
    name = "tcp.7687"
    value = "default/cluster:7687"
  }
  set {
    name = "tcp.7474"
    value = "default/cluster:7474"
  }
  set {
    name = "tcp.7473"
    value = "default/cluster:7473"
  }
  set {
    name = "tcp.6362"
    value = "default/cluster-admin:6362"
  }
  set {
    name = "tcp.7687"
    value = "default/cluster-admin:7687"
  }
  set {
    name = "tcp.7474"
    value = "default/cluster-admin:7474"
  }
  set {
    name = "tcp.7473"
    value = "default/cluster-admin:7473"
  }

  set {
    name  = "controller.extraArgs.default-ssl-certificate"
    value = "default/tls-secret"
  }

  set {
    name  = "controller.service.externalTrafficPolicy"
    value = "Local"
  }
  set {
    name = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-internal"
    value = "true"
  }
  set {
    name  = "controller.service.loadBalancerIP"
    value = var.public_ip_address
  }
  set {
    name = "controller.service.annotations.service.beta.kubernetes.io/azure-dns-label-name"
    value = "xxx.westeurope.cloudapp.azure.com"
  }
  set {
    name  = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"
    value = "/healthz"
  }
}

I then have an Ingress with paths pointing to Neo4j services so on https://xxx.westeurope.cloudapp.azure.com/neo4j-tcp-http/browser/ I can get to the browser.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2$3$4
    # nginx.ingress.kubernetes.io/rewrite-target: /
    # certmanager.k8s.io/acme-challenge-type: http01
    nginx.ingress.kubernetes/cluster-issuer: letsencrypt-issuer
    ingress.kubernetes.io/ssl-redirect: "true"
    # kubernetes.io/tls-acme: "true"

spec:
  ingressClassName: nginx
  tls:
  - hosts: 
    - xxxx.westeurope.cloudapp.azure.com
    secretName: tls-secret 
  rules:
    # - host: xxx.westeurope.cloud.app.azure.com #dns from Azure PublicIP


### Node.js server
  - http:
      paths:
      - path: /(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: server-clusterip-service
            port:
              number: 80 

  - http:
      paths:
      - path: /server(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: server-clusterip-service
            port:
              number: 80

##### Neo4j

  - http:
      paths:            
        # 502 bad gateway
        # /any character 502 bad gatway
        - path: /neo4j-tcp-bolt(/|$)(.*)
          pathType: Prefix
          backend:
            service:
              # neo4j chart
              # name: cluster

              # neo4j-standalone chart
              name: neo4j
              port: 
                # name: tcp-bolt
                number: 7687
  - http:
      paths:   
        # /browser/ show browser
        #/any character shows login to xxx.westeurope.cloudapp.azure.com:443 from https, :80 from http
        - path: /neo4j-tcp-http(/|$)(.*) 
          pathType: Prefix
          backend:
            service:
              # neo4j chart
              # name: cluster

              # neo4j-standalone chart
              name: neo4j
              port: 
                # name: tcp-http
                number: 7474
  - http:
      paths:            
        - path: /neo4j-tcp-https(/|$)(.*) 
        # 502 bad gateway
        # /any character 502 bad gatway
          pathType: Prefix
          backend:
            service:
              # neo4j chart
              # name: cluster

              # neo4j-standalone chart
              name: neo4j
              port: 
                # name: tcp-https
                number: 7473

I can get to the Neo4j Browser on the https://xxx.westeurope.cloudapp.azure.com/neo4j-tcp-http/browser/ but using the Connect Url bolt+s//server.bolt it won’t connect to the server with the error ServiceUnavailable: WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver..
Now I’m guessing that is because Neo4j bolt connector is not using the Certificate used by the ingress-nginxcontroller.

vincenzocalia@vincenzos-MacBook-Air helm_charts % kubectl describe secret tls-secret
Name:         tls-secret
Namespace:    default
Labels:       controller.cert-manager.io/fao=true
Annotations:  cert-manager.io/alt-names: xxx.westeurope.cloudapp.azure.com
              cert-manager.io/certificate-name: tls-certificate
              cert-manager.io/common-name: xxx.westeurope.cloudapp.azure.com
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-group: 
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: letsencrypt-issuer
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
tls.crt:  5648 bytes
tls.key:  1679 bytes

I tried to use it overriding the chart values, but then the Neo4j driver from Node.js server won’t connect to the server ..

  ssl:
  # setting per "connector" matching neo4j config
    bolt:
      privateKey:
        secretName: tls-secret # we set up the template to grab `private.key` from this secret
        subPath:  tls.key # we specify the privateKey value name to get from the secret
      publicCertificate:
        secretName: tls-secret # we set up the template to grab `public.crt` from this secret
        subPath:  tls.crt # we specify the publicCertificate value name to get from the secret
      trustedCerts:
        sources: [ ] # a sources array for a projected volume - this allows someone to (relatively) easily mount multiple public certs from multiple secrets for example.
      revokedCerts:
        sources: [ ]  # a sources array for a projected volume
    https:
      privateKey:
        secretName: tls-secret
        subPath: tls.key
      publicCertificate:
        secretName: tls-secret
        subPath: tls.crt
      trustedCerts:
        sources: [ ]
      revokedCerts:
        sources: [ ]

Is there a way to use it or should I setup another certificate just for Neo4j? If so what would it be the dnsNames to set on it?

Is there something else I’m doing wrong?
Thank you very much.

Answers

Chosen as BEST ANSWER

Finally after a few days of going in circles I found what the problems were..

First using a Staging certificate will cause Neo4j bolt connection to fail, as it's not Trusted, with error:

ServiceUnavailable: WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver. Please use your browsers development console to determine the root cause of the failure. Common reasons include the database being unavailable, using the wrong connection URL or temporary network problems. If you have enabled encryption, ensure your browser is configured to trust the certificate Neo4j is configured to use. WebSocket readyState is: 3

found here https://grishagin.com/neo4j/2022/03/29/neo4j-websocket-issue.html

Then I was missing to assign a general listening address to the bolt connector as it's listening by default only to 127.0.0.0:7687 https://neo4j.com/docs/operations-manual/current/configuration/connectors/

To listen for Bolt connections on all network interfaces (0.0.0.0)

so I added server.bolt.listen_address: "0.0.0.0:7687" to Neo4j chart values config.

Next, as I'm connecting the default neo4j ClusterIP service tcp ports to the ingress controller's exposed TCP connections through the Ingress as described here https://neo4j.com/labs/neo4j-helm/1.0.0/externalexposure/ as an alternative to using a LoadBalancer, the Neo4j LoadBalancer services is not needed so the services:neo4j:enabled gets set to "false", in my tests I actually found that if you leave it enabled bolt won't connect despite setting everything correctly..

Other Neo4j missing config where server.bolt.enabled : "true", server.bolt.tls_level: "REQUIRED", dbms.ssl.policy.bolt.client_auth: "NONE" and dbms.ssl.policy.bolt.enabled: "true" the complete list of config options is here https://neo4j.com/docs/operations-manual/current/reference/configuration-settings/

Neo4j chart's values for ssl config were fine.

So now I can use the (renamed for brevity) path /neo4j/browser/ to serve the Neo4j Browser app, and either the /bolt path as the browser Connect URL, or PublicIP's <DSN>:<bolt port>.

You are connected as user neo4j to bolt+s://xxxx.westeurope.cloudapp.azure.com/bolt Connection credentials are stored in your web browser.

Hope this explanation and the code recap below will help others.

Cheers.

ingress controller

resource "helm_release" "nginx" {
  name      = "ingress-nginx"
  namespace = "default"
  repository = "https://kubernetes.github.io/ingress-nginx" 
  chart = "ingress-nginx"
  set {
    name  = "version"
    value = "4.4.2"

  }
### expose tcp connections for neo4j service
  ### bolt url connection port
  set {
    name = "tcp.7687"
    value = "default/neo4j:7687"
  }
  ### http browser app port
  set {
    name = "tcp.7474"
    value = "default/neo4j:7474"
  }


  set {
    name  = "controller.extraArgs.default-ssl-certificate"
    value = "default/tls-secret"
  }

  set {
    name  = "controller.service.externalTrafficPolicy"
    value = "Local"
  }
  set {
    name = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-internal"
    value = "true"
  }
  set {
    name  = "controller.service.loadBalancerIP"
    value = var.public_ip_address
  }
  set {
    name = "controller.service.annotations.service.beta.kubernetes.io/azure-dns-label-name"
    value = "xxx.westeurope.cloudapp.azure.com"
  }
  set {
    name  = "controller.service.annotations.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"
    value = "/healthz"
  }
}

Ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2$3$4
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes/cluster-issuer: letsencrypt-issuer

spec:
  ingressClassName: nginx
  tls:
  - hosts: 
    - xxx.westeurope.cloudapp.azure.com
    secretName: tls-secret 
  rules:


### Node.js server
  - http:
      paths:
      - path: /(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: server-clusterip-service
            port:
              number: 80 
            

  - http:
      paths:
      - path: /server(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: server-clusterip-service
            port:
              number: 80

##### Neo4j

  - http:
      paths:            
        - path: /bolt(/|$)(.*)
          pathType: Prefix
          backend:
            service:
              name: neo4j
              port: 
                # name: tcp-bolt
                number: 7687
  - http:
      paths:   
        - path: /neo4j(/|$)(.*) 
          pathType: Prefix
          backend:
            service:
              name: neo4j
              port: 
                # name: tcp-http
                number: 7474

Values.yaml (Umbrella chart)

neo4j-db: #chart dependency alias
  nameOverride: "neo4j"
  fullnameOverride: 'neo4j'
  neo4j:
   # Name of your cluster
    name: "xxxx" # this will be the label: app: value for the service selector
    password: "xxxxx"
    ##
    passwordFromSecret: ""
    passwordFromSecretLookup: false
    edition: "community"
    acceptLicenseAgreement: "yes"
    offlineMaintenanceModeEnabled: false 
    resources:
      cpu: "1000m"
      memory: "2Gi"

  volumes:
    data:

      mode: 'volumeClaimTemplate'
      volumeClaimTemplate:
        accessModes:
          - ReadWriteOnce
        storageClassName: neo4j-sc-data
        resources:
          requests:
            storage: 4Gi


    backups:
      mode: 'share' # share an existing volume (e.g. the data volume)
      share:
        name: 'logs'

    logs:
      mode: 'volumeClaimTemplate'
      volumeClaimTemplate:
        accessModes:
          - ReadWriteOnce
        storageClassName: neo4j-sc-logs
        resources:
          requests:
            storage: 4Gi


  services:
  # A LoadBalancer Service for external Neo4j driver applications and Neo4j Browser, this will create "cluster-neo4j" svc
    neo4j:
      enabled: false 
  
  config:
     
    server.bolt.enabled : "true"
    server.bolt.tls_level: "REQUIRED"
    server.bolt.listen_address: "0.0.0.0:7687"
    dbms.ssl.policy.bolt.client_auth: "NONE"
    dbms.ssl.policy.bolt.enabled: "true"
    

  
  startupProbe:
    failureThreshold: 1000
    periodSeconds: 50

  ssl:
    bolt:
      privateKey:
        secretName: tls-secret 
        subPath:  tls.key 
      publicCertificate:
        secretName: tls-secret 
        subPath:  tls.crt 
      trustedCerts:
        sources: [ ] 
      revokedCerts:
        sources: [ ]  # a sources array for a projected volume

(Edit)

- MaxAndersson
- February 6, 2023 at 1:28 pm
- 0 votes
0
From what I can gather from your information, the problem seems to be that you’re trying to expose the bolt port behind an ingress. Ingresses are implemented as an L7 (protocol aware) reverse proxy and manage load-balancing etc. The bolt protocol has its load balancing and routing for cluster applications. So you will need to expose the network service directly for every instance of neo4j you are running.

Check out this part of the documentation for more information:
https://neo4j.com/docs/operations-manual/current/kubernetes/accessing-neo4j/#access-outside-k8s

Login or Signup to reply.

Please signup or login to give your own answer.

Click here to cancel reply.

Can't connect externally to Neo4j server in AKS cluster using Neo4j browser – Nginx

Answers

ingress controller

Ingress.yaml

Values.yaml (Umbrella chart)