Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Ubuntu – SRV Record Integration with Consul and Nginx for Dynamic Service Discovery in Nomad

Hi I have one small confusion why SRV records are not working with Consul and Nginx.

My goal is to use Consul DNS to route traffic to services deployed via Nomad, where dynamic port allocation is used.

I have 2 ec2 instance where Nomad, Consul, Nginx are installed for hosting services.

Specifically, I want Nginx to automatically resolve the correct service IP and port from Consul’s SRV records,
allowing me to route traffic to multiple instances of the zapper service.

I have two instances of zapper service.

ubuntu@master:/etc/nginx/sites-available$ consul catalog services -tags

consul
dns                            primary
nomad                          http,rpc,serf
nomad-client                   http
python-http-server             http,python-http-server
zapper-apps-lovestaco-com      zapper.apps.lovestaco.com

I have enabled DNS forwarding in consul as per the documentation.

When I dig with service, it gives me the IP of service.

ubuntu@master:~$ dig @localhost -p 8600 zapper-apps-lovestaco-com.service.consul

; <<>> DiG 9.18.28-0ubuntu0.20.04.1-Ubuntu <<>> @localhost -p 8600 zapper-apps-lovestaco-com.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40800
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;zapper-apps-lovestaco-com.service.consul. IN A

;; ANSWER SECTION:
zapper-apps-lovestaco-com.service.consul. 0 IN A    132.71.22.71

;; Query time: 4 msec
;; SERVER: 127.0.0.1#8600(localhost) (UDP)
;; WHEN: Tue Sep 10 19:15:27 UTC 2024
;; MSG SIZE  rcvd: 82

When I try to dig with SRV, it gives me the IP & Dynamically allocated Ports of service.

ubuntu@master:~$ dig @localhost -p 8600 zapper-apps-lovestaco-com.service.consul SRV

; <<>> DiG 9.18.28-0ubuntu0.20.04.1-Ubuntu <<>> @localhost -p 8600 zapper-apps-lovestaco-com.service.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43024
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 7
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;zapper-apps-lovestaco-com.service.consul. IN SRV

;; ANSWER SECTION:
zapper-apps-lovestaco-com.service.consul. 0 IN SRV  1 1  ac1f1647.addr.dc1.consul.
zapper-apps-lovestaco-com.service.consul. 0 IN SRV  1 1 25268 ac1f1647.addr.dc1.consul.

;; ADDITIONAL SECTION:
ac1f1647.addr.dc1.consul. 0 IN  A   132.71.22.71
consul-master-server.node.dc1.consul. 0 IN TXT  "consul-version=1.18.0"
consul-master-server.node.dc1.consul. 0 IN TXT  "consul-network-segment="
ac1f1647.addr.dc1.consul. 0 IN  A   132.71.22.71
consul-master-server.node.dc1.consul. 0 IN TXT  "consul-version=1.18.0"
consul-master-server.node.dc1.consul. 0 IN TXT  "consul-network-segment="

;; Query time: 8 msec
;; SERVER: 127.0.0.1#8600(localhost) (UDP)
;; WHEN: Tue Sep 10 19:15:31 UTC 2024
;; MSG SIZE  rcvd: 352

When I curl to the service it points me to the Nginx welcome page.

ubuntu@master:~$ curl zapper-apps-hexmos-com.service.consul

<!DOCTYPE html>
<html>
  <title>Welcome to nginx!</title>
  ...
</html>

When I curl with the port from SRV record, it reaches the service.

ubuntu@master:~$ curl zapper-apps-hexmos-com.service.consul:24717

Accessing port 24717, version 79

Health works aswell.
ubuntu@master:~$ curl zapper-apps-hexmos-com.service.consul:24717/health

{"status": "healthy"}

Now coming to the main part, when I try to access the service via Nginx, it gives me the welcome page.
Meaning Nginx is not able to route the traffic to the service to the port.

ubuntu@master:/etc/nginx/sites-available$ cat zapper.apps.lovestaco.com

resolver 127.0.0.1:8600 valid=10s;

upstream zapper_service {
    zone upstream_dynamic 64k;
    server zapper-apps-lovestaco-com.service.consul resolve;
}

server {
    listen 80;
    server_name zapper.apps.lovestaco.com;

    location / {
        proxy_pass http://zapper_service;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

ubuntu@master:/etc/nginx/sites-available$ s /usr/sbin/nginx -V

nginx version: nginx/1.19.10
built by gcc 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)
built with OpenSSL 1.1.1f  31 Mar 2020
TLS SNI support enabled
configure arguments:
    --prefix=/etc/nginx 
    --sbin-path=/usr/sbin/nginx 
    --conf-path=/etc/nginx/nginx.conf 
    --pid-path=/var/run/nginx.pid 
    --lock-path=/var/run/nginx.lock 
    --error-log-path=/var/log/nginx/error.log 
    --http-log-path=/var/log/nginx/access.log 
    --with-http_ssl_module 
    --with-http_stub_status_module 
    --with-http_realip_module 
    --with-http_auth_request_module 
    --with-http_v2_module 
    --with-http_dav_module 
    --with-http_slice_module 
    --with-http_addition_module 
    --with-http_gunzip_module 
    --with-http_gzip_static_module 
    --with-http_sub_module 
    --with-mail_ssl_module 
    --with-stream_ssl_module 
    --add-module=/usr/local/src/nginx-1.19.10/nginx-upstream-dynamic-servers

Recently added nginx-upstream-dynamic-servers thinking it would fix the SRV problem, didn’t work.

Resources I found:

  1. nginx-upsync-module
  2. ngx_http_consul_backend_module

I do not have nginx plus, hence I am going this way.

Please let me know what would be the best way to deal with this.

Tags:

2

Answers


  1. 0

    the best way to deal with this.

    The best way it generate nginx configuration from nomad template like described in the documentation https://developer.hashicorp.com/nomad/tutorials/load-balancing/load-balancing-nginx . Everything is explained there.

    To summarize, create a nginx job with the following template:

          template {
        data = <<EOF
upstream zapper_service {
{{ range service "zapper_service" }}
  server {{ .Address }}:{{ .Port }};
{{ else }}server 127.0.0.1:65535; # force a 502
{{ end }}
}

server {
   listen 8080;

    location / {
        proxy_pass http://zapper_service;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
EOF
    Login or Signup to reply.
  2. 0

    I know OP was asking for Nginx but maybe someone is stumbling across this.
    Traefik has native support for Consul and Nomad Service Discovery.

    I was also using Nginx at the beginning but then switched to Traefik because of its native integration.

    https://doc.traefik.io/traefik/providers/consul-catalog/

    https://doc.traefik.io/traefik/providers/nomad/

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search