skip to Main Content

Summary:
I want to make NGINX (not NGINX Plus) re-resolve the IP address from the DNS name by using a variable in proxy_pass (as suggested in this official Nginx article in the section "Setting the Domain Name in a Variable"). But when I do that it won’t set/forward the correct Content-Type header, but always use text/html, for .css, .js etc. files.

My setup:
I have a frontend and a backend service running in separate Docker containers (to be deployed to OpenShift in production). There’s also a third container that runs nginx:latest (v1.19.9 as of today) and acts as a reverse proxy, forwarding calls made to /my-app to the frontend and /my-app/api to the backend container. The NGINX reverse proxy has set them up as upstream servers using their DNS names.
All three containers run within the same custom Docker network, so resolving in itself works fine – but only, when NGINX is (re-)started.

The problem:
When the frontend or backend container gets restarted it may get a new IP address. Since NGINX caches IP addresses I’m getting a 502 when I make calls to them. I want NGINX to re-resolve the IP address more frequently, like NGINX Plus does. This is how the problem with the Content-Type came up, when I tried to have NGINX re-resolve DNS names.

The configuration:
Here’s my NGINX configuration (simplified to the relevant stuff only):

  index    index.html index.htm;

  upstream upstream_frontend {
    server frontend:8080;
  }

  upstream upstream_backend {
    server backend:8000;
  }

  server {
    listen       8080;
    root         /usr/share/nginx/html;

    try_files    $uri$args $uri$args/ $uri $uri/ /index.html =404;
    
    rewrite ^/my-app$ $scheme://$http_host/my-app/ permanent;

    location /my-app {
      resolver 127.0.0.11 ipv6=off valid=1s;

      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto  $scheme;
      proxy_set_header X-NginX-Proxy true;
      proxy_ssl_session_reuse off;
      proxy_set_header Host $http_host;
      proxy_redirect off;

      proxy_http_version 1.1;
      proxy_pass_request_headers on;
      proxy_pass_header Content-Type;

#     Attempt #1, NOT working: using a variable and an upstream server setup
      set $frontend_var "http://upstream_frontend/";
      proxy_pass http://$frontend_var;

#     Attempt #2, NOT working: using a variable and directly using the container name
#      set $frontend_var "http://frontend:8080/";
#      proxy_pass $frontend_var;

#     Attempt #3, working fine:  using NO variable and an upstream server setup, , but no DNS re-resolving happening :-(
#      proxy_pass http://upstream_frontend/;

#     Attempt #4, working fine:  using NO variable and directly using the container name, but no DNS re-resolving happening :-(
#      proxy_pass http://frontend:8080/;
    }

    location /my-app/api {
      resolver 127.0.0.11 ipv6=off valid=1s;

      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto  $scheme;
      proxy_set_header X-NginX-Proxy true;
      proxy_ssl_session_reuse off;
      proxy_set_header Host $http_host;
      proxy_redirect off;

#     Attempt #1, NOT working: using a variable and an upstream server setup
      set $backend_var "http://upstream_backend/api";
      proxy_pass http://$backend_var;

#     Attempt #2, NOT working:  using a variable and directly using the container name
#      set $backend_var "http://backend:8000/api";
#      proxy_pass http://$backend_var;

#     Attempt #3, working fine:  using NO variable and an upstream server setup, , but no DNS re-resolving happening :-(
#      proxy_pass http://upstream_backend/api;

#     Attempt #4, working fine:  using NO variable and directly using the container name, but no DNS re-resolving happening :-(
#      proxy_pass http://backend:8000/api;
    }
  }

(Please note the location /my-app {...} and the / at the end of the proxy_pass, as the frontend is served from /my-app, whereas the frontend container itself, also served from NGINX, is operating directly on ...:8080/, without the /my-app context path. And we have the same setup for the backend container that is listening on :8000, but when calls are forwarded to it we’re removing the /my-app from them so they reach the container directly at ...:8000/api.)

NGINX is running on port 9000 and when I open http://localhost:9000/my-app using the above Attempt #1 or Attempt #2 the .css, .js, image files etc. are all served with Content-Type: text/html, which prevents the browser from properly rendering the page, and shows messages like this in the debugger pane:

The stylesheet http://localhost:9000/my-app/static/css/main.df1d2133.chunk.css  was not loaded because its MIME type, ”text/html“ is not ”text/css"

At first I thought proxy_pass_header Content-Type; would fix this, but that doesn’t help either.

And then I also learned from this NGINX bug ticket that "When using variables in proxy_pass, if URI is specified, it is passed to the server as is, replacing the original request URI.". This seems to be a possible cause for the Content-Type issue I have here.

Also, as you can see in the configuration I have set the Docker DNS IP: resolver 127.0.0.11 ipv6=off valid=1s;. I have also tried setting it outside the location block, that also didn’t help.

Question:
So how do I have DNS re-resolving in NGINX (Open-Source) with the proper Content-Type still set? I don’t have to use upstream servers, so I can get rid of them, should that be part of a possible fix.

P.S.:
I can’t, as suggested in other StackOverflow comments, add separate location blocks to "retroactively" fix the Content-Type depending on folder, file name etc., because this is an evolving project and I worry it would require me to keep adding such hotfixes to the NGINX configuration on a regular basis.

EDIT:
The configuration I posted is included by NGINX’s Docker container’s configuration file which is this one, in case you were wondering about a few "missing" settings:

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    # This line includes my configuration above
    include /etc/nginx/conf.d/*.conf;
}

2

Answers


  1. Chosen as BEST ANSWER

    So here's my final working setup. Thanks again, @richard-smith!

    Please note that...:

    • I'm not using the upstream block as I'm deploying this to OpenShift. There's the concept of "Services" that fronts Pods (~= servers or containers), which is basically what upstream does in Nginx,
    • I've re-ordered the location blocks so the more specific one, /my-app/api, gets hit first, as otherwise things will fail,
    • I had to add the $isargs$args part as otherwise the URLs are not constructed properly and will fail, specifically for calls to my backend,
    • and for OpenShift I had to add .<PROJECT>.svc.cluster.local on the proxy_pass, as otherwise the resolver will not be able to resolve the service.

    This is the relevant setup, for the remaining (default) Nginx configuration see the configuration I posted in the EDIT of my initial question above.

    index    index.html index.htm;
    
    server {
      listen       8080;
      root         /usr/share/nginx/html;
    
      try_files    $uri$args $uri$args/ $uri $uri/ /index.html =404;
        
      rewrite ^/my-app$ $scheme://$http_host/my-app/ permanent;
    
      # Backend
      location ~ ^/my-app/api/?(.*)$ {
        # Use this for Docker...
        # resolver 127.0.0.11 ipv6=off valid=1s;
        # ... or use this for OpenShift
        # resolver dns-default.openshift-dns.svc.cluster.local ipv6=on valid=1s;
    
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_set_header X-NginX-Proxy true;
        proxy_ssl_session_reuse off;
        proxy_set_header Host $http_host;
        proxy_redirect off;
    
        proxy_http_version 1.1;
        proxy_pass_request_headers on;
        proxy_pass_header Content-Type;
    
        proxy_pass http://backend.<PROJECT>.svc.cluster.local:8000/api/$1$is_args$args;
      }
    
      # Frontend
      location ~ ^/my-app/?(.*)$ {
        # Use this for Docker...
        # resolver 127.0.0.11 ipv6=off valid=1s;
        # ... or use this for OpenShift
        # resolver dns-default.openshift-dns.svc.cluster.local ipv6=on valid=1s;
    
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_set_header X-NginX-Proxy true;
        proxy_ssl_session_reuse off;
        proxy_set_header Host $http_host;
        proxy_redirect off;
    
        proxy_http_version 1.1;
        proxy_pass_request_headers on;
        proxy_pass_header Content-Type;
    
        proxy_pass http://frontend.<PROJECT>.svc.cluster.local:8080/$1$is_args$args;
      }
    }
    

  2. This isn’t a problem with Content-Type. You are using proxy_pass incorrectly. As you have already noted:

    When using variables in proxy_pass, if URI is specified, it is passed to the server as is, replacing the original request URI.

    You currently have the following working configuration, but you want to replace it with a variable to force DNS resolution:

    location /my-app {
        proxy_pass http://frontend:8080/;
    }
    

    The simplest solution is to use rewrite...break to make the changes to the URI, and remove the trailing / from the variable.

    For example:

    location /my-app {
        rewrite ^/my-app/?(.*)$ /$1 break;
        set $frontend_var "frontend:8080";
        proxy_pass http://$frontend_var;
    }
    

    Alternatively, capture the remainder of the original request using a regular expression location.

    For example:

    location ~ ^/my-app/?(.*)$ {
        proxy_pass http://frontend:8080/$1;
    }
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search