Nginx Question posted in
The official documentation can be found here.

Nginx – .NET 8 api with Caddy reverse proxy not working for SSL

I’m trying to set up SSL for a domain that is pointing to a simple api. I’m running .net 8 and have a Caddy server set up as a reverse proxy, all set up using docker compose. I initially had nginx running using certbot/letsencrypt but couldn’t get that working so switched to Caddy because I heard it works out of the box. I thought my issue was the web server but I think the issue is with the API or kesterel handing the SSL?

This is my Caddy file:

example.com {

  redir /api /api/
  handle_path /api/* {
    reverse_proxy backend:8000
  }

  handle {
    root * /usr/share/caddy
    file_server
    try_files {path} /index.html
  }

  log {
    output stdout
  }
}

my docker compose:

name: myapp

services:
  backend:
    container_name: mycontainer
    image: myimage
    ports:
      - 8000:8080
  caddy:
    container_name: caddy
    image: caddy
    restart: always
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - caddy-config:/config
      - caddy-data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./index.html:/usr/share/caddy/index.html

volumes:
  caddy-config:
  caddy-data:

this is when I start everythign with docker compose up:

[+] Running 3/3
 ✔ Network myapp_default  Created                                                                                             0.1s
 ✔ Container caddy         Created                                                                                             0.0s
 ✔ Container mycontainer     Created                                                                                             0.1s
Attaching to caddy, mycontainer
caddy      | {"level":"info","ts":1725743811.7998998,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
caddy      | {"level":"info","ts":1725743811.8028176,"msg":"adapted config to JSON","adapter":"caddyfile"}
caddy      | {"level":"warn","ts":1725743811.803073,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy      | {"level":"info","ts":1725743811.8087811,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy      | {"level":"info","ts":1725743811.8092213,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy      | {"level":"info","ts":1725743811.8094192,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy      | {"level":"info","ts":1725743811.8110456,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy      | {"level":"info","ts":1725743811.811443,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
caddy      | {"level":"info","ts":1725743811.8148913,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy      | {"level":"info","ts":1725743811.815138,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy      | {"level":"info","ts":1725743811.8152907,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["example.com"]}
caddy      | {"level":"info","ts":1725743811.8163688,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy      | {"level":"info","ts":1725743811.8165393,"msg":"serving initial configuration"}
caddy      | {"level":"info","ts":1725743811.818198,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000942200"}
caddy      | {"level":"info","ts":1725743811.8239324,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
caddy      | {"level":"info","ts":1725743811.8250542,"logger":"tls","msg":"finished cleaning storage units"}
caddy      | {"level":"info","ts":1725743812.0530305,"logger":"http.acme_client","msg":"got renewal info","names":["example.com"],"window_start":1730607560,"window_end":1730780360,"selected_time":1730768979,"recheck_after":1725765412.053012,"explanation_url":""}
caddy      | {"level":"info","ts":1725743812.0540433,"logger":"tls","msg":"updated ACME renewal information","identifiers":["example.com"],"cert_hash":"5a638e82e1d8085182c5w44372dae49ac0c0425a171d3b9a7147df","ari_unique_id":"kydGmAOpUWiOmNbEQkjbI7I.BG3jz_njzgdss2X27mGXu","cert_expiry":1733284790,"selected_time":1730764641,"next_update":1725765412.053012,"explanation_url":""}

mycontainer| info: Microsoft.Hosting.Lifetime[14]
mycontainer|       Now listening on: http://[::]:8080
mycontainer| info: Microsoft.Hosting.Lifetime[0]
mycontainer|       Application started. Press Ctrl+C to shut down.
mycontainer| info: Microsoft.Hosting.Lifetime[0]
mycontainer|       Hosting environment: Production
mycontainer| info: Microsoft.Hosting.Lifetime[0]
mycontainer|       Content root path: /App

when I hit my domain with curl on the non https url, it works:

curl -v -L http://example.com:8000/api/heartbeat

*   Trying 2600:3c03::f03c:92ff:fe92:c4a9...
* TCP_NODELAY set
* Expire in 149994 ms for 3 (transfer 0x558166564f50)
* Expire in 200 ms for 4 (transfer 0x558166564f50)
* Connected to example.com (2600:3c03::f03c:92ff:fe92:c4a9) port 8000 (#0)
> GET /api/heartbeat HTTP/1.1
> Host: example.com.com:8000
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Date: Sat, 07 Sep 2024 21:48:29 GMT
< Server: Kestrel
< Transfer-Encoding: chunked
<
* Connection #0 to host example.com left intact
hello

When I try https, it doesn’t:

curl -v -L https://example.com:8000/api/heartbeat

* TCP_NODELAY set
* Expire in 149998 ms for 3 (transfer 0x55ee5dca7f50)
* Expire in 200 ms for 4 (transfer 0x55ee5dca7f50)
* Connected to example.com port 8000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

and when I try without the port number on https:

curl -v -L https://example.com/api/heartbeat

 TCP_NODELAY set
* Expire in 149992 ms for 3 (transfer 0x563d5ed06f50)
* Expire in 200 ms for 4 (transfer 0x563d5ed06f50)
* Connected to example.com (2600:3c03::f03c:92ff:fe92:c4a9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=example.com
*  start date: Sep  5 03:59:51 2024 GMT
*  expire date: Dec  4 03:59:50 2024 GMT
*  subjectAltName: host "example.com" matched cert's "example.com"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x563d5ed06f50)
> GET /api/heartbeat HTTP/2
> Host: example.com
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 502
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Sat, 07 Sep 2024 21:51:12 GMT
<
* Connection #0 to host example.com left intact

and caddy logs this:

caddy      | {"level":"error","ts":1725745872.6222372,"logger":"http.log.error.log0","msg":"dial tcp 172.27.0.2:8000: connect: connection refused","request":{"remote_ip":"172.27.0.1","remote_port":"33794","client_ip":"172.27.0.1","proto":"HTTP/2.0","method":"GET","host":"example.com","uri":"/api/heartbeat","headers":{"User-Agent":["curl/7.64.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"example.com"}},"duration":0.00286854,"status":502,"err_id":"psdgnenq6","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
caddy      | {"level":"error","ts":1725745872.6229415,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.27.0.1","remote_port":"33794","client_ip":"172.27.0.1","proto":"HTTP/2.0","method":"GET","host":"example.com","uri":"/api/heartbeat","headers":{"User-Agent":["curl/7.64.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"example.com"}},"bytes_read":0,"user_id":"","duration":0.00286854,"size":0,"status":502,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=":443"; ma=2592000"]}}

In .NET, I’ve tried a few things. I tried both using UseHttpsRedirection and not using it and also setting the https port in my appsettings file and not setting it. Currently, I removed setting the https port from my app settings and this is what my code looks like:

builder.Services.AddControllers();
builder.Services.AddCors(options => options.AddPolicy("CorsPolicy",
builder =>
{
    builder.AllowAnyMethod()
           .AllowAnyHeader()
           .AllowAnyOrigin();
}));
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
//    app.UseForwardedHeaders();
}
else
{
    app.UseExceptionHandler("/Error");
  //  app.UseForwardedHeaders();
  //  app.UseHsts();
}

app.UseCors("CorsPolicy");
//app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.Run();


[Route("api/[controller]")]
[ApiController]
public class HeartBeatController : ControllerBase
{
    [HttpGet]
    public IActionResult Get()
    {
        return Ok("hello");
    }
}

So I’ve been trying to solve this for about a week now and can’t figure out what’s wrong. Anything stand out or suggestions on what to do to debug further?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    So I figured this out and there were two issues. I had to change the port in my Caddy file from 8000 to 8080.

    The other issue is was the handle_path directive in the Caddy file. Apparently, this will match and then strip out the text, so instead of the request being http://example.com/api/heartbeat what was being sent to the api was http://example.com/heartbeat. So, I changed my Caddy file to:

     handle /api/* {
      reverse_proxy http://backend:8080
    }

    So I'm using handle instead of handle_path, which doesn't strip out the "api" from the URL.


  2. 0

    Looks like you are trying to access the same server on the same port 8000 both with http and https. This will not work and this also completely bypasses Caddy. The error "wrong version number" you get there comes from the fact that you try to access a plain HTTP server with HTTPS and the plain HTTP response gets wrongly interpreted as HTTPS.

    That Caddy works with HTTPS can be seen when you use HTTPS with the default port 443 (i.e. "without port" as you say), where Caddy is properly setup and the SSL handshake clearly succeeds. The error 502 produced by Caddy comes from the fact that your server is not reachable by Caddy. Looks like you’ve setup 172.27.0.2:8000 as target here, which does not seem to be the IP + port where your server is running – hence "connection refused". Unfortunately it is not clear from your post where your internal backend server is really running, but maybe it is 127.0.0.1:8080 not 172.27.0.2:8000? At least this different IP and different port would be more in line in what your docker file suggests where the backend should be accessible from inside docker.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search