Azure Question posted in
The official documentation to get you started can be found here.

Upgraded a Containerized .NET App From 6 to 8 and Startup Probe begins to fail in Azure Kubernetes Services

Recently, we’ve upgraded all our services to .NET 8 from .NET 6 and they all completely stopped working on our Azure Kubernetes Cluster. All our services fail by the health check in AKS not being able to reach the container, Hence, having AKS restart the container over and over.

Startup probe failed: Get "http://x.x.x.x:80/health/startup": dial tcp x.x.x.x:80: connect: connection refused

My first suspicion was the breaking change that happened to the Default Ports on .NET 8 Containerized Applications, that changed default port 80 to 8080, specified in the documentation, it says to set the environment variables ASPNETCORE_URLS, ASPNETCORE_HTTP_PORTS, ASPNETCORE_HTTPS_PORTS to the ports that you want the app to use.

It turns out that we already had all our services specify the ASPNETCORE_URLS in the environment variables of the container AND the appsettings.json file before we did the upgrade, which means that the application should be still using those ports. I also modified the Dockerfile to include those Environment variables just in case, but still not luck.

I tried increasing the timeout for the startup Probe and nothing happened either.

startupProbe:
 httpGet:
   path: /health/startup
   port: http
 timeoutSeconds: 30

Notes:

  • I have not changed ANY of the deployment.yaml, service.yaml, or ingress.yaml files for this app because they previously were listening and fully working on PORT 80.
  • When the app is downgraded to .NET 6 it works flawlessly without any other changes needed.

Dockerfile:

FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
ENV ASPNETCORE_URLS=http://+:80
ENV ASPNETCORE_HTTP_PORTS=80
ENV ASPNETCORE_HTTPS_PORTS=443

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY ["nuget.config", "."]
COPY ["NID.Support.API/NID.Support.API.csproj", "NID.Support.API/"]
RUN dotnet restore "NID.Support.API/NID.Support.API.csproj"
COPY . .
WORKDIR "/src/NID.Support.API"
RUN dotnet build "NID.Support.API.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "NID.Support.API.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "NID.Support.API.dll"]

EDIT:

It seems as if Kestrel service tries to start AFTER the application has been shutdown, not before. Can’t tell why that is.

Logs from kubectl

[15:34:51 DBG] Registered model binder providers, in the following order: ["Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BinderTypeModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ServicesModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.HeaderModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FloatingPointTypeModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.EnumTypeModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DateTimeModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.SimpleTypeModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.TryParseModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CancellationTokenModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ByteArrayModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormFileModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormCollectionModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.KeyValuePairModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DictionaryModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ArrayModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CollectionModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ComplexObjectModelBinderProvider"]
[15:34:53 DBG] Hosting starting
[15:34:53 WRN] Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. For more information go to https://aka.ms/aspnet/dataprotectionwarning
[15:34:53 INF] User profile is available. Using '/root/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest.
[15:34:53 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation.
[15:34:53 DBG] Policy resolution states that a new key should be added to the key ring.
[15:34:53 INF] Creating key {xxxx} with creation date 2024-06-12 15:34:53Z, activation date 2024-06-12 15:34:53Z, and expiration date 2024-09-10 15:34:53Z.
[15:34:53 DBG] Descriptor deserializer type for key {xxxx} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
[15:34:53 DBG] No key escrow sink found. Not writing key {xxxx} to escrow.
[15:34:53 WRN] No XML encryptor configured. Key {xxxx} may be persisted to storage in unencrypted form.
[15:34:53 INF] Writing data to file '/root/.aspnet/DataProtection-Keys/key-xxxx.xml'.
[15:34:53 DBG] Key cache expiration token triggered by 'CreateNewKey' operation.
[15:34:53 DBG] Reading data from file '/root/.aspnet/DataProtection-Keys/key-xxxx.xml'.
[15:34:53 DBG] Found key {xxxx}.
[15:34:53 DBG] Considering key {xxxx} with expiration date 2024-09-10 15:34:53Z as default key.
[15:34:53 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
[15:34:53 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
[15:34:53 DBG] Using key {xxxx} as the default key.
[15:34:53 DBG] Key ring with default key {xxxx} was loaded during application startup.
[15:36:05 INF] Application is shutting down...
[15:36:05 WRN] Overriding HTTP_PORTS '80' and HTTPS_PORTS '443'. Binding to values defined by URLS instead 'http://*:80;'.
[15:36:06 DBG] Middleware configuration started with options: {AllowedHosts = *, AllowEmptyHosts = True, IncludeFailureMessage = True}
[15:36:06 DBG] Wildcard detected, all requests with hosts will be allowed.
[15:36:06 ERR] Hosting failed to start
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__14_1(IHostedService service, CancellationToken token)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)

System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__14_1(IHostedService service, CancellationToken token)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Program.<Main>$(String[] args) in /src/NID.Support.API/Program.cs:line 182
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    For anyone experiencing the same situation. Here was the solution to my problems. There was a NuGet Package causing the issues, in their GitHub page there was no release notes at the time for the new version so it would've been almost impossible to know apart from just updating everything we could for our project.

    We upgraded and the Web API started working again without any issues.

    <PackageReference Include="Microsoft.ApplicationInsights.Kubernetes" Version="6.1.2" />

    to

    <PackageReference Include="Microsoft.ApplicationInsights.Kubernetes" Version="7.0.0" />

  2. 0

    The issue seems to be related to a configuration mismatch between the application’s expected port and the port probed by the Kubernetes health check.

    Here are some things you can try to troubleshoot the problem:

    Verify Startup Probe Configuration:

    1. Double-check the startupProbe configuration in the deployment YAML
      file. Ensure the port specified matches the port your application
      listens on (likely 80 based on ASPNETCORE_URLS).

    Check Environment Variable Application:

    1. Verify that the environment variables (ASPNETCORE_URLS,
      ASPNETCORE_HTTP_PORTS) are actually being set within the container.
      You can use tools like kubectl exec to enter the container and check
      the environment variables.

    Address Potential Conflicts:

    1. If other processes are listening on port 80 within the container,
      they might conflict with the application. Consider using a different
      port for the health check or stopping any conflicting processes.

    Review Kubelet Logs:

    1. Check the Kubelet logs for any additional information about the
      startup probe failure. These logs might provide clues about the
      connection attempt.

    Consider Code Changes:

    1. While less likely, there could be code changes in .NET 8 that affect
      the default behavior or health endpoint. Check the official .NET 8
      documentation for any known issues or breaking changes related to
      health checks.

    If the above ways doesn’t work try these steps below:

    Test Locally:

    1. Try deploying the containerized app locally (e.g., using Docker
      Compose) to verify if the issue persists outside of AKS. This can
      help isolate the problem to Kubernetes configuration or the
      application itself.

    Simplify Deployment YAML:

    1. If possible, consider removing unnecessary environment variable
      settings from the deployment YAML and relying solely on the
      Dockerfile settings. This can help reduce configuration complexity.

    Update Documentation:

    1. If the solution involves code changes or configuration adjustments
      specific to .NET 8, try to update the documentation
      to reflect the changes required for a smooth upgrade path.
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search