Azure Web App with four virtual applications fails to start one ASP.NET Core 6 Web API

We have an app service that hosts four virtual applications: a Vue app (static), an Angular app (static) and two APIs written in .NET:

Four virtual applications

These four run side by side successfully, they were written in .NET Framework 4.7.

After upgrading both to .NET 6, one of the two APIs keeps restarting and does not return anything.

We’ve done:

• API 1 is upgraded to .NET 6 and deployed in AspNetCoreHostingModel InProcess. Both APIs still work

• API 2 is upgraded to .NET 6 and deployed in AspNetCoreHostingModel InProcess. API 2 throws an error

  HTTP Error 500.35 – ANCM Multiple In-Process Applications in same Process: multiple InProcess is not allowed.

• API 2 is changed to AspNetCoreHostingModel OutOfProcess. API 2 now throws an error

  HTTP Error 500.34 – ANCM Mixed Hosting Models Not Supported: It’s not supported to have one InProcess and one OutOfProcess.

• API 1 and 2 are changed to AspNetCoreHostingModel OutOfProcess. API 2 is working successfully, API 1 is not working anymore, it keeps starting over and over without giving a result, and no exceptions are logged.

When looking at the logging only the kestrel lines are shown, apparently it doesn’t get to our code. While API 2 shows that each time it starts on localhost with a random port, API 1 consistently starts on localhost, port 5000.

When I ancm log it reads that it starts to listen for API 1 on a random port, until it times out since it can’t connect.

It appears API 1 is incorrectly starting on port 5000 even though we haven’t configured this anywhere explicitly.

UPDATE:

The project file:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
      <AspNetCoreHostingModel>OutOfProcess</AspNetCoreHostingModel>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="7.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="7.0.1" />
    <PackageReference Include="Serilog.AspNetCore" Version="7.0.0" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.5.0" />
    <PackageReference Include="Swashbuckle.AspNetCore.Newtonsoft" Version="6.5.0" />
    <PackageReference Include="System.Net.Http" Version="4.3.4" />
  </ItemGroup>

  <ItemGroup>
    ...
  </ItemGroup>

</Project>

The web.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="dotnet" arguments=".Rdc.Caremail.Web.dll" stdoutLogEnabled="false" stdoutLogFile=".logsstdout" hostingModel="OutOfProcess" />
    </system.webServer>
  </location>
</configuration>

UPDATE 2:

Log file from ANCM on the working api (the startup):

dbug: Microsoft.AspNetCore.Mvc.ModelBinding.ModelBinderFactory[12]
      Registered model binder providers, in the following order: Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BinderTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ServicesModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.HeaderModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FloatingPointTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.EnumTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DateTimeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.SimpleTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CancellationTokenModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ByteArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormFileModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormCollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.KeyValuePairModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DictionaryModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ComplexObjectModelBinderProvider
dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
      Hosting starting
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://127.0.0.1:13460

Log file from ANCM on the api with error (the startup):

dbug: Microsoft.AspNetCore.Mvc.ModelBinding.ModelBinderFactory[12]
      Registered model binder providers, in the following order: Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BinderTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ServicesModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.HeaderModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FloatingPointTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.EnumTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DateTimeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.SimpleTypeModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CancellationTokenModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ByteArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormFileModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormCollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.KeyValuePairModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DictionaryModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ArrayModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CollectionModelBinderProvider, Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ComplexObjectModelBinderProvider
dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
      Hosting starting
dbug: Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer[1]
      Unable to locate an appropriate development https certificate.
dbug: Microsoft.AspNetCore.Server.Kestrel[0]
      No listening endpoints were configured. Binding to http://localhost:5000 by default.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:5000

UPDATE 3:
I did find an article from Microsoft (app service .net core one odd cause) in which the symptom described in the logging also occurs, and it does seem to be something broken in the web app. Note: I already checked, we don’t have site extensions installed.
I guess I will try to rebuild the web app and see if this resolves it.