Azure Web Application (windows) – stalling during CILJit::compileMethod in calls to Entity Framework Core

I have been looking into performance, specifically calls to an ASP.NET Core 3.1 Web API project that is running on Azure.

Note: yes, we should be moving to a later version of .NET Core, and that’s in the pipeline, but it’s not something I can just switch to without a bit of effort.

We are targeting netcoreapp3.1 for our libraries, and are referencing Entity Framework Core v3.1.5.

Looking at a typical end-to-end trace in Application Insights, we see this:

If I’m reading this correctly, we’re spending a grand total of 135ms in the database executing queries, but between the last 2 queries we appear to stall for ~12 seconds!

When I dig into the profiler trace for this request, I see this:

Again, if I read this right, that means during the second DB call (from our end to end transaction above), we spend ~12.4 seconds inside the call to EntityFrameworkQueryableExtensions.ToListAsync() doing some jit compilation.

That looks excessive to me.

This appears to be a pattern that I see through out the day, even though the application is set to be Always On and there are no restarts of the application between occurrences of this.

The questions I have around this are:

• is this to be typically expected?
• if so, should it really be taking this long?
• is there a way to reduce the need to jit as often as we appear to be doing?
• will a move to .NET 6 (and future framework versions) help us here?

On average, the API operates pretty well, and does have a typical avg response time in the < 1 second range. However, when these do occur, they are noticeable and are causing me a headache.