Question 108210 in Azure Question posted in
The official documentation to get you started can be found here.

Exception about running out of free space on D:localTemp in .Net 6 Web API hosted on Azure App Service (even though there's lots of space)

I have a .Net 6 Web API running on Azure App Service (Windows, Standard Service Plan (S1: 1))

Occasionally, I get this exception:

System.IO.IOException: There is not enough space on the disk. :
‘D:localTempASPNETCORE_7aca67c0-7d1f-40ba-bef3-9e58ea744487.tmp’

This is the stack trace for the last time it happened:

System.IO.IOException:
   at System.IO.RandomAccess.WriteAtOffset (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.Strategies.OSFileStreamStrategy.Write (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.Strategies.OSFileStreamStrategy.Write (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.WebUtilities.PagedByteBuffer.MoveTo (Microsoft.AspNetCore.WebUtilities, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingWriteStream.Write (Microsoft.AspNetCore.WebUtilities, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.Write (Microsoft.AspNetCore.WebUtilities, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Newtonsoft.Json.JsonTextWriter.WritePropertyName (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal (Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed)
   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter+<WriteResponseBodyAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.NewtonsoftJson, Version=6.0.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter+<WriteResponseBodyAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.NewtonsoftJson, Version=6.0.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonOutputFormatter+<WriteResponseBodyAsync>d__13.MoveNext (Microsoft.AspNetCore.Mvc.NewtonsoftJson, Version=6.0.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeResultAsync>g__Logged|22_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeNextResultFilterAsync>g__Awaited|30_0>d`2.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeFilterPipelineAsync>g__Awaited|20_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware+<<Invoke>g__AwaitRequestTask|6_0>d.MoveNext (Microsoft.AspNetCore.Routing, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at PropWorx.API.Middlewares.TenantIdentifier+<Invoke>d__4.MoveNext (PropWorx.API, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:UsersfabsrsourcereposPropWorx.APIPropWorx.APIMiddlewaresTenantIdentifier.cs:50)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at PropWorx.API.Middlewares.WebSocketsMiddleware+<Invoke>d__2.MoveNext (PropWorx.API, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:UsersfabsrsourcereposPropWorx.APIPropWorx.APIMiddlewaresWebSocketsMiddleware.cs:30)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler+<HandleAsync>d__0.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Session.SessionMiddleware+<Invoke>d__8.MoveNext (Microsoft.AspNetCore.Session, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Session.SessionMiddleware+<Invoke>d__8.MoveNext (Microsoft.AspNetCore.Session, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware+<Invoke>d__5.MoveNext (Swashbuckle.AspNetCore.SwaggerUI, Version=6.3.1.0, Culture=neutral, PublicKeyToken=4232c99127b3c254)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware+<Invoke>d__4.MoveNext (Swashbuckle.AspNetCore.Swagger, Version=6.3.1.0, Culture=neutral, PublicKeyToken=62657d7474907593)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware+<Invoke>d__9.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)

This happens on various API endpoints (i.e. various action methods in various controllers).
The strange thing is that it’s happening in methods where I’m not performing any file writing activities. It’s simple GET request which performs a basic SELECT * FROM table query from a MySql database, and returns the results.

Now, if I go to Azure Portal, and then to my app service, and then to Quotas, I have plenty of free space left:

enter image description here

If I go to Kudu, and then Environment, I see this:

System info

System up time: 12:22:00.1560000
OS version: Microsoft Windows NT 10.0.14393.0
64 bit system: True
64 bit process: True
Processor count: 1
Machine name: RD2818786D33D4
Instance id: eeb07ed981018f5b52e2782779334447fffe71fbeeea9200f63be35f6a12f44a
Short instance id: eeb07e
CLR version: 4.0.30319.42000
System directory: D:Windowssystem32
Current working directory: D:Program Files (x86)SiteExtensionsApplicationInsightsAgent2.8.43ExtensionManager
IIS command line: D:Windowssystem32inetsrvw3wp.exe -ap "~1propworx-api-san" -v "v4.0" -a "\.pipeiisipmf473d9e2-218c-4cdb-abbe-f3fa8fd44d9a" -h "C:DWASFilesSites#1propworx-api-sanConfigapplicationhost.config" -w "C:DWASFilesSites#1propworx-api-sanConfigrootweb.config" -m 0
D:home usage: 51,200 MB total; 47,873 MB free
D:local usage: 11,264 MB total; 10,894 MB free

If you look at the two last lines, I have plenty of space on both D:home and D:local

Any ideas?

Tags:

3

Answers


  1. Chosen as BEST ANSWER
    0

    Shortly after posting this question, I deleted all the logs in the LogFiles folder on Azure (I did this via FTP / FileZilla). There were tons of log files going back a long time. I wasn't sure if these contributed towards, what I think could, be a limit on the available temporary storage space, but it's been nearly two weeks, and I haven't had any more "There is not enough space on the disk" exceptions.

    I will continue to monitor it. I'm not really sure if this was the fix, but it appears to have helped. I'm half-reluctantly closing this post though as I don't want to keep it open indefinitely.

    Thanks to everyone for your input.


  2. 0

    Your application process experienced a crash. This could be caused by application code.

    Suggestions for Monitoring Crash

    Exception Code  
1 crash due to (0xE0434352 - CLR Exception)

    If the detector below does not have exception details for the crash, suggest the customer to enable Crash Monitoring by following these steps.

    1. Go to App Service Diagnostics
    2. Choose Diagnostic Tools .
    3. Click on Crash Monitoring from menu options and follow the instructions.

    Suggestions for HTTP Server Errors

    ExceptionType                              Method                         Count
System.IO.IOException   PropWorx.API.Services.Auth0Service+d__4.MoveNext    5
System.Threading.Tasks.TaskCanceledException    PropWorx.API.Middlewares.TenantIdentifier+d__4.MoveNext 3
System.IO.IOException   PropWorx.API.Services.Auth0Service+d__4.MoveNext    2

    HTTP 500.0 error typically indicates an application code issue. An un-handled exception in the application code or an error in application is what typically causes this error. Please check the code.

    The issue I found, please check it in your side

    1. NLog and MySQL, please check it in your Application Exceptions

      System.AggregateException: An error occurred while writing to logger(s). (Unable to connect to any of the specified MySQL hosts.)

    2. Slow Request Execution

    3. You need to check your AppInsights. You can find a lot of exceptions in it.

    Summary

    1. please contact your developer Teams, let them review the code according the exceptions.

    2. Enable Local Cache

    3. please read this article, it also will useful to you.

      The Ultimate Guide to Running Healthy Apps in the Cloud

    Login or Signup to reply.
  3. 0

    Looks like it is explained here. The solution is to enable buffering on the requests. ASP.NET CORE writes request body to Windows TEMP folder

    M$ doc for request buffering. https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.http.httprequestrewindextensions.enablebuffering?view=aspnetcore-6.0

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search