Execute certain background Tasks (not threads) in separate ThreadPool to avoid starvation to critical Tasks (not threads) executed in main thread
Our Scenario
We host a high-volume WCF web service, which logically has the following code:
void WcfApiMethod()
{
// logic
// invoke other tasks which are critical
var mainTask = Task.Factory.StartNew(() => { /* important task */ });
mainTask.Wait();
// invoke background task which is not critical
var backgroundTask = Task.Factory.StartNew(() => { /* some low-priority background action (not entirely async) */ });
// no need to wait, as this task is best effort. Fire and forget
// other logic
}
// other APIs
Now, the issue, in certain scenarios, the low-priority background task may take longer (~ 30 sec), for e.g., to detect SQL connection issue, DB perf issues, redis cache issues, etc. which will make those background threads delayed, which means TOTAL PENDING TASK COUNT will increase, due to high volume.
This creates a scenario where, newer executions of the API cannot schedule high-priority task, because, lot of background tasks are in queue.
Solutions we tried
-
Adding TaskCreationOptions.LongRunning to high-pri task will immediately execute it.
However, this cannot be a solution for us, as there are lot of tasks being invoked everywhere in the system, we cannot make them long-running everywhere.
Also, WCF handling of incoming APIs would rely on .NET thread pool, which is in starvation now. -
Short-circuit low-pri-background task creation, via Semaphore. Only spawn threads if the system has capacity to process them (check if earlier created threads have exited). If not, just don’t spawn up threads.
For e.g., due to an issue (say DB perf issue), ~ 10,000 background threads (non-async) are on IO wait, which may cause thread-starvation in main .net thread pool.
In this specific case, we could add a Semaphore to limit creation to 100, so if 100 tasks are stuck, 101st task won’t be created in the first place.
Ask on alternative solution
Is there a way to specifically spawn “tasks” on “custom threads/ thread pool”, instead of the default .NET thread pool.
This is for the background tasks I mentioned, so in case they get delayed, they don’t bring down the whole system with them.
May be override and create a custom TaskScheduler to be passed into Task.Factory.StartNew() so, tasks created would NOT be on default .NET Thread Pool, rather some other custom pool.
2
Answers
Based on https://codereview.stackexchange.com/questions/203213/custom-taskscheduler-limited-concurrency-level?newreg=acb8e97fe4c94844a660bcd7473c4876, there does exist an inbuilt solution to limit thread-spawns via limited concurrency TaskScheduler.
Inbuilt
ConcurrentExclusiveSchedulerPair.ConcurrentScheduler
could be used to achieve this.For the above scenario, following code limits background threads from wrecking the application/ prevents thread-starvation.
Thanks @Theodor Zoulias for the pointer.
Here is a static
RunLowPriority
method you could use in place ofTask.Run
. It has overloads for simple and generic tasks, and for normal and asynchronous delegates.The actions scheduled through the
RunLowPriority
method will run onThreadPool
threads, but at maximum 2 of all the availableThreadPool
threads can be concurrently assigned toRunLowPriority
tasks.Keep in mind that the
Elapsed
event of aSystem.Timers.Timer
that has itsSynchronizingObject
property set tonull
runs inThreadPool
threads too. So if you are doing low priority work inside this handler, you should probably schedule it through the same limited concurrency scheduler: