I need help designing a web API system (.NET 6.0) that can serve 100K requests per second (in bursts).
All I need to do is expose an HTTP endpoint that will do a few calculations using Cosmos DB and SQL DB (read optimised replica) and return a result in under a couple of seconds.
The catch is that this HTTP endpoint should be able to serve 100K requests a second.
I havent dealt with such huge volumes before so I am not sure what is the best way to approach this in Azure.
Plase also note that this load is not constant and there could be prolonged periods of inactivity followed by bursts of up to 100K requests per second.
Could you please share your insights on what could be the best approach to solve this problem?
Cost is not an issue (up to a point of course. paying tens of thousands a month for this would be an overkill).
Based on my research so far there are currently 3 options that I can see:
- Host API in Azure App Service
- Use either an http triggered Azure functions on premium plan or API M in front of those Azure functions
- Use Azure container Apps for hosting this Web API.
Are any of these options even capable of serving such loads?
Is their scaling out/in fast enough for such bursts?
What are the possible problems I could run into?
If you think any other approach can work please share by all means as well.
2
Answers
Based on your input the answer is: we don’t know.
Just picking .net 7 as an example, the official benchmarks reports 14.6MM requests per second:
source:
https://devblogs.microsoft.com/dotnet/performance-improvements-in-aspnet-core-7/
Does it mean you’ll achieve the same throughput hosting in any of the options you listed? No.
What you really need to do to is perform a load test in each of the options and realize which one best fits your needs.
I was facing the similar issue, hosting the api in Azure App Service. Identical scenario using CosmosDB and dealing with huge volumes(load was not constant and there were prolonged periods of inactivity followed by bursts of up to 100K requests per second). I started with designing the api and setting CosmosDB Throughput 4,000 RU/s, min charge 400 RU/s, still within the free tier limit. After that I was using Jmeter for load testing(do not even think about using Azure Load Testing it is incredibly inaccurate). You have to start small and work your way up. So obviously to determine the throughput for Cosmos DB to handle 100k calls per second I think the very important thing is the document size, you said "will do a few calculations", the size of the document that your api reads will impact the throughput especially if it is large(Caching can become a miracle and also you can use Load balancer for API in Azure, can probably give you more info on this if you have never used these tools). You will have to do thousands of integration tests(will cost you money as well). Start small with 1000 request per second and see if the api is able to respond(Jmeter has got excellent tooling for this, you will see how many of those request were actually successful). Start with a basic api plan and basic throughput and once you get to the point your api is struggling, just increase the throughput and also the price plan. From my experience as long as "Cost is not an issue" Azure can definitely handle that(just to give you peace of mind), however as you said "up to a point of course. paying tens of thousands a month for this would be an overkill". 100k calls per second will be super expensive(not joking here), I remember when I estimated the bill for the client they were not happy(even though they first said cost is not an issue) so make sure they know that and you get some estimates on small numbers(assuming if your client gets 100k calls per second the cost for them might be negligible). However this is only my experience with hosting API in Azure App Service. I have not tried http triggered Azure functions or Azure container Apps so I cannot say that this approach is the best but it definitely worked for me. The total cost of your project will also depend on the total amount of calls you make so if you have only a few bursts of up to 100K requests per second every month it is going to cost you significantly less than having 10000000000 calls each month.