My application sends requests to Azure Machine Learning REST API in order to invoke a batch endpoint and start scoring jobs as described here. It works well for small number of requests, but if the app sends many concurrent requests the REST API sometimes responds with status code 429 "TooManyRequests" and message "Received too many requests in a short amount of time. Retry again after 1 seconds.". For example, it happened after sending 77 requests at once.
The message is pretty clear and the best solution I can think about is to throttle outgoing requests. That is making sure the app doesn’t exceed limits when it sends concurrent requests. But the problem is I don’t know what are the request limits for Azure Machine Learning REST API. Looking through the Microsoft documentation I could only find this article which provides limits for Managed online endpoints whereas I’m looking for Batch endpoints.
I would really appreciate if someone helped me to find the Azure ML REST API request limits or suggested a better solution. Thanks.
UPDATE 20 Jun 2022:
I couldn’t find out how many concurrent requests are allowed by Azure Machine Learning batch endpoints. So I ended with a limit of 10 outgoing requests which solved the "TooManyRequests" problem. In order to throttle requests I used SemaphoreSlim as described here.
2
Answers
According to the document, there is chance to enhance the quota of the request limit which is the way to solve the request limit exceed issue. Regarding batch quota limit, here is the document designed by Microsoft.
According to the above image, change the quota values.
Document Credit: prkannap and team
Alternatively, you could reduce the number of requests by storing multiple input files in a folder and invoking the job with the folder path.
If you want further assistance, please file a support ticket and a customer support engineer will assist you.