Is it possible to implement the following pseudo scenario with an Azure Service Bus?
I have a function that can scale out to 50 instances, it uses a service bus trigger. I would like to guarantee that related messages are only processed if an existing related message is NOT currently being processed.
Let’s say I have a message (Message A) being processed by a function instance that’s associated with UserID 1234. Another message (Message B) appears on the queue which is also associated with UserID 1234, the service bus should "ignore" it because a related message is already being processed. Another message (Message C) with UserID 9876 appears on the queue, this gets handled straight away because there is no in action message with UserID 9876.
Message A finishes processing and Message B is now picked up.
Currently I have a routing function which consumes the initial service bus trigger and then routes it to one of 10 functions each of which is responsible for messages where the last digit of the UserID is 0-9.
This means that if function "4" is busy with a request it won’t be able to process any other requests where the UserID ends with 4, thus guaranteeing the system cannot process a related message at the same time. It does it’s job but doesn’t scale.
2
Answers
As far as I know, azure service bus has a locking mechanism built into it. So no matter the message, if you have a single queue you are reading from, when a function picks that message up the rest of your functions would not pick up that same message. To solve the issue of not processing a duplicate message by userID I would recommend using something table storage to validate if that userID has been processed already (So in your function when you pick up a message you insert that into azure table storage before doing any processing and also have a check to see if it exists before processing).
service bus – https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement
table storage – https://learn.microsoft.com/en-us/azure/cosmos-db/table/quickstart-dotnet?toc=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Fstorage%2Ftables%2Ftoc.json&bc=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Fbread%2Ftoc.json&tabs=azure-cli%2Cwindows
In summary I think the solution here would be using multiple technologies and use a central point your scaled out functions could validate against whats processed by other functions, whatever that central database is
There’s no conditional locking. From the description, sounds like you want to process messages associated with the same user ID, one at a time. For that, Azure Service Bus has a feature called Message Sessions.