Question 103030 in Azure Question posted in
The official documentation to get you started can be found here.

Azure Event Hub: Receive events as a batch that were sent to hub as individual messages?

I have the following pattern,

Http Trigger Function App -> Event Hub -> Event Hub Listener Function App

Unfortunately, the sender to the HTTP Trigger app sends many individual requests, meaning that the execution of the send to event hub is also a small amount of data per message.

This causes the listener to trigger on every small message, creating many, many files at the end of a data load.

My listener is already configured to receive batch events, and I have seen in the past, that when the listener function was disabled for a while, and then re-enabled, the data would be received in a big batch.

How can I ensure that multiple events are processed in a batch once it gets to the listener, even though their send to the event hub are as individual requests/messages?

Is there some delay setting for the listener, perhaps?

Tags:

2

Answers


  1. 0

    The MaxBatchSize parameter of the EventProcessorOptions object can be used to receive events in a batch.

    To receive events as a batch, you can use the MaxBatchSize property of the EventProcessorOptions object.

    Reference :

    Thank for the reference @ciaranodonnell

    Sending Message:

    EventHubProducerClient producer = new EventHubProducerClient(namespaceConnectionString, eventHubName);
var batch = await producer.CreateBatchAsync(new CreateBatchOptions { PartitionKey = "this is another string" });
for (int i = 0; i < 10; i++)
{
batch.TryAdd(new EventData($"This is event {i}"));
}
await producer.SendAsync(batch);
}

    enter image description here

    enter image description here

    Receiving Message :

    
            EventProcessorOptions options = new EventProcessorOptions
            {
                MaxBatchSize = 100,
                PrefetchCount = 100,
                InvokeProcessorAfterReceiveTimeout = true
            };

            await processorHost.RegisterEventProcessorAsync<MyEventProcessor>(options);
    • The other possible method is message content contains the specific item. In both ways receive batch. For multiple events are processed in a batch using consumer group to send and receive the messages.

    Output :

    enter image description here

    Reference :

    Login or Signup to reply.
  2. 0

    There’s not enough context available to understand how the maxEventBatchSize is configured in your host.json, but I’ll assume that is set to something larger than 1. In that case, it sounds like the partitions do not have a backlog of events and are being read as they arrive. Since there are not enough events to keep the prefetch queue full, events are being dispatched as they stream in.

    I’d recommend taking advantage of the minEventBatchSize that was added in v5.3.0 of the Event Hubs trigger. This works with maxWaitTime to ask the trigger to wait for a bit and build up a batch rather its default behavior of preferring to maximize throughput.

    A minimal example would look something like:

    {
    "version": "2.0",
    "extensions": {
        "eventHubs": {
            "maxEventBatchSize" : 100,
            "minEventBatchSize" : 25,
            "maxWaitTime" : "00:05:00",            
            "batchCheckpointFrequency" : 1,
            "prefetchCount" : 300,
        }
    }
}

    More detail is available in the host.json settings section of the docs.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search