Question 19175 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Concurrency of Lambda not increasing as expected

We’ve configured an MSK (kafka) event source as the trigger for our Lambda function. Even though the offset lag is increasing the lambda concurrency is limited to 4-5 almost all the time as can be seen in the graph below. The configuration used for the MSK event source is:

Batch Size: 50
Batch window: 30 seconds
Number of partitions in the Kafka topic: 10

I made sure that the load is distributed equally across all the partitions. Is there anything I’m missing here which is causing the concurrency issue? Any solution is appreciated. Thanks in advance.

Offset Lag vs Concurrent Executions

Tags:

2

Answers


  1. 0

    I think you are hitting the same limitation we found some months ago, this link led us in the right way (aka workaround in our case):

    AWS MSK lambda concurrent consumers

    It honestly makes sense that the partitions are not being used in all their capability because the jump from the msk EC2 setup to the lambda runtime is not something trivial. Maybe you can try other connectors.

    https://docs.confluent.io/kafka-connectors/aws-lambda/current/overview.html#multiple-tasks

    It also makes sense that bridging through Kinesis you would not have these specific issues as it is all Amazon native stuff.

    Login or Signup to reply.
  2. 0

    Ideally concurrency should be a number with no decimals match the count of consumers count.

    When you initially create an an Apache Kafka event source, Lambda allocates one consumer to process all partitions in the Kafka topic. Each consumer has multiple processors running in parallel to handle increased workloads. Additionally, Lambda automatically scales up or down the number of consumers, based on workload. To preserve message ordering in each partition, the maximum number of consumers is one consumer per partition in the topic.

    source: https://docs.amazonaws.cn/en_us/lambda/latest/dg/with-kafka.html#services-kafka-scaling

    The Offset Lag indicates performance issue for this blog gives better explanation Offset lag metric

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search