Question 19225 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

How does AWS Lambda determine if messages are still in SQS queue?

When using AWS Lambda with a SQS queue (as event source), it is written in the doc

If messages are still available, Lambda increases the number of
processes that are reading batches by up to 60 more instances per
minute.
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

My question here is how does the Lambda service determine "If messages are still available" ?

Tags:

5

Answers


  1. 0

    I believe the documentation refers to Lambda polling the queue to know whether there are still messages. Read more about it here.

    Lambda polls the queue and invokes your Lambda function synchronously
    with an event that contains queue messages. Lambda reads messages in
    batches and invokes your function once for each batch. When your
    function successfully processes a batch, Lambda deletes its messages
    from the queue.

    Event Source Mapping:

    Lambda only sees messages that are visible, via the visibility timeout setting on the SQS queue. This is to prevent other queue consumers processing the message. I believe as an event-source, Lambda receives messages from the SQS queue, via being mapped to it.

    Login or Signup to reply.
  2. 0

    As per the documentation you shared,for standard queues, Long Polling is in effect. Long polling basically waits for a certain amount of time to verify if there is a message in the queue. refer to the following docs :

    https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html

    https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/confirm-queue-is-empty.html

    Login or Signup to reply.
  3. 0

    One of the calls to the SQS API is to get queue attributes (Java API, others similar). This returns a response and one of the attributes of the response is "approximate number of messages". With this you or AWS can determine about how many messages are in the queue.

    From this, AWS can determine if it’s worth spinning up additional instances. You too can get this information from the queue.

    Login or Signup to reply.
  4. 0

    I imagine it uses the ApproximateNumberOfMessagesVisible metric on the SQS queue to check how many messages are available, and uses that number, plus your batch size configuration, to determine how many more Lambda instances your function needs to be scaled out to.

    Login or Signup to reply.
  5. 0

    Answering the "how" question in a slightly different way:

    Behind the scenes, Lambda operates a "State Manager" control-plane service that discovers work from the queue. State Manager also manages scaling of the fleet of "Poller" workers that do the actual retrieving, batching, invoking, and deleting.

    These implementation details are from the Event Source Mapping section of the re:Invent 2022 video A closer look at AWS Lambda (SVS404-R). Here is a screenshot:

    enter image description here

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search