Need help configuring DLQ for Lambda triggered by SNS - Amazon Web Sevices

Dave
January 20, 2023
281 views
1 vote
2 Answers

I’d like to receive an email if my Lambda fails. The Lambda is triggered via SNS (which is triggered by SES).

When I publish to the SNS Topic, the Lambda runs and throws an error (for testing) due to a missing package. I see from the console logs that the Lambda runs 3 times.

I have an SQS queue attached to the Redrive policy (dead-letter queue) of the SNS Topic’s subscription (that triggers the lambda).

{
  "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:myproj-sns-topic-dlq"
}

I tested, and things didn’t work. I noticed a warning in the AWS console for the SNS Topic’s subscription:

Dead-letter queue (redrive policy) permissions The Amazon SQS queue
specified as the dead-letter queue for your subscription (redrive
policy) doesn’t permit deliveries from topics. To allow an Amazon SNS
topic to send messages to an Amazon SQS queue, you must create an
Amazon SQS queue policy.

Following the steps Subscribing an Amazon SQS queue to an Amazon SNS topic, I added the 2nd statement to my SQS queue’s Access policy:

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:myproj-sns-topic-dlq"
    },
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:myproj-sns-topic-dlq",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-1:123456789012:myproj-snstopic"
        }
      }
    }
  ]
}

The Principal was {"Service": "sns.amazonaws.com"}, but that results in a warning in the AWS console saying it can’t test permissions. I tested anyway and it didn’t work. (Lambda runs 3 times, but nothing gets put in the DLQ.)

I set the Principal to * for now (per snippet above). That eliminates the warning in the console, but things still don’t work.

My goal it to have the event drop into the SQS DLQ after the Lambda fails. I have an alarm on that queue that will notify me by email…

Edit: added missing condition

Answers

Chosen as BEST ANSWER

As pointed out by @fedonev, the SNS Subscription's (for lambda) DLQ is used when the event cannot be delivered. If the event is delivered (but the Lambda fails), you can use Lambda's async event DLQ or wire-up the 'on failed' destination of the Lambda.

I'm using AWS Amplify and decided to use the Lambda's "async" DLQ as opposed to a lambda destination.

Step 1 - Add a custom category to add:

SQS (dlq) to save the failed attampt's event
CloudWatch Alarm to watch the SQS resource
SNS Topic and Subscription(s) used by the Alarm

And "Output" the SQS queue's ARN which is needed by the Lambda.

Step 2 - Add a "DeadLetterConfig" to the Lambda that pushes fails into the above queue.

amplify add custom

name: LambdaAlarm

File: amplify/backend/custom/LambdaAlarm/LambdaAlarm-cloudformation-template.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Parameters": {
    "env": {
      "Type": "String"
    }
  },
  "Resources": {
    "SQSDLQ": {
      "Type": "AWS::SQS::Queue",
      "Properties": {
        "QueueName": {
          "Fn::Join": [
            "",
            [
              "myproject-lambdafailed-dlq",
              {
                "Fn::Select": [
                  3,
                  {
                    "Fn::Split": [
                      "-",
                      {
                        "Ref": "AWS::StackName"
                      }
                    ]
                  }
                ]
              },
              "-",
              {
                "Ref": "env"
              }
            ]
          ]
        },
        "MessageRetentionPeriod": 1209600,
        "VisibilityTimeout": 5432,
        "SqsManagedSseEnabled": false
      }
    },
    "SNSTOPIC": {
      "Type": "AWS::SNS::Topic",
      "Properties": {
        "TopicName": {
          "Fn::Join": [
            "",
            [
              "myproject-lambda-failed-alarm-topic",
              {
                "Fn::Select": [
                  3,
                  {
                    "Fn::Split": [
                      "-",
                      {
                        "Ref": "AWS::StackName"
                      }
                    ]
                  }
                ]
              },
              "-",
              {
                "Ref": "env"
              }
            ]
          ]
        }
      }
    },
    "SNSSubscriptionEmailJoeAtGmail": {
      "Type": "AWS::SNS::Subscription",
      "Properties": {
        "Protocol": "email",
        "TopicArn": {
          "Ref": "SNSTOPIC"
        },
        "Endpoint": "[email protected]"
      }
    },
    "SNSSubscriptionEmailJillAtQuad": {
      "Type": "AWS::SNS::Subscription",
      "Properties": {
        "Protocol": "email",
        "TopicArn": {
          "Ref": "SNSTOPIC"
        },
        "Endpoint": "[email protected]"
      }
    },
    "ALARM": {
      "Type": "AWS::CloudWatch::Alarm",
      "Properties": {
        "AlarmName": {
          "Fn::Join": [
            "",
            [
              "myproject-lambda-failed-dlq-alarm",
              {
                "Fn::Select": [
                  3,
                  {
                    "Fn::Split": [
                      "-",
                      {
                        "Ref": "AWS::StackName"
                      }
                    ]
                  }
                ]
              },
              "-",
              {
                "Ref": "env"
              }
            ]
          ]
        },
        "AlarmDescription": "There are messages in the 'Lambda Failed' dead letter queue.",
        "Namespace": "AWS/SQS",
        "MetricName": "ApproximateNumberOfMessagesVisible",
        "Dimensions": [
          {
            "Name": "QueueName",
            "Value": {
              "Fn::GetAtt": [
                "SQSDLQ",
                "QueueName"
              ]
            }
          }
        ],
        "Statistic": "Sum",
        "Period": 60,
        "EvaluationPeriods": 1,
        "Threshold": 0,
        "ComparisonOperator": "GreaterThanThreshold",
        "AlarmActions": [
          {
            "Ref": "SNSTOPIC"
          }
        ]
      }
    }
  },
  "Outputs": {
    "SQSDLQArn": {
      "Value": {
        "Fn::GetAtt": [
          "SQSDLQ",
          "Arn"
        ]
      }
    }
  },
  "Description": ""
}

Next, update and add the new custom resource as a dependency of the Lambda(s) to monitor.

File: backend-config.json

"function": {
  "MYLambda": {
    "build": true,
    "dependsOn": [
      {
        "attributes": [
          "SQSDLQArn"
        ],
        "category": "custom",
        "resourceName": "LambdaAlarm"
      }
    ],
    "providerPlugin": "awscloudformation",
    "service": "Lambda"
  },
},

In the Lambda(s) you want to monitor, make 3 changes to the cloudformation:

Pull in the output variable (customLambdaAlarmSQSDLQArn) from your custom category and add it to the Parameters
Add the DeadLetterConfig property to the Lambda
Add a policy to the LambdaExecutionRole

File: amplify/backend/function/MyLambda/MyLambda-cloudformation-template.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "...",
  "Parameters": {
    ...snip...
    "customLambdaAlarmSQSDLQArn": {
      "Type": "String"
    },
    ...snip...
  },
  "Conditions": {...snip...},
  "Resources": {
    "LambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Metadata": {...snip},
      "Properties": {
        "Code": {...snip...},
        "Handler": "index.handler",
        "FunctionName": {...snip...},
        "Environment": {
          "Variables": {...snip...}
        },
        "Role": {...snip...},
        "Runtime": "nodejs18.x",
        "Architectures": ["arm64"],
        "Layers": [],
        "MemorySize": 256,
        "Timeout": 120,
        "DeadLetterConfig": {
          "TargetArn": {
            "Ref": "customLambdaAlarmSQSDLQArn"
          }
        }
      }
    },
    "LambdaExecutionRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": {...snip...},
        "Policies": [
          {
            "PolicyName": "custom-lambda-execution-policy",
            "PolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Sid": "AllowSQSSendMessage",
                  "Effect": "Allow",
                  "Action": [
                    "SQS:SendMessage"
                  ],
                  "Resource": {
                    "Ref": "customLambdaAlarmSQSDLQArn"
                  }
                }
              ]
            }
          }
        ],
        "AssumeRolePolicyDocument": {...snip...}
      }
    },
    "lambdaexecutionpolicy": {...snip...},
    "AmplifyResourcesPolicy": {...snip...},
    "CustomLambdaExecutionPolicy": {...snip...}
  },
  "Outputs": {...snip...}
}

Finally, due to an Amplify quirk you must amplify env checkout dev because you manually touched the backend-config.json file.

Then you can deploy your changes. The above is not specific to AWS Amplify.

(Edit)

- AlbertMarrero
- January 22, 2023 at 9:41 am
- 0 votes
0
According to this article, you can use a CloudWatch Log filter to parse a log for a Lambda function and get an email notification.

To implement this solution, you must create the following:
- An SNS topic
- An IAM role
- A Lambda function
- A CloudWatch log trigger
Login or Signup to reply.

Please signup or login to give your own answer.

Click here to cancel reply.

Need help configuring DLQ for Lambda triggered by SNS – Amazon Web Sevices

Answers