Amazon Web Services Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – How to connect lambda with self-hosted kafka

I am trying to connect kafka and aws lambda on a different hosting provider. When I connect via AWS EC2, it works fine because they are in the same VPC, but when I try to connect kafka and lambda on a different hosting provider, it does not work for the following reason.

Last processing result: PROBLEM: Connection error. Please check your event source connection configuration. If your event source lives in a VPC, try setting up a new Lambda function or EC2 instance with the same VPC, Subnet, and Security Group settings. Connect the new device to the Kafka cluster and consume messages to ensure that the issue is not related to VPC or Endpoint configuration. If the new device is able to consume messages, please contact Lambda customer support for further investigation.

I’m currently set up to access the internet by connecting a subnet to my internet gateway, but I can’t connect this way?

Reference links

I tried connecting to kafka on a different hosting with a NAT gateway and subnet, and I tried connecting to it with an internet gateway and subnet, but that didn’t solve the problem.

Tags:

2

Answers


  1. 0

    As suggested in the error message, create an ec2 instance in the same subnet as your Lambda function and try to connect to Kafka.

    You can create a t2.micro ec2 instance in that subnet.

    Install telnet with

    sudo yum install telnet

    Try to connect to Kafka with (assuming Kafka is listening at 9092)

    telnet <hostname> 9092

    If you cannot connect to it then you know there is no network route to it.

    Then try to connect to any internet site

    telnet google.com 443

    If you cannot connect to it then you know there is no network route to the internet.

    Review your route tables. In the private subnet route table, make sure your route to the internet points to the NAT gateway.

    Check if there are any firewall rules in your other hosting provider in case you have to safelist ingress for the AWS Lambda function.

    Once you are done with the troubleshooting, you can terminate the ec2 instance you temporarily created.

    Login or Signup to reply.
  2. 0

    There may be additional network connectivity required. The networking section may be helpful: https://serverlessland.com/content/guides/lambda-kafka/configuring-kafka-lambda

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search