I am new to the AWS ecosystem and I am trying to design a POT that involves sending data from an AWS Kinesis Data Stream to an AWS Step Function. I know I can make a Lambda function to consume the Kinesis Data Stream and invoke the Step Function from that Lambda, but I’m curious if I can bypass the Lambda and have the Step Function consume the Kinesis Data Stream directly. Within the Step Function designer you can make the Kinsis Data Stream’s GetRecords API a state.
But once added, there is no way to point that state to my specific stream that I can see within the configuration. I can’t find any documentation or examples on how to make an AWS Step Function a consumer of a Kinesis Data Stream. Being new to AWS I’m curious about two things:
- Is this possible to do
- If it is possible is this a good idea from an architecture standpoint.
2
Answers
No. Step Functions cannot be invoked directly as a Kinesis Data Streams consumer.
References:
You can now do this using EventBridge Pipes, launched at re:Invent 2022.
Pipes allows you to configure a source, including Kinesis Data Streams, and a target, including Step Functions, then it will take care of polling from your stream and reliably delivering events to the targets. You can optionally filter and enrich the data on the way through the pipe.
For more information on configuring Pipes with Kinesis Data Streams, check out this documentation page.