I have a node app that puts events into a kinesis stream. If the partition key is different, is it possible for those 2 events to go into the same shard? The kinesis is configured to be on-demand
Question posted in Amazon Web Sevices
The official Amazon Web Services documentation can be found here.
2
Answers
Yes, it’s possible to send records with different partition keys to same shard. It’s because the kinesis uses an MD5 hashing algorithm to determine the shard location for the events. If the hashing algorithm yields the same hash key ranges for two different partition keys, then those events will be sent to the same shard.
Yes it’s possible, as you won’t have a 1:1 mapping of shards and partition keys.
Each shard will be responsible for a range of values. Your Partition key will MD5 hashed and that will determine which shard it belongs to.
Partition Key
A partition key is used to group data by shard within a stream. Kinesis Data Streams segregates the data records belonging to a stream into multiple shards. It uses the partition key that is associated with each data record to determine which shard a given data record belongs to. Partition keys are Unicode strings, with a maximum length limit of 256 characters for each key. An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards using the hash key ranges of the shards. When an application puts data into a stream, it must specify a partition key.
src