skip to Main Content

I am currently working on a stream of data that comes from an IoT rule and inputs the readings to a dynamoDB table.

From that I need to rearrange the data to make predictions. For every new data entry, every new row I trigger a lambda function, but I need to get the last 96 table rows to manipulate.

My problem is
1- how can I query the table
2- and transform to something familiar like a pandas dataframe?

In the table I have a timestamp column (the format is still open to be decided)

2

Answers


    1. To query the last 96 items in the table
       dynamodb = boto3.resource('dynamodb', region_name='your_region_name')
       table = dynamodb.Table('your_table_name')
       response = table.scan(
            Limit=96,
            ScanIndexForward=False  # To retrieve the latest rows
        )
       items = response['Items']
    
    1. Convert the data to a Pandas DataFrame

    df = pd.DataFrame(items)

    Login or Signup to reply.
  1. Ultimately it depends on how your data is structured and how you want to access it. But to answer your question, when DynamoDB Streams invokes your Lambda you can use a Query or Scan to retrieve the data from DynamoDB.

    My assumption here is that you need to retrieve the last 96 updates based on timestamp, Scan will not allow you to do that efficiently. You would need to use Query however, that will dictate your data model.

    Depending on your specific use-case needs, I would create a Global Secondary Index with a static partition key such as GSI_PK=1 and your timestamp as the sort key

    gsi_pk gsi_sk data
    1 2023-01-28T07:25:40.824Z data
    1 2023-02-28T07:25:40.824Z data
    1 2023-03-28T07:25:40.824Z data
    1 2023-04-28T07:25:40.824Z data

    Now you can Query your GSI and be sure you are being returned the last 96 items. GSI’s are eventually consistent so be aware of that.

       response = table.query(
            IndexName='my-index',
            KeyConditionExpression=Key('gsi_pk').eq('1'),
            Limit=96,
            ScanIndexForward=False
        )
    
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search