Question 245338 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – How to get total count from a Dynamo DB based on the ttl value

I am trying to get how many data are present in Dynamo DB based on the ttl column value.
In the source, we could see only 55k events are there. but in the ddb table we could see 122k events are there. By default our ttl value is 4 days. Hence, based on particular period we want to see how many events/data present in the ddb table. we want to ignore the data where ttl value is less than current time. I’m new to AWS. So, kindly help me with detailed code. Thank you 🙂

Primary key : customer_id (String)
ttl column name : ttl (TTL)
other generic column names are record_insert_time, record_update_time etc..

I tried below code. But getting error as in the below

import boto3

dynamodb = boto3.client('dynamodb')

table_name = 'table name' # i used my table name here.
ttl_threshold = 1631303477  


count = 0

query_params = {
    'TableName': 'table name', # i used my table name here.
    'KeyConditionExpression': 'TTLAttributeName > :threshold',
    'ExpressionAttributeValues': {
        ':threshold': {'N': str(ttl_threshold)}
    }
}


response = dynamodb.query(**query_params)

count += response['Count']

while 'LastEvaluatedKey' in response:
    query_params['ExclusiveStartKey'] = response['LastEvaluatedKey']
    response = dynamodb.query(**query_params)
    count += response['Count']

print(f"Total count of items with TTL greater than {ttl_threshold}: {count}")

Below is the Error:

ClientError                               Traceback (most recent call last)
<ipython- in <module>
     19 
     20 
---> 21 response = dynamodb.query(**query_params)
     22 
     23 count += response['Count']

~/anaconda3/python3.7/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    513                 )
    514             # The "self" in this scope is referring to the BaseClient.
--> 515             return self._make_api_call(operation_name, kwargs)
    516 
    517         _api_call.__name__ = str(py_operation_name)

~/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    932             error_code = parsed_response.get("Error", {}).get("Code")
    933             error_class = self.exceptions.from_code(error_code)
--> 934             raise error_class(parsed_response, operation_name)
    935         else:
    936             return parsed_response

ClientError: An error occurred (ValidationException) when calling the Query operation: Query condition missed key schema element: customer_id
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Thank you @LeeroyHannigan and @Nadav Har'El based on your comments. I have updated the code. And below code is giving desired output. Thank you again :)

    import boto3
from botocore.exceptions import ClientError

# Initialize the DynamoDB client
session = boto3.session.Session()
client = session.client('dynamodb')

# Name of your DynamoDB table
table_name = 'table_name'

# TTL threshold (in seconds since epoch)
ttl_threshold = 1694510790

try:
    total_count = 0
    paginator = client.get_paginator('scan')
    
    while True:
        response_iterator = paginator.paginate(
            TableName=table_name,
            FilterExpression='#ttl > :threshold',  # Using an alias for the reserved attribute name
            ExpressionAttributeNames={'#ttl': 'ttl'},  # Alias for the reserved keyword 'ttl'
            ExpressionAttributeValues={':threshold': {'N': str(ttl_threshold)}}
        )

        for response in response_iterator:
            items = response['Items']
            # Count only items where ttl is greater than the threshold
            filtered_items = [item for item in items if int(item['ttl']['N']) > ttl_threshold]
            total_count += len(filtered_items)

        # Check if there are more pages to scan
        if 'LastEvaluatedKey' not in response:
            break

    print(f"Total count of items with TTL > {ttl_threshold}: {total_count}")

except ClientError as error:
    print(f"Error: {error.response['Error']['Message']}")

  2. 0

    You cannot use Query as that relies on the partition key being the value you are searching on, which it is not. You will need to use Scan API to read and filter all of the items, you may also need to introduce pagination if you have more than 1MB of data in the table.

    import boto3
from boto3.dynamodb.conditions import Attr
from botocore.exceptions import ClientError

session = boto3.session.Session()
client = session.client('dynamodb')

table_name = 'table name'
ttl_threshold = 1631303477 


try:
    response = client.scan(
        TableName=table_name, 
        FilterExpression = Attr('Number_Attribute').gte(ttl_threshold)
    )

    print(len(response['Items']))

except ClientError as error:
    print(error.response['ResponseMetadata'])
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search