Amazon Web Services Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – Deleting records from a DynamoDB table where the TTL value has been set incorrectly

I’ve inherited a service where the TTL value for the DynamoDB records is in the following format

2022-11-03T10:35:42.920Z

Does anyone know how I can run a query to delete all the records with a TTL value that is in the above format rather than Unix epoch time format?

Tags:

2

Answers


  1. 0

    The only way to retrieve and delete all items with a wrongly formatted TTL field is to perform a table scan, because there is no way to query for only those items. You can then either programmatically process all items in a programming language of your choice or use a filter expression that utilizes the begins_with function, to reduce the amount of data returned. Keep in mind that this will not reduce the necessary read capacity, only the bytes sent over the network. If you only have a few years in your TTL column, you can run multiple queries and adjust the corresponding year in the begins_with function.

    The following query would give you all items with a ISO formatted TTL field for 2022:

    aws dynamodb scan 
    --table-name YourTableName 
    --filter-expression "begins_with(#ttl, :datePrefix)" 
    --expression-attribute-names '{"#ttl": "TTL"}' 
    --expression-attribute-values '{":datePrefix": {"S": "2022-"}}' 
    --output json
    Login or Signup to reply.
  2. 0

    If you want to find out which items are requiring deletion you can return items which have TTL set as a String:

    aws dynamodb scan 
    --table-name MyTable 
    --filter-expression "attribute_type(#ttl, :string)" 
    --expression-attribute-names '{"#ttl": "TTL"}' 
    --expression-attribute-values '{":string": {"S":"S"}}'

    Now, you must update those values for each item returned. If its a small dataset then you can easily iterate using the CLI and update each item. This would look something like this:

    TABLE_NAME="MyTable"

ITEMS=$(aws dynamodb scan 
    --table-name $TABLE_NAME 
    --filter-expression "attribute_type(ttl, :type)" 
    --expression-attribute-values '{":type":{"S":"S"}}' 
    --query "Items" 
    --output json)

echo $ITEMS | jq -c '.[]' | while read item; do
    # Extract the primary key(s) of the item
    KEY=$(echo $item | jq '{YourPrimaryKeyAttribute: .YourPrimaryKeyAttribute}')

    aws dynamodb update-item 
        --table-name $TABLE_NAME 
        --key "$KEY" 
        --update-expression "SET #ttl = :ttl" 
        --expression-attribute-names '{"#ttl":"ttl"}' 
        --expression-attribute-values '{":ttl":{"N":"new_ttl_value"}}'
done

    There is also alternatives for larger data sets, such as Step Functions Distributed Map.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search