Dynamo DB pagination in .net - Amazon Web Sevices

Brenda
February 6, 2023
175 views
1 vote
2 Answers

We have been using high level API’s to connect to dynamo db and fetching the data. Pagination was not setup before but since we have lot of data now, we want to setup pagination.

var scanConditions = new List<ScanCondition>();
scanConditions.Add(new ScanCondition("PartitionKey", ScanOperator.BeginsWith, "data"));
var files = await Context.ScanAsync(scanConditions).GetRemainingAsync();
return files.Select(i => i.Data).OrderByDescending(i => i.Date).ToList();

I read the aws documetation but did not find any info regarding pagination for high level api. Is pagination available for high level api? If not what options do I have here?

Thanks

Answers

- smac2020
- February 6, 2023 at 6:45 pm
- 0 votes
0
You can use paginator methods using this .NET API

https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/DynamoDBv2/TIDynamoDBv2PaginatorFactory.html

THis is part of the AWS SDK for .NET V3.

Login or Signup to reply.

As per your comment, here is an idea on how an implementation with low-level API would look like.

The idea is to either design the table PKs and SKs in a way that allows you to query with pagination in mind or to add GSI that allows you to query with pagination in mind.

if we assume that page size is fixed (ex: 20 items), we should be able to fetch the next page and/or fetch a specific page by page number; you would do something similar to the following.

Set PK: ITEM#<PageIndex> and SK: ITEM#<ItemIndex>
Keep a reference of ItemsCount (ex: PK: ITEMSREF, SK: ITEMSREF,
Count: <int>)
During Item insertion, you will do two operations

a. Update ITEMSREF by incrementing the Count property by one and set the ReturnValues option to UPDATED_NEW

b. Insert Item by calculating the PK and SK
var calculatedPage = (int)(newIdNum / PARTITION_SIZE); var calculatedItemIndex = (newIdNum - 1) % PARTITION_SIZE;
To fetch a specific page, you should query the table with PK set to ITEM#<PageIndex>

Example Code:

    public async Task AddItem(ItemDM item)
    {
        try
        {
            var uir = new UpdateItemRequest()
            {
                TableName = TABLE_NAME_TEST,
                Key = new Dictionary<string, AttributeValue>()
                {
                    { "PK", new AttributeValue("ITEMSREF") },
                    { "SK", new AttributeValue("ITEMSREF") }
                },
                UpdateExpression = "SET #count = #count + :incr",
                ExpressionAttributeNames = new Dictionary<string, string> { { "#count", $"Count" } },
                ExpressionAttributeValues = new Dictionary<string, AttributeValue> { { ":incr", new AttributeValue() { N = "1" } } },
                ReturnValues = ReturnValue.UPDATED_NEW,
            };

            var uirResponse = await _client.UpdateItemAsync(uir);
            int.TryParse(uirResponse.Attributes[$"Count"].N, out var newIdNum);
            var calculatedPage = (int)(newIdNum / PARTITION_SIZE);
            var calculatedItemIndex = (newIdNum - 1) % PARTITION_SIZE;

            var itemJson = JsonConvert.SerializeObject(item);
            var doc = Document.FromJson(itemJson);
            var itemAttrMap = doc.ToAttributeMap();

            var pir = new PutItemRequest()
            {
                TableName = TABLE_NAME_TEST,
                Item = itemAttrMap,
                ConditionExpression = ATTR_NOT_EXISTS + "(PK) AND " + ATTR_NOT_EXISTS + "(SK)",
            };

            var response = await _client.PutItemAsync(pir);
            if (response.HttpStatusCode != System.Net.HttpStatusCode.OK)
                throw new ApplicationException($"Failed to add new item. ({response.HttpStatusCode})");

        }
        catch (Exception ex)
        {
            Debug.WriteLine(ex);
            throw;
        }
    }

    public Task<List<ItemDM>> GetItems(int pageIndex)
    {
        var keyDict = ItemDM.GenerateItemWithPageQueryKey(pageIndex.ToString());

        var qir = new QueryRequest()
        {
            TableName = TABLE_NAME_TEST,
            KeyConditionExpression = "#pk = :pk",
            ExpressionAttributeNames = new Dictionary<string, string>()
                {
                    { "#pk", keyDict.ElementAt(0).Key }
                },
            ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
                {
                    { ":pk", keyDict.ElementAt(0).Value }
                }
        };

        var response = await _client.QueryAsync(qir);
        if (response.HttpStatusCode != System.Net.HttpStatusCode.OK)
            throw new ApplicationException(response.HttpStatusCode.ToString());
        if (response.Items == null || response.Items.Count <= 0)
            return new List<ItemDM>();

        var items = new List<ItemDM>();
        foreach (var item in response.Items)
        {
            var doc = Document.FromAttributeMap(item);
            var itemJson = doc.ToJson();
            var itemDM = JsonConvert.DeserializeObject<ItemDM>(itemJson);
            items.Add(itemDM);
        }
        return items;
    }

    public static Dictionary<string, AttributeValue> GenerateItemWithPageQueryKey(string pageIndex)
    {
        return new Dictionary<string, AttributeValue>()
        {
            { "PK" , new AttributeValue( $"ITEM#{pageIndex}") }
        };
    }

Finally, to allow for dynamic page sizes, you will have a rule and a decision to make as follows.

Set a rule of allowed page size to must for example be divisible by 5 and minimum page size could be 20 for example
Decide between under-fetch or over-fetch data and based on that decision you set the PARTITION_SIZE to either be the minimum page size (ex: 20) or the maximum page size (ex: 60)
Then you would need to change the GetItems method code a bit as follows. The example is of over-fetching

    private async Task<List<ItemDM>> GetItems(int pageSize, int pageIndex)
        {
            try
            {
                int calculatedPageIndex = (int)((pageSize * pageIndex) / PARTITION_SIZE);
                int calculatedItemStartIndex = (pageIndex) * pageSize % PARTITION_SIZE;

                var keyDict = ItemDM.GenerateItemTypeWithPageQueryKey(calculatedPageIndex.ToString());
                var startkey = calculatedItemStartIndex < pageSize ? null : GenerateStartKey(calculatedPageIndex, calculatedItemStartIndex - 1);

                var gir = new GetItemRequest()
                {
                    TableName = TABLE_NAME_TEST,
                    Key = new Dictionary<string, AttributeValue>()
                    {
                        { "PK", new AttributeValue("ITEMSREF") },
                        { "SK", new AttributeValue("ITEMSREF") },
                    },
                };

                var girResponse = await _client.GetItemAsync(gir);
                if (girResponse.HttpStatusCode != System.Net.HttpStatusCode.OK)
                    throw new ApplicationException(girResponse.HttpStatusCode.ToString());

                int.TryParse(girResponse.Item[$"Count"].N, out var count);

                var qir = new QueryRequest()
                {
                    TableName = TABLE_NAME_TEST,
                    IndexName = GSI1_NAME_TEST,
                    KeyConditionExpression = "#pk = :pk",
                    ExpressionAttributeNames = new Dictionary<string, string>()
                    {
                        { "#pk", keyDict.ElementAt(0).Key }
                    },
                    ExpressionAttributeValues = new Dictionary<string, AttributeValue>()
                    {
                        { ":pk", keyDict.ElementAt(0).Value }
                    },
                    Limit = pageSize,
                };

                if (startkey?.Count > 0)
                    qir.ExclusiveStartKey = startkey;

                var response = await client.QueryAsync(qir);
                if (response.HttpStatusCode != System.Net.HttpStatusCode.OK)
                    throw new ApplicationException(response.HttpStatusCode.ToString());
                if (response.Items == null || response.Items.Count <= 0)
                    return new new List<ItemDM>();

                //ToDo Handle case where the returned items less than the requested pageSize

                var items = new List<ItemDM>();
                foreach (var item in response.Items)
                {
                    var doc = Document.FromAttributeMap(item);
                    var itemJson = doc.ToJson();
                    var itemDM = JsonConvert.DeserializeObject<ItemDM>(itemJson);
                    items.Add(itemDM);
                }

                return items;
            }
            catch (Exception ex)
            {
                Debug.WriteLine(ex);
                throw;
            }
        }


        public static Dictionary<string, AttributeValue> GenerateStartKey(int pageIndex, int itemIndex)
        {
            return new Dictionary<string, AttributeValue>()
            {
                { "GSI1PK" , new AttributeValue( $"ORD#{orderType}#{pageIndex}") },
                { "GSI1SK" , new AttributeValue( $"ORD#{itemIndex.ToString("00")}") }
            };
        }

P.S.:

if you are going to go GSI route, then the ExclusiveStartKey should have all PKs and SKs. In other words, both table PK and SK and GSI PK and SK. So you may need to query the item before the calculated starting index using calculated PageIndex and ItemIndex to get the PK and SK.
You should do fault handling as you see fit, don’t copy what I included in the psuedo-code

Please signup or login to give your own answer.

Click here to cancel reply.

Dynamo DB pagination in .net – Amazon Web Sevices

Answers