Question 18873 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Single Table Design – DynamoDB – Merge two tables into one? – Amazon Web Sevices

I’ve watched this video and I was thinking about the Single Table Design. Is it a good idea to merge these two tables into one, just like the example in the video? Since they are related to each other (user trades and user transfers i.e. deposits and withdrawals) and the access patterns are all about given an exchange name and an account name.

enter image description here

The access patterns I’m interested in right now:

  • Get the last user transfer for a given exchange and an account name.
  • Get all user trades by date range.
  • Get all user transfers by date range.

Is it a good idea to merge both tables into one with the same PK and SK? There will be items with SK TradeDate#12/17/2022 and others with TransferDate#12/17/2022.

I’m about to figure out how to do the date range queries. I would have to use GSIs.

User Trades

+-------------------------------+----------------------------------------+---------+----------+-----+-------------+------------+-------+-------------+----------+-------------------------------+-----------+-----------------------------------+
|              PK               |                   SK                   | Account | Exchange | Fee | FeeCurrency | Instrument | Price | ProductType | Quantity |           TradeDate           |  TradeId  |             UpdatedAt             |
+-------------------------------+----------------------------------------+---------+----------+-----+-------------+------------+-------+-------------+----------+-------------------------------+-----------+-----------------------------------+
| Exchange#Binance#Account#main | TradeDate#12/17/2022 4:59:12 PM +02:00 | main    | Binance  |   0 | BNB         | BTCBUSD    |     0 | Spot        |  0.00086 | 2023-01-26T23:17:56.436+00:00 | 814696766 | 2023-01-26T23:17:57.7786154+00:00 |
| Exchange#Binance#Account#main | TradeDate#12/17/2022 5:38:23 PM +02:00 | main    | Binance  |   0 | BNB         | BTCBUSD    |     0 | Spot        |  0.00086 | 2023-01-26T23:57:06.876+00:00 | 814746356 | 2023-01-26T23:57:08.3696852+00:00 |
| Exchange#FTX#Account#main     | TradeDate#12/17/2021 5:38:23 PM +02:00 | main    | Binance  |   0 | BNB         | BTCBUSD    |     0 | Spot        |  0.00086 | 2023-01-26T23:57:21.226+00:00 | 814746461 | 2023-01-26T23:57:21.8543695+00:00 |
| Exchange#FTX#Account#main     | TradeDate#12/19/2022 4:59:12 PM +02:00 | main    | Binance  |   0 | BNB         | BTCBUSD    |     0 | Spot        |  0.00086 | 2023-01-26T23:57:21.901+00:00 | 814746513 | 2023-01-26T23:57:22.528155+00:00  |
| Exchange#Binance#Account#main | TradeDate#12/17/2022 4:59:12 PM +02:00 | main    | Binance  |   0 | BNB         | BTCBUSD    |     0 | Spot        |  0.00086 | 2023-01-26T23:57:22.348+00:00 | 814746517 | 2023-01-26T23:57:22.9753506+00:00 |
| Exchange#Binance#Account#main | TradeDate#12/17/2022 5:38:23 PM +02:00 | main    | Binance  |   0 | BNB         | BTCBUSD    |     0 | Spot        |  0.00086 | 2023-01-26T23:57:22.802+00:00 | 814746518 | 2023-01-26T23:57:23.429097+00:00  |
| Exchange#FTX#Account#main     | TradeDate#12/17/2021 5:38:23 PM +02:00 | main    | Binance  |   0 | BNB         | BTCBUSD    |     0 | Spot        |  0.00086 | 2023-01-26T23:57:23.252+00:00 | 814746521 | 2023-01-26T23:57:23.8756532+00:00 |
| Exchange#FTX#Account#main     | TradeDate#12/19/2022 4:59:12 PM +02:00 | main    | Binance  |   0 | BNB         | BTCBUSD    |     0 | Spot        |  0.00086 | 2023-01-26T23:57:23.759+00:00 | 814746524 | 2023-01-26T23:57:24.3824745+00:00 |
+-------------------------------+----------------------------------------+---------+----------+-----+-------------+------------+-------+-------------+----------+-------------------------------+-----------+-----------------------------------+

User Transfers (Deposits + Withdrawals)

+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
|              PK               |                    SK                     | Account |                  Address                   | AddressTag | Exchange | Instrument | Network | Quantity  | TransactionFee |                           TransactionId                            |       TransferDate        |            TransferId            | TransferType |          UpdatedAt           |
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+
| Exchange#Binance#Account#main | TransferDate#12/17/2022 4:59:12 PM +02:00 | main    | 0xF76d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null       | Binance  | USDT       | ETH     | 97.500139 |            3.2 | 0x46d28f7d0e1e5b1d074a65dcfbb9d90b3bcdc7e6fca6b1f1f7abb5ab219feb24 | 2022-12-17T16:59:12+02:00 | 1b56485f6a3446c3b883f4f485039260 |            0 | 2023-01-28T20:19:59.9181573Z |
| Exchange#Binance#Account#main | TransferDate#12/17/2022 5:38:23 PM +02:00 | main    | 0xF76d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null       | Binance  | USDT       | ETH     | 3107.4889 |            3.2 | 0xbb2b92030b988a0184ba02e2e754b7a7f0f963c496c4e3473509c6fe6b54a41d | 2022-12-17T17:38:23+02:00 | 4747f6ecc74f4dd8a4b565e0f15bcf79 |            0 | 2023-01-28T20:20:00.4536839Z |
| Exchange#FTX#Account#main     | TransferDate#12/17/2021 5:38:23 PM +02:00 | main    | 0x476d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null       | FTX      | USDT       | ETH     |        20 |            3.2 | 0xaa2b92030b988a0184ba02e2e754b7a7f0f963c496c4e3473509c6fe6b54a41d | 2021-12-17T17:38:23+02:00 | 4747f6ecc74f4dd8a4b565e0f15bcf79 |            0 | 2023-01-28T20:20:00.5723855Z |
| Exchange#FTX#Account#main     | TransferDate#12/19/2022 4:59:12 PM +02:00 | main    | 0xc46d3f20bF155681b0b983bFC3ea5fe43A2A6E3c | null       | FTX      | USDT       | ETH     |        15 |            3.2 | 0xddd28f7d0e1e5b1d074a65dcfbb9d90b3bcdc7e6fca6b1f1f7abb5ab219feb24 | 2022-12-19T16:59:12+02:00 | 1b56485f6a3446c3b883f4f485039260 |            0 | 2023-01-28T20:20:00.5207119Z |
+-------------------------------+-------------------------------------------+---------+--------------------------------------------+------------+----------+------------+---------+-----------+----------------+--------------------------------------------------------------------+---------------------------+----------------------------------+--------------+------------------------------+

public async Task<UserTransferDto?> GetLastAsync(string exchange, string account)
{
    var queryRequest = new QueryRequest
    {
        TableName = TableName,
        KeyConditionExpression = "#pk = :pk",
        ExpressionAttributeNames = new Dictionary<string, string>
        {
            { "#pk", "PK" }
        },
        ExpressionAttributeValues = new Dictionary<string, AttributeValue>
        {
            { ":pk", new AttributeValue { S = $"Exchange#{exchange}#Account#{account}" } }
        },
        ScanIndexForward = false,
        Limit = 1
    };

    var response = await _dynamoDb.QueryAsync(queryRequest);
    if (response.Items.Count == 0)
    {
        return null;
    }

    var itemAsDocument = Document.FromAttributeMap(response.Items[0]);
    return JsonSerializer.Deserialize<UserTransferDto>(itemAsDocument.ToJson());;
}

A little edit:

I realized I needed the transfer type too, so I changed SK to TransferType#Withdraw#TransferDate#2022-12-17 14:59:12

and now the code looks like:

public async Task<UserTransferDto?> GetLastAsync(string exchange, string account, TransferType transferType)
{
    var queryRequest = new QueryRequest
    {
        TableName = TableName,
        KeyConditionExpression = "#pk = :pk and begins_with(#sk, :sk)",
        ExpressionAttributeNames = new Dictionary<string, string> { { "#pk", "PK" }, { "#sk", "SK" } },
        ExpressionAttributeValues = new Dictionary<string, AttributeValue>
        {
            { ":pk", new AttributeValue { S = $"Exchange#{exchange}#Account#{account}" } },
            { ":sk", new AttributeValue { S = $"TransferType#{transferType}" } }
        },
        ScanIndexForward = false,
        Limit = 1
    };

    var response = await _dynamoDb.QueryAsync(queryRequest);
    if (response.Items.Count == 0)
    {
        return null;
    }

    var itemAsDocument = Document.FromAttributeMap(response.Items[0]);
    return JsonSerializer.Deserialize<UserTransferDto>(itemAsDocument.ToJson());
}
Tags:

2

Answers


  1. 0

    My personal attitude – not to change anything if:

    1. It works right now
    2. I don’t see any future risks on my system
    3. There is no benefits on performance/durability/maintenance/ease of support

    And coming to your question I don’t see how merging two tables can help you.

    I’d challenge that your table structure won’t help you to cover these queries:

    • Get the last user transfer for a given exchange and an account name.
    • Get all user trades by date range.
    • Get all user transfers by date range.

    If I understand correctly you don’t need to query anything by your current PK => why did you choose this PK? Of course you can solve it by using GSI but you won’t get them for free and if you can restructurise your table to make your 3 queries easier I’d do that.

    Hope I answered your question. Good luck, have fun)

    Login or Signup to reply.
  2. 0

    A query for the newest/latest item for a given partition key (pk) is possible if the sort key (sk) has a lexicographically sortable date/time representation e.g. ISO 8601. This is possible because items with the same pk value are stored in sorted order by sk.

    To do this, simply issue a query for the pk + sk prefix, and supply Limit=1 and ScanIndexForward=False. For example:

    KeyConditionExpression = "pk=:pk and begins_with(sk, :trade_prefix)"
Limit=1
ScanIndexForward=False

    You can query for items with a given partition key (pk) and a sort key (sk) within a given range of dates using a key condition expression on the query which looks like a BETWEEN b and c.

    For example:

    KeyConditionExpression = "pk=:pk and sk between :t1 and :t2"

    On the question of using DynamoDB as a simple key/value store where the value is a JSON blob, it depends on your access requirements. Some reasons not to have single JSON blob are that:

    1. any update to the item means you have to rewrite the whole item which costs more WCUs
    2. you can’t create GSIs on most of the item’s attributes because they’re not top-level
    3. it can makes concurrent puts more challenging
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search