skip to Main Content

Trying to write a query from the following table that calculates a count of records for every day date(timestamp) and the prior 4 days. So basically a rolling count of records for every previous 5 days. Every time i do the calculation returns slightly off.

Symbol timestamp high number date(timestamp) date(timestamp – interval 1 day) date(timestamp – interval 2 day) date(timestamp – interval 3 day) date(timestamp – interval 4 day)
SPY 2021-04-26 04:00:00+00:00 416.97 1 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22
SPY 2021-04-26 06:20:00+00:00 416.91 2 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22
SPY 2021-04-26 08:00:00+00:00 416.84 3 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22
SPY 2021-04-26 08:05:00+00:00 416.8 4 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22
SPY 2021-04-26 08:10:00+00:00 416.81 5 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22
SPY 2021-04-26 08:15:00+00:00 416.78 6 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22
SPY 2021-04-26 08:20:00+00:00 416.75 7 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22
SPY 2021-04-26 08:25:00+00:00 416.54 8 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22
SPY 2021-04-26 08:30:00+00:00 416.51 9 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22
SPY 2021-04-26 08:35:00+00:00 416.34 10 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22
SPY 2021-04-26 08:40:00+00:00 416.33 11 2021-04-26 2021-04-25 2021-04-24 2021-04-23 2021-04-22

The following query returns the counts but they are slightly off. For example of 2021-4-30 the count should match the max(number), the enumerated rows but it does not. Please help

select date(t.timestamp),date(t.timestamp - interval 4 day), 
(count(t.timestamp)+count(t.timestamp - interval 4 day)+count(t.timestamp - interval 3 day)+count(t.timestamp - interval 2 day)+count(t.timestamp - interval 1 day)) as cntrecords, 
max(number), min(number)
from
(SELECT Symbol,
timestamp, 
high, 
ROW_NUMBER() OVER (ORDER BY timestamp) AS number,
date(timestamp),
date(timestamp - interval 4 day)
#BETWEEN DATE_SUB(date(timestamp), INTERVAL 4 DAY) AND date(timestamp)
FROM test.rawdata 
#WHERE date(timestamp) BETWEEN DATE_SUB('2021-04-30', INTERVAL 4 DAY) AND '2021-04-30'
#group by date(timestamp)
order by timestamp) as t
group by date(t.timestamp);

2

Answers


  1. We will start by getting the counts per symbol per day with a simple GROUPed query. If you want the results for the week commencing 26th April 2021 you need to remember to include the four days prior to the start of the desired date range –

    SELECT `Symbol`, DATE(`timestamp`) `date`, COUNT(*) `count`
    FROM `test`.`rawdata`
    WHERE `timestamp` BETWEEN '2021-04-22 00:00:00' AND '2021-05-02 23:59:59'
    GROUP BY `Symbol`, `date`;
    

    Now we can use SUM as a window function with a 5 day frame to get your counts –

    SELECT *, SUM(`count`) OVER (
        PARTITION BY `Symbol`
        ORDER BY `date` ASC
        RANGE INTERVAL 4 DAY PRECEDING
    ) `5_day_count`
    FROM (
        SELECT `Symbol`, DATE(`timestamp`) `date`, COUNT(*) `count`
        FROM `test`.`rawdata`
        WHERE `timestamp` BETWEEN '2021-04-22 00:00:00' AND '2021-05-02 23:59:59'
        GROUP BY `Symbol`, `date`
    ) tbl;
    

    The above query will return the leading four rows which we can remove by adding another level of nesting (we cannot use a HAVING clause as the filter would be applied before the SELECT list is evaluated) –

    SELECT *
    FROM (
        SELECT *, SUM(`count`) OVER (
            PARTITION BY `Symbol`
            ORDER BY `date` ASC
            RANGE INTERVAL 4 DAY PRECEDING
        ) `5_day_count`
        FROM (
            SELECT `Symbol`, DATE(`timestamp`) `date`, COUNT(*) `count`
            FROM `test`.`rawdata`
            WHERE `timestamp` BETWEEN '2021-04-22 00:00:00' AND '2021-05-02 23:59:59'
            GROUP BY `Symbol`, `date`
        ) tbl
    ) t2
    WHERE `date` BETWEEN '2021-04-26' AND '2021-05-02';
    

    As an alternative approach, you could use a recursive cte to build a list of ranges to join and group by –

    WITH RECURSIVE `cte` (`date`, `start`, `end`) AS (
        SELECT
            CAST('2021-04-26' AS DATE),
            CAST('2021-04-22 00:00:00' AS DATETIME),
            CAST('2021-04-26 23:59:59' AS DATETIME)
        UNION ALL
        SELECT
            `date` + INTERVAL 1 DAY,
            `start` + INTERVAL 1 DAY,
            `end` + INTERVAL 1 DAY
        FROM `cte`
        WHERE `date` < '2021-05-02'
    )
    SELECT `rawdata`.`symbol`, `cte`.`date`, COUNT(*) `count`
    FROM `cte`
    JOIN `test`.`rawdata`
        ON `rawdata`.`timestamp` BETWEEN `cte`.`start` AND `cte`.`end`
    GROUP BY `rawdata`.`symbol`, `cte`.`date`;
    
    Login or Signup to reply.
  2. This was not stated in the question, but I’m guessing that columns like number and date(timestamp - interval 3 day) were added to try and help solve this problem, or are results of an intermediate query. Either way, they are unnecessary.

    Let’s solve this problem for a table which only has symbol, timestamp, and high, using a window function with frame specification:

    -- Get the counts for each day (factored out as a CTE)
    with Counts AS (
        SELECT 
          symbol,
          DATE(timestamp) AS day,
          COUNT(*) AS day_count
        FROM 
         prices
        GROUP BY symbol, day
    
    )
    SELECT
        symbol,
        day,
        -- Here we use the window function with a frame spec to sum for the last 4 days per row
        SUM(day_count) OVER (
            PARTITION BY symbol 
            ORDER BY day
            RANGE BETWEEN INTERVAL '4' DAY PRECEDING AND CURRENT ROW
        )
    FROM
        Counts
    

    Note that I’m using RANGE instead of ROW because if there are days for which there are no data, that would throw off the counts.

    Also note that this will not work in MariaDb which does not yet support RANGE expressions with TIME like fields and intervals.

    You can play around with this example in this DB Fiddle:
    https://www.db-fiddle.com/f/ctN927ouAMHrWK1QJQD6Z2/0

    (Note I only used 1 day in the range there so I didn’t have to generate as much test data.)

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search