Question 221433 in Javascript Question posted in
A very good W3school tutorial can be found here.

Javascript – JS generator function with preload feature

I use a generator function to consume logs and serve it piece by piece. Furthermore the function preload the next logs once the queue is empty. I now like to extend the preload feature so that the preload is triggered once the log array falls below a threshold of 500 entries. I don’t came up with a good solution that’s why I’m asking for help!


async function* getSwaps(project, fromBlock, toBlock, limit) {
    const startBlock = fromBlock;
    const endBlock = toBlock;
    
    let logs = [];
    let queue = [];
    do {
        if (!queue.length && toBlock < endBlock) {
            if (endBlock > fromBlock + limit) {
             toBlock = fromBlock + limit;
          } else {
             toBlock = endBlock;
          }
            const promise = project.getSwaps(fromBlock, toBlock);
            queue.push(promise);
            fromBlock += limit + 1;
        }
        if (!logs.length && queue.length > 0) {
            logs = await queue.shift();
        }
        
        yield logs.shift();
    } while(logs.length > 0 || fromBlock < endBlock);
}
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Made some small adjustments as it's possible that a block range doesn't return data

    async function* getSwaps(pair, fromBlock, toBlock, limit, threshold) { 
   let promise, queue = [];
   do {
      // need more
       if (!promise && queue.length <= threshold && fromBlock <= toBlock) {
           const endBlock = Math.min(fromBlock + limit - 1, toBlock);
           promise = pair.getSwaps(fromBlock, endBlock);
           fromBlock = endBlock + 1;
           promise.then(data => { // got more
            queue.push(...data);
               promise = null;
           });
       }
       // Nothing in the queue? Wait for promise to populate it
       if (!queue.length) await promise;
       // Still nothing? Skip step
       if (queue.length) yield queue.shift();
   } while(fromBlock <= toBlock || queue.length > 0)

  2. 0

    You don’t really need queue. It is enough to know there is an unresolved promise. Also, I would expect promise to resolve to an array of data (for blocks fromBlock to toBlock), so you shouldn’t do queue.push(promise), but push the values from that promised array (individually) to your array (but that could be logs).

    I think you need two concepts of limits:

    • loadSize: An indication how many items you will request per call of project.getSwaps
    • lowThreshold: if the size of logs drops below this level, a next batch should be requested with a call to project.getSwaps, unless all blocks were already requested.

    Here is what you could do:

    async function* getSwaps(project, fromBlock, toBlock, lowThreshold, loadSize) {
    const logs = [];
    let promise;
    while (true) {
        if (!promise && logs.length <= lowThreshold && fromBlock <= toBlock) { // need more
            const doneBlock = Math.min(fromBlock + loadSize - 1, toBlock)
            promise = project.getSwaps(fromBlock, doneBlock);
            fromBlock = doneBlock + 1;
            promise.then(data => { // got more
                logs.push(...data);
                promise = null;
            });
        }
        // Nothing in the logs? Wait for promise to populate it
        if (!logs.length) await promise;
        // Still nothing in the logs? Then we're done
        if (!logs.length) return;
        yield logs.shift();
    }
}
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search