Question 375448 in Javascript Question posted in
A very good W3school tutorial can be found here.

Javascript – Do callbacks from I/O operations that return promises go to the I/O queue or the microtask queue in Node?

I’m just learning the details of how the event loop in Node works.

I learned that the promise queue takes priority over the timer queue, which takes priority over the I/O queue.

async function asyncFunc() {
  let sum = 0;
  for (let i = 0; i < 100000; i++) {
    sum += i;
  }
}
async function main() {
  let start = Date.now();
  let interval = setInterval(() => {
    let time = Date.now() - start;
    console.log(time);
  }, 1000);
  while (true) {
    await asyncFunc();
  }
}
main();

This code never logs the time because the promise queue never clears.

import fs from "fs-extra";

async function asyncFunc(j) {
  let sum = 0;
  console.log("checkpoint 1");
  await fs.copy("file.txt", `file${j}.txt`, { overwrite: false });
  for (let i = 0; i < 1000000000; i++) {
    sum += i;
  }
  console.log("checkpoint 2");
}

async function main() {
  let start = Date.now();
  let interval = setInterval(() => {
    let time = Date.now() - start;
    console.log(time);
  }, 1000);

  while (true) {
    const promises = [];
    for (let j = 0; j < 4; j++) {
      promises.push(asyncFunc(j));
    }
    await Promise.all(promises);
  }
}

main();

Why does this code output

checkpoint 1
checkpoint 1
checkpoint 1
checkpoint 1
checkpoint 2
checkpoint 2
checkpoint 2
2123
checkpoint 2

?

All of the copies are called on an empty file, before the cpu-bound work begins and they should take a minuscule amount of time.

If the file copy returns a callback to the promise queue when it completes, I think the time should never be logged, since the promise queue will always have a callback waiting by the time cpu-bound work is completed.

If it returns a callback to the I/O queue, I would think the time log would happen after the last checkpoint 1, as the promise queue would be empty and the I/O queue would have four callbacks.

What’s going on here? Is this just a weird race condition?

Tags:

2

Answers


  1. 0

    The reason that the job in the timer queue gets the priority over a job in the I/O queue is explained in the article The Node.js Event Loop (nodejs.org), where the relevant two phases are named "timers" and "poll" (for I/O):

    To prevent the poll phase from starving the event loop, libuv (the C library that implements the Node.js event loop and all of the asynchronous behaviors of the platform) also has a hard maximum (system dependent) before it stops polling for more events.

    And again:

    If the poll queue is not empty, the event loop will iterate through its queue of callbacks executing them synchronously until either the queue has been exhausted, or the system-dependent hard limit is reached.

    This is what apparently happens in your scenario: after three busy loops have executed, the event loop considers that it has been long enough in the poll phase, and goes to the next phases of the loop, abandoning for now the non-empty I/O queue.

    And so the timer callback gets a chance to execute, after which the loop arrives back in the poll phase to continue processing the remaining job in that queue.

    Login or Signup to reply.
  2. 0

    Do callbacks from I/O operations that return promises go to the I/O queue or the microtask queue in Node?

    These callbacks go to the IO queue, from where they’ll resolve the Promise, that will execute your callback (passed in the .then(), or implied by await) which is executed in the promise queue.

    Promises and nextTick callbacks (a.k.a. microtasks) aren’t part of the event-loop’s phases per se. As the docs put it:

    You may have noticed that process.nextTick() was not displayed in the diagram, even though it’s a part of the asynchronous API. This is because process.nextTick() is not technically part of the event loop. Instead, the nextTickQueue will be processed after the current operation is completed, regardless of the current phase of the event loop. Here, an operation is defined as a transition from the underlying C/C++ handler, and handling the JavaScript that needs to be executed.

    I learned that the promise queue takes priority over the timer queue, which takes priority over the I/O queue.

    This isn’t correct. The Promise queue doesn’t contribute to any prioritization system. What happens is that the microtask queue(s) (which comprises both promise jobs and nextTick callbacks) run until it’s empty. And thus if you do queue a new microtask from a microtask, it will get executed before the engine gives back the control to the event-loop.

    Note that this is also true in the HTML’s event-loop, but node has another level of complexity here where each of the nextTick queue and the promise queue will run until empty on their own.
    So, if from a nextTick callback, you do queue another nextTick callback, it will be executed even before a promise job. If you do queue a promise job from another promise job, it will be executed even before a nextTick callback. And if you do queue a nextTick callback from a promise job, it will get executed before the event-loop starts again, and same in the other way.

    So when you await a Promise returned from an IO callback, what you’re actually awaiting for is a (macro)task, that will resolve the promise. The event-loop can thus go to the other phases in between each of these (macro)tasks. (Note that once again, it’s the same in browsers).

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search