Question 211826 in Javascript Question posted in
A very good W3school tutorial can be found here.

Javascript – Nodejs : Should I use worker threads for sending 5000+ http requests?

if I want to send like messages to 5000+ users in nodejs, which takes a long time + I dont want it to be render blocking, is Worker threads what I need?

the docs says:
Workers (threads) are useful for performing CPU-intensive JavaScript operations. They do not help much with I/O-intensive work. The Node.js built-in asynchronous I/O operations are more efficient than Workers can be.

what is the Node.js built-in asynchronous I/O operations it referring to?

I need to loop like 5000+ contacts:

store a new row in database for each contact
send message with axios http request to an external api for each contact

is this considered as an I/O operations?

Tags:

2

Answers


  1. 0

    When using third-party API’s with rate-limiting, I like to pace my async requests to match their rate limits. The number of concurrent requests will vary based on latency, but shouldn’t ever be enough to overtax your resources.

    rateLimiter = {
    lastVisited:{},
    pace:async function(entity, millis) {
            let durationAlreadyWaited=Math.min(new Date() - this.lastVisited[entity], millis);
            await new Promise(r => setTimeout(r, millis-durationAlreadyWaited));
            this.lastVisited[entity]=new Date();
    }
}

const MS_PER_MEMBER=13;
async function processList(myList) {
   for (const member of myList) {
      await rateLimiter.pace('myList',MS_PER_MEMBER);
      APICall(member).then(/* handle it */);
   }
}
    Login or Signup to reply.
  2. 0

    Yes, for your use case, the operations you described (storing a new row in the database and sending an HTTP request to an external API for each contact) are considered I/O operations. I/O stands for Input/Output, and it refers to operations that involve reading from or writing to external resources, such as databases, files, or network requests.

    Node.js is particularly well-suited for I/O-intensive operations due to its non-blocking, asynchronous nature. When Node.js performs an I/O operation (e.g., reading from a file or making an HTTP request), it doesn’t block the entire program’s execution while waiting for the operation to complete. Instead, it continues executing other tasks in parallel, making efficient use of resources.

    The Node.js built-in asynchronous I/O operations include:

    1. File System Operations: Reading and writing files using functions like fs.readFile, fs.writeFile, etc.

    2. Network Operations: Making HTTP requests or other network-related tasks using modules like http, https, net, etc.

    3. Database Operations: Performing queries and CRUD (Create, Read, Update, Delete) operations on databases using modules like mysql, mongodb, etc.

    Now, regarding the use of Worker Threads for your specific scenario:

    Worker Threads: As the documentation states, worker threads are primarily useful for CPU-intensive operations. If your task involves a significant amount of computational work that can be parallelized (like heavy calculations), then using Worker Threads could be beneficial. It allows you to offload CPU-intensive tasks to separate threads, leaving the main thread available for other work.

    However, for I/O-intensive tasks like the ones you mentioned (database operations and making HTTP requests), using Worker Threads might not provide significant benefits. Node.js’s built-in asynchronous I/O model is already designed to handle such tasks efficiently.

    Handling I/O-Intensive Tasks Efficiently: To process 5000+ contacts without blocking the event loop and leveraging Node.js’s strengths, you can use asynchronous programming techniques, such as Promises, async/await, or callbacks. By doing so, you can initiate I/O operations for each contact, and Node.js will handle them efficiently in a non-blocking manner.

    Here’s a high-level example using async/await (requires Node.js 7.6+):

    async function processContacts(contacts) {
  for (const contact of contacts) {
    // Store a new row in the database (using async/await or Promises)
    await db.storeContact(contact);

    // Send HTTP request to an external API (using async/await or Promises)
    await axios.post('external-api-url', contact);
  }
}

// Assuming you have an array of contacts
const contacts = [...]; // Your array of 5000+ contacts
processContacts(contacts)
  .then(() => console.log('All contacts processed successfully!'))
  .catch((error) => console.error('Error processing contacts:', error));

    By using asynchronous I/O operations with async/await or Promises, Node.js will efficiently handle the database and HTTP operations in a non-blocking way, making sure your application remains responsive and performant.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search