Question 256274 in Javascript Question posted in
A very good W3school tutorial can be found here.

Javascript – Error handling with Transfer-Encoding chunked http requests with express and axios

Im trying to fetch a big amount of data from a db in JS using streaming to prevent loading all the data into memory at once. Im using express as my server and a nodeJS client that fetches the data using Axios. I’ve succesfully fetched the data with streaming but couldn’t figure out how to handle errors that happen while streaming.

Express Server:

app.get('/stream', async (req, res) => {
    try {
      const cursor = //fetch data with a limit & skip (MongoDB)
      while(cursor.hasNext()) {
        const data = await cursor.next()

        const writeToStream = new Promise((resolve) => {
          res.write(data, 'utf8', () => {
            console.log("batch sent");
            resolve()
          })
  
        })
        await writeToStream
      }
      res.end()
    } catch(error) {
      console.log(`error: ${error.message}`)
      
      //How do i send the status & error message to the requestor? 
      //return res.status(400).end(error.message) // <-- wanted behavior
  })

Client:

    try {
      const res = await axios({
        url: 'http://localhost:3000/test',
        responseType: 'stream'
      })

      const { data } = res
      data.pipe(someStream)

      data.on('end', () => { 
        //stream finished
      })

      data.on('error', (error) => { // First Option
        //error without a status or message
        //res.status(error.status).send(error.message) // <-- wanted behavior  
      })
    } catch(error) {  // Second Option
      //error without a status or message
      //return res.status(error.status).send(error.message) // <-- wanted behavior
    }

The error handeling on the client works (The code runs) but i couldn’t figure out how to send a status & message from the server to the client indicating an error and specifying it.

Versions:
"axios": "^1.5.1", "express": "^4.18.2"

Would appreciate some help.
Thanks ahead of time!

Tags:

2

Answers


  1. 0

    The thing is, you can not set headers after they are sent to the client. So when you start a stream using res.write the header is already sent as 200.

    What you can do is to use a trick. You can set a fixed prefix for both DATA and ERROR. In this way you can distinguish between what is real data and what is real error. I know its not the most efficient way, but works and seems plausible as the streaming stack does not provide the error management mechanism by itself at all.

    server.ts:

    import express from 'express'

const app = express()
const port = 3000

const DATA_PREFIX = "DATA:"
const ERROR_PREFIX = "ERROR:"

app.get('/stream', async (req, res) => {
  try {
    // I have replaced the query to MongoDB with some sample data.
    const sample_datas = ["data1", "data2", "data3", "data4", "data5"]

    for (let dt of sample_datas) {
      const writeToStream = new Promise((resolve) => {

        // if dt == "data4", then simulate an error
        if (dt == "data4") {
          throw new Error("data4 has problems.")
        }

        res.write(DATA_PREFIX + dt, 'utf8', () => {
          console.log("batch sent");
          resolve(true)
        })
      })
      await writeToStream
    }

    res.end()
  } catch (error) {
    console.log(`error: ${error.message}`)

    //How do i send the status & error message to the requestor?
    res.write(ERROR_PREFIX + error.message)
    res.end()
  }
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

    client.ts

    import axios from 'axios'
import fs from 'fs'
import stream from 'stream'

const DATA_PREFIX = "DATA:"
const ERROR_PREFIX = "ERROR:"

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

const main = async () => {

  await sleep(2000)

  try {
    const res = await axios({
      url: 'http://localhost:3000/stream',
      responseType: 'stream'
    })

    const { data } = res

    const writableStream = fs.createWriteStream('__output__.txt');

    const appendText = new stream.Transform({
      transform(chunk: Buffer, encoding, callback) {

        const chunk_str = chunk.toString('utf8')

        if (chunk_str.startsWith(DATA_PREFIX)) {
          this.push(Buffer.from(chunk_str.replace(DATA_PREFIX, ""), 'utf8'))
        }
        else if (chunk_str.startsWith(ERROR_PREFIX)) {
          const error = chunk_str.replace(ERROR_PREFIX, "")
          error_function(new Error(error))
        }

        callback();
      }
    });

    data.pipe(appendText).pipe(writableStream);

    data.on('end', () => {
      console.log("stream finished")
    })

    const error_function = (error: Error) => {
      console.log("data.on error:", error)
    }

    data.on('error', error_function)

  } catch (error) {
    console.log("catch error:", error)
  }
}

main()
    Login or Signup to reply.
  2. 0

    Express Server:
    In your server code, you want to make sure that if something goes wrong while sending data, you tell the client about it clearly. Here’s how:

    1. We’re using a method called cursor.forEach() to send data to the client as we get it from the database.
    2. If something bad happens while sending data, we’ll let the client know by saying, "Oops, something went wrong!" and we’ll also tell them it’s our fault (HTTP status code 500).
    3. If something goes wrong even before we start sending data, we’ll still tell the client the same thing.

    Client Side:
    On the client’s side, you’re doing well with the error part, but let’s break it down:

    1. When the server sends data, we’re catching it and putting it somewhere, like saving it to a file.
    2. We’re checking to see when all the data is sent (like finishing a download).
    3. If there’s a problem while getting data, we’ll say, "Oops, something’s wrong!" and log what happened. We can also handle the problem, like stopping the download.

    By making these changes, you’ll be able to handle problems while sending data, and you’ll tell the client about it in a simple and clear way.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search