Question 403431 in Javascript Question posted in
A very good W3school tutorial can be found here.

Javascript – How to split chunks into individual JSON objects

I am fetching from an api which sends data in a stream. Unfortunately when I process the data in chunks, the chunks come as multiple JSON strings like this;

         { 
           "productName": "bag",
           "code": "BGX-112"
         } 
         { 
           "productName": "purse",
           "code": "PUSR-112"
         } 
         etc..

Here is the code that process the data:


            const getProductData = async (url:string) => {
               const decoder = new TextDecoderStream()
               let chunks: any[];
               chunks = [];
               try {
                   const response = await fetch(url)
                   const stream = response?.body.pipeThrough(new TextDecoderStream());
                   const reader = stream.getReader()
                   while (true) {
                       const {value, done} = await reader.read();
                       if (done) {
                           // Flush any buffered characters.
                           const stringArray = value?.split(/bs/);
                           chunks.push(value);
                           return chunks;
                       }
                       if (value) {
                           const stringArray = value?.split(/bs/);
                           chunks.push(value);
                       }
        
                   }
               }catch (e) {
                   console.error("Error ",e)
               }
           }

I want to get the chunks as individual JSON objects so I can parse them, but I don’t know how. How can I achieve this?

Tags:

2

Answers


  1. 0

    To handle streamed JSON data, use TextDecoder to assemble chunks into a string Look for delimiters (e.g., }n{) to identify complete JSON objects split the string at these points and parse each segment with JSON.parse()

    const getProductData = async (url: string) => {
    const chunks: string[] = [];
    try {
        const response = await fetch(url);
        const reader = response.body?.getReader();

        let decoder = new TextDecoder();
        let partialData = '';

        if (!reader) throw new Error("Stream reader not available");

        while (true) {
            const { value, done } = await reader.read();
            if (done) {
                // flush any remaining data in partialData
                if (partialData) {
                    chunks.push(partialData);
                }
                break;
            }

            // decode the chunk and append to partialData
            partialData += decoder.decode(value, { stream: true });

            // process complete json objects
            let endIndex = 0;
            while ((endIndex = partialData.indexOf('}n{')) !== -1) {
                // extract json objects from partialData
                const jsonStr = partialData.slice(0, endIndex + 1);
                partialData = partialData.slice(endIndex + 1);

                chunks.push(...jsonStr.split('n').filter(Boolean));
            }
        }

        // parse json objects
        const jsonObjects = chunks.map(chunk => JSON.parse(chunk));

        return jsonObjects;
    } catch (e) {
        console.error("Error ", e);
        return [];
    }
}
    Login or Signup to reply.
  2. 0

    Even if the application sends chunks that consist of exactly one JSON string each, the stream that comes out of the text decoder may concatenate several of the sent chunks into one chunk that is seen by the reader.

    If the JSON strings were huge, it could happen that even one of them is split into two chunks seen by the reader. The solution below assumes that this is not the case (in other words: the application streams many small JSON strings, not one huge).

    With this assumption, the code below splits each chunk seen by the reader at those positions where a closing brace is followed by (whitespace and) an opening brace. This is sufficient if each JSON string is just a productName/code object, as indicated in your question. If the JSON strings are more complex, the splitting may need to be adapted.

    (async function() {
  const reader = (await fetch("https://httpbin.org/stream/10")).body.pipeThrough(new TextDecoderStream()).getReader();
  while (true) {
    const {
      value,
      done
    } = await reader.read();
    console.log("new chunk read, done =", done);
    for (const v of value.split(/(?<=})s*(?={)/))
      console.log(JSON.parse(v).id);
    if (done) return;
  }
})();

    The 10 chunks of JSON sent by the application give rise to 2 chunks (plus an empty one) seen by the reader.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search