Question posted in Json
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Converting huge JSON object to Blob by stringifying "directly into" an ArrayBuffer/Blob to avoid maximum string length error

Joe
June 13, 2023
129 views
1 vote
2 Answers

Context:

I have some code like this in my application:

let blob = new Blob([JSON.stringify(json)], {type: "application/json"});

However, it sometimes fails because in Chrome the maximum string length is ~500MB, and json can sometimes be larger than this.

Question:

I’m looking for a way to go straight from my json variable (i.e. a POJO) to a Blob, probably via some sort of streaming stringification that saves to an ArrayBuffer as it goes. Or any other way to get a large json object into a Blob without running into a ‘maximum string length’ error.

Notes:

Solution must work in the browser.
If an existing library is proposed in an answer, it must not be one that expects the json to simply be an array, since that case is very easy to handle. It must instead expect an arbitrarily nested JSON object where e.g. 90% of the data could be in foo.bar.whatever rather than spread evenly over the top-level keys, or whatever.
I am not looking for a solution that expects a stream as an input, and results in a stream of string chunks as the output, like json-stream-stringify or streaming-json-stringify, for example. Instead, I’d like to input an already-in-memory POJO, and get a Blob out which contains the stringified JSON.

Related:

How to use JSONStream to stringify a large object – OP seems to have a similar problem, but is specifically asking about JSONStream, which is for Node.js rather than the browser, and I think the solution given just saves to a file key-by-key, rather than in a "fully nested" manner? If there’s a way to get this working in the browser in a way that results in an ArrayBuffer that contains the larger-than-maximum-string-length JSON string for arbitrarily nested, objects, then that would definitely qualify as an answer.
How to use streams to JSON stringify large nested objects in Node.js? – same as above.

Answers

Chosen as BEST ANSWER

A possible solution here (note: written via some back and forth with ChatGPT-4), and I've yet to rigorously test, but seems to make sense and hold up in my testing so far.

function jsonToBlob(json) {
  const textEncoder = new TextEncoder();
  const seen = new WeakSet();

  function processValue(value) {
    if(seen.has(value)) {
      throw new TypeError("Converting circular structure to JSON");
    }

    if(value && typeof value.toJSON === "function") {
      value = value.toJSON();
    }

    if(typeof value === 'object' && value !== null) {
      seen.add(value);

      const blobParts = [];
      const entries = Array.isArray(value) ? value : Object.entries(value);
      for(let i = 0; i < entries.length; i++) {
        if(Array.isArray(value)) {
          if(typeof entries[i] !== "function") {
            blobParts.push(processValue(typeof entries[i] !== "undefined" ? entries[i] : null));
          }
        } else {
          const [key, val] = entries[i];
          if(typeof val !== "undefined" && typeof val !== "function") {
            blobParts.push(textEncoder.encode(JSON.stringify(key) + ':'), processValue(val));
          }
        }
        if(i !== entries.length - 1) blobParts.push(textEncoder.encode(','));
      }

      const startBracket = Array.isArray(value) ? '[' : '{';
      const endBracket = Array.isArray(value) ? ']' : '}';
      return new Blob([textEncoder.encode(startBracket), ...blobParts, textEncoder.encode(endBracket)]);
    } else {
      // For primitives we just convert it to string and encode
      return textEncoder.encode(JSON.stringify(typeof value !== "function" ? value : null));
    }
  }

  return processValue(json);
}

✅ Test 1:

let blob = jsonToBlob([{hello:{foo:[1,2,3], a:1, bar:["a", 2, {$hi:[1,2,3, {a:3}]}]}}, 4, new Date()]);
console.log(JSON.parse(await blob.text()));

✅ Test 2:

let json = {};
for(let i = 0; i < 600000; i++) {
  json[Math.random()] = Math.random().toString().repeat(100);
}
let blob = jsonToBlob(json);
console.log(blob); // ~1 GB

Will update this answer if I find any errors/problems when this gets to production.

(Edit)

We can actually workaround that limitation by generating the Blob through chunks of strings.

const header = 24;
const bytes = new Uint8Array((512 * 1024 * 1024) - header);
const bigStr = new TextDecoder().decode(bytes);
const arr = [];
for (let i=0; i<5; i++) {
  arr.push(bigStr);
}
console.log(new Blob(arr).size); // 2.7GB

Given that the Blob constructor also accepts other Blobs in its blobParts input, we can even reuse a simple recursive stringifier and replace all the parts that would join() the inner temporary list of values to instead produce a list of Blob objects, interleaved with the DOMString separator.

So at the end we produce something like

new Blob(["{", <Blob>, ":", <Blob>, "}"]);

And we’re safer from the 500MiB limit.

Here I quickly patched this implementation, but I didn’t perform any serious testing against it, so you might want to do it again yourself:

/*
    json2.js
    2015-05-03

    Public Domain.

    NO WARRANTY EXPRESSED OR IMPLIED. USE AT YOUR OWN RISK.

    See http://www.JSON.org/js.html


    This code should be minified before deployment.
    See http://javascript.crockford.com/jsmin.html

    USE YOUR OWN COPY. IT IS EXTREMELY UNWISE TO LOAD CODE FROM SERVERS YOU DO
    NOT CONTROL.


    This file creates a global JSON object containing two methods: stringify
    and parse. This file is provides the ES5 JSON capability to ES3 systems.
    If a project might run on IE8 or earlier, then this file should be included.
    This file does nothing on ES5 systems.

        JSON.stringify(value, replacer, space)
            value       any JavaScript value, usually an object or array.

            replacer    an optional parameter that determines how object
                        values are stringified for objects. It can be a
                        function or an array of strings.

            space       an optional parameter that specifies the indentation
                        of nested structures. If it is omitted, the text will
                        be packed without extra whitespace. If it is a number,
                        it will specify the number of spaces to indent at each
                        level. If it is a string (such as 't' or '&nbsp;'),
                        it contains the characters used to indent at each level.

            This method produces a JSON text from a JavaScript value.

            When an object value is found, if the object contains a toJSON
            method, its toJSON method will be called and the result will be
            stringified. A toJSON method does not serialize: it returns the
            value represented by the name/value pair that should be serialized,
            or undefined if nothing should be serialized. The toJSON method
            will be passed the key associated with the value, and this will be
            bound to the value

            For example, this would serialize Dates as ISO strings.

                Date.prototype.toJSON = function (key) {
                    function f(n) {
                        // Format integers to have at least two digits.
                        return n < 10 
                            ? '0' + n 
                            : n;
                    }

                    return this.getUTCFullYear()   + '-' +
                         f(this.getUTCMonth() + 1) + '-' +
                         f(this.getUTCDate())      + 'T' +
                         f(this.getUTCHours())     + ':' +
                         f(this.getUTCMinutes())   + ':' +
                         f(this.getUTCSeconds())   + 'Z';
                };

            You can provide an optional replacer method. It will be passed the
            key and value of each member, with this bound to the containing
            object. The value that is returned from your method will be
            serialized. If your method returns undefined, then the member will
            be excluded from the serialization.

            If the replacer parameter is an array of strings, then it will be
            used to select the members to be serialized. It filters the results
            such that only members with keys listed in the replacer array are
            stringified.

            Values that do not have JSON representations, such as undefined or
            functions, will not be serialized. Such values in objects will be
            dropped; in arrays they will be replaced with null. You can use
            a replacer function to replace those with JSON values.
            JSON.stringify(undefined) returns undefined.

            The optional space parameter produces a stringification of the
            value that is filled with line breaks and indentation to make it
            easier to read.

            If the space parameter is a non-empty string, then that string will
            be used for indentation. If the space parameter is a number, then
            the indentation will be that many spaces.

            Example:

            text = JSON.stringify(['e', {pluribus: 'unum'}]);
            // text is '["e",{"pluribus":"unum"}]'


            text = JSON.stringify(['e', {pluribus: 'unum'}], null, 't');
            // text is '[nt"e",nt{ntt"pluribus": "unum"nt}n]'

            text = JSON.stringify([new Date()], function (key, value) {
                return this[key] instanceof Date 
                    ? 'Date(' + this[key] + ')' 
                    : value;
            });
            // text is '["Date(---current time---)"]'


        JSON.parse(text, reviver)
            This method parses a JSON text to produce an object or array.
            It can throw a SyntaxError exception.

            The optional reviver parameter is a function that can filter and
            transform the results. It receives each of the keys and values,
            and its return value is used instead of the original value.
            If it returns what it received, then the structure is not modified.
            If it returns undefined then the member is deleted.

            Example:

            // Parse the text. Values that look like ISO date strings will
            // be converted to Date objects.

            myData = JSON.parse(text, function (key, value) {
                var a;
                if (typeof value === 'string') {
                    a =
/^(d{4})-(d{2})-(d{2})T(d{2}):(d{2}):(d{2}(?:.d*)?)Z$/.exec(value);
                    if (a) {
                        return new Date(Date.UTC(+a[1], +a[2] - 1, +a[3], +a[4],
                            +a[5], +a[6]));
                    }
                }
                return value;
            });

            myData = JSON.parse('["Date(09/09/2001)"]', function (key, value) {
                var d;
                if (typeof value === 'string' &&
                        value.slice(0, 5) === 'Date(' &&
                        value.slice(-1) === ')') {
                    d = new Date(value.slice(5, -1));
                    if (d) {
                        return d;
                    }
                }
                return value;
            });


    This is a reference implementation. You are free to copy, modify, or
    redistribute.
*/

/*jslint 
    eval, for, this 
*/

/*property
    JSON, apply, call, charCodeAt, getUTCDate, getUTCFullYear, getUTCHours,
    getUTCMinutes, getUTCMonth, getUTCSeconds, hasOwnProperty, join,
    lastIndex, length, parse, prototype, push, replace, slice, stringify,
    test, toJSON, toString, valueOf
*/


// Create a JSON object only if one does not already exist. We create the
// methods in a closure to avoid creating global variables.

if (typeof JSON !== 'object') {
    JSON = {};
}

(function () {
    'use strict';
    
    var rx_one = /^[],:{}s]*$/,
        rx_two = /\(?:["\/bfnrt]|u[0-9a-fA-F]{4})/g,
        rx_three = /"[^"\nr]*"|true|false|null|-?d+(?:.d*)?(?:[eE][+-]?d+)?/g,
        rx_four = /(?:^|:|,)(?:s*[)+/g,
        rx_escapable = /[\"u0000-u001fu007f-u009fu00adu0600-u0604u070fu17b4u17b5u200c-u200fu2028-u202fu2060-u206fufeffufff0-uffff]/g,
        rx_dangerous = /[u0000u00adu0600-u0604u070fu17b4u17b5u200c-u200fu2028-u202fu2060-u206fufeffufff0-uffff]/g;

    function f(n) {
        // Format integers to have at least two digits.
        return n < 10 
            ? '0' + n 
            : n;
    }
    
    function this_value() {
        return this.valueOf();
    }

    if (typeof Date.prototype.toJSON !== 'function') {

        Date.prototype.toJSON = function () {

            return isFinite(this.valueOf())
                ? this.getUTCFullYear() + '-' +
                        f(this.getUTCMonth() + 1) + '-' +
                        f(this.getUTCDate()) + 'T' +
                        f(this.getUTCHours()) + ':' +
                        f(this.getUTCMinutes()) + ':' +
                        f(this.getUTCSeconds()) + 'Z'
                : null;
        };

        Boolean.prototype.toJSON = this_value;
        Number.prototype.toJSON = this_value;
        String.prototype.toJSON = this_value;
    }

    var gap,
        indent,
        meta,
        rep;

    const join = (arr, joint) => {
        return arr.map((v) => [v, joint]).flat().slice(0, -1);
    };

    function quote(string) {

// If the string contains no control characters, no quote characters, and no
// backslash characters, then we can safely slap some quotes around it.
// Otherwise we must also replace the offending characters with safe escape
// sequences.

        rx_escapable.lastIndex = 0;
        return rx_escapable.test(string) 
            ? '"' + string.replace(rx_escapable, function (a) {
                var c = meta[a];
                return typeof c === 'string'
                    ? c
                    : '\u' + ('0000' + a.charCodeAt(0).toString(16)).slice(-4);
            }) + '"' 
            : '"' + string + '"';
    }


    function str(key, holder) {

// Produce a string from holder[key].

        var i,          // The loop counter.
            k,          // The member key.
            v,          // The member value.
            length,
            mind = gap,
            partial,
            value = holder[key];

// If the value has a toJSON method, call it to obtain a replacement value.

        if (value && typeof value === 'object' &&
                typeof value.toJSON === 'function') {
            value = value.toJSON(key);
        }

// If we were called with a replacer function, then call the replacer to
// obtain a replacement value.

        if (typeof rep === 'function') {
            value = rep.call(holder, key, value);
        }

// What happens next depends on the value's type.

        switch (typeof value) {
        case 'string':
            return quote(value);

        case 'number':

// JSON numbers must be finite. Encode non-finite numbers as null.

            return isFinite(value) 
                ? String(value) 
                : 'null';

        case 'boolean':
        case 'null':

// If the value is a boolean or null, convert it to a string. Note:
// typeof null does not produce 'null'. The case is included here in
// the remote chance that this gets fixed someday.

            return String(value);

// If the type is 'object', we might be dealing with an object or an array or
// null.

        case 'object':

// Due to a specification blunder in ECMAScript, typeof null is 'object',
// so watch out for that case.

            if (!value) {
                return 'null';
            }

// Make an array to hold the partial results of stringifying this object value.

            gap += indent;
            partial = [];

// Is the value an array?

            if (Object.prototype.toString.apply(value) === '[object Array]') {

// The value is an array. Stringify every element. Use null as a placeholder
// for non-JSON values.

                length = value.length;
                for (i = 0; i < length; i += 1) {
                    partial[i] = str(i, value) || 'null';
                }

// Join all of the elements together, separated with commas, and wrap them in
// brackets.

                v = partial.length === 0
                    ? new Blob(['[]'])
                    : gap
                        ? new Blob(['[n', gap, ...join(partial, ',n' + gap), 'n', mind, ']'])
                        : new Blob(['[', ...join(partial, ','), ']']);
                gap = mind;
                return v;
            }

// If the replacer is an array, use it to select the members to be stringified.

            if (rep && typeof rep === 'object') {
                length = rep.length;
                for (i = 0; i < length; i += 1) {
                    if (typeof rep[i] === 'string') {
                        k = rep[i];
                        v = str(k, value);
                        if (v) {
                            partial.push(new Blob([quote(k) + (
                                gap 
                                    ? ': ' 
                                    : ':'
                            ), v]));
                        }
                    }
                }
            } else {

// Otherwise, iterate through all of the keys in the object.

                for (k in value) {
                    if (Object.prototype.hasOwnProperty.call(value, k)) {
                        v = str(k, value);
                        if (v) {
                            partial.push(new Blob([quote(k), (
                                gap 
                                    ? ': ' 
                                    : ':'
                            ), v]));
                        }
                    }
                }
            }

// Join all of the member texts together, separated with commas,
// and wrap them in braces.
            v = partial.length === 0
                ? new Blob(['{}'])
                : gap
                    ? new Blob(['{n', gap, ...join(partial, ',n' + gap), 'n',  mind, '}'])
                    : new Blob(['{', ...join(partial, ','), '}']);
            gap = mind;
            return v;
        }
    }
// If the JSON object does not yet have a stringify method, give it one.

    if (true) {
        meta = {    // table of character substitutions
            'b': '\b',
            't': '\t',
            'n': '\n',
            'f': '\f',
            'r': '\r',
            '"': '\"',
            '\': '\\'
        };
        JSON.blobify = function (value, replacer, space) {

// The stringify method takes a value and an optional replacer, and an optional
// space parameter, and returns a JSON text. The replacer can be a function
// that can replace values, or an array of strings that will select the keys.
// A default replacer method can be provided. Use of the space parameter can
// produce text that is more easily readable.

            var i;
            gap = '';
            indent = '';

// If the space parameter is a number, make an indent string containing that
// many spaces.

            if (typeof space === 'number') {
                for (i = 0; i < space; i += 1) {
                    indent += ' ';
                }

// If the space parameter is a string, it will be used as the indent string.

            } else if (typeof space === 'string') {
                indent = space;
            }

// If there is a replacer, it must be a function or an array.
// Otherwise, throw an error.

            rep = replacer;
            if (replacer && typeof replacer !== 'function' &&
                    (typeof replacer !== 'object' ||
                    typeof replacer.length !== 'number')) {
                throw new Error('JSON.stringify');
            }

// Make a fake root object containing our value under the key of ''.
// Return the result of stringifying the value.

            return str('', {'': value});
        };
    }


// If the JSON object does not yet have a parse method, give it one.

    
}());

(async () => {
  const asBlob = JSON.blobify({foo: {bar: "baz", bla: [new Date(), ()=>{},,1,null]}});
  console.log({asBlob});
  const asString = await asBlob.text();
  console.log({asString});
  console.log("parsed", JSON.parse(asString));
})();

Please signup or login to give your own answer.

Click here to cancel reply.