Question 184491 in Javascript Question posted in
A very good W3school tutorial can be found here.

Fastest way to split object into smaller objects in Javascript?

I have a big object like below containing 1mil props.

{ 
"Name1":{"some":"object"},
"Name2":{"some":"object"},
"Name1000000":{"some":"object"}
}

I want to split this object into N parts so I wrote the following code

var bigObject = {
  "Name1": {
    "some": "object"
  },
  "Name2": {
    "some": "object"
  },
  "Name1000000": {
    "some": "object"
  }
};

const names = Object.keys(bigObject);
const partsCount = 4;
const parts = names
  .reduce((acc, name, idx) => {
    const reduceIndex = idx % partsCount;
    if (acc[reduceIndex] == null) {
      acc[reduceIndex] = {};
    }
    acc[reduceIndex][name] = request.body[name];
    return acc;
  }, new Array(Math.min(partsCount, names.length)));

While this works, the problem is with performance. Currently it’s taking 1.2 to 1.5 seconds!! Is there any performant way to write this code? My expectation is that this should be hardly double digit milliseconds for new processors running at ~3 GHz. Is my expectation wrong?

Update:
I somewhat don’t understand why people are looking for solutions outside of the given problem/question by asking "why" and "what" you are doing with this code. some have assumed it to be database use case. There are literally 1000s of use cases, anyway mine is IOT sensor data ingestion where max of 1 million sensors data is aggregated into a bulk REST call which needs to be processed, now here comes second issue as soon as i reveal the use-case people will try to optimize that, so let me be clear i cannot change anything apart for the code shown above Its already a parsed object now i have to divide it and process it.

Tags:

2

Answers


  1. 0

    But my approach here is to use Object.entries to divide the object into an array of its properties. Then divide those properties into N chunks using Array.slice, and then reassemble each chunk into an object using Object.fromEntries and finally push each object into an array of parts.

    function partitionProperties(obj, partsCount) {
  const entries = Object.entries(obj);
  const parts = [];
  for (let i = 1, a = 0, b = 0; i <= partsCount; ++i) {
    a = b;
    b = i / partsCount * entries.length | 0;
    parts.push(Object.fromEntries(entries.slice(a, b)));
  }
  return parts;
}

    It’s important to acknowledge that, while this does divide the object into parts, it doesn’t shuffle/arrange the properties into the same groups that you did. You shuffled/distributed them into groups one at a time like the way you would deal cards: [{a, d, g}, {b, e, h}, {c, f, i}] whereas this just partitions them [{a, b, c}, {d, e, f}, {g, h, i}]. But it’s not clear that you were attached to the order or grouping.

    I also don’t know if this is the fastest, but it is relatively simple and straightforward, which has its own advantages.

    Login or Signup to reply.
  2. 0

    Testing if (acc[reduceIndex] == null) in every iteration seems inefficient, and since it sometimes accesses properties that don’t exist, might considerably slow down your code. Try to pre-populate the array:

    const parts = names.reduce((acc, name, idx) => {
  acc[idx % partsCount][name] = request.body[name];
  return acc;
}, Array.from({length: Math.min(partsCount, names.length)}, () => ({})));
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search