Question 245115 in Javascript Question posted in
A very good W3school tutorial can be found here.

Javascript – Cache local File/Image in RAM

The setup

I am working on a vuejs WebApp that loads and displays a lot of images (~10-20k). These images are opened from a network drive.

When the user opens a folder from that drive I collect all files and then create URLs for them.

// get blob objects for all files
const dirHandle = await window.showDirectoryPicker();
const filePromises: Promise<File>[] = getFilePromises(dirHandle);
const files = await Promise.all(filePromises);

const imageURLs = files.map((file) => URL.createObjectURL(file))

In there getFilePromises() simply recursively loops over dirHandle.values() and pushes value.getFile() to the promises array.

These image urls are then used to render a grid <img> tags. The grid is of course virtualized to not overload the DOM.

The problem

Now the problem is that the network drive is very slow sometimes.

Never the less the above code runs pretty fast (couple of seconds only for 10k images). But it really takes forever until the first images are displayed/loaded in the grid. Until then I only see empty boxes.
enter image description here
(I display the images with a border and a fixed size, which is why I see the boxes before the image is loaded).

Then step by step the images slowly appear.

Repeating this with the same images stored on local SSD leads to nearly instant displaying. So it’s definitely a loading issue.

Theories and Questions

As the delay happens after the above code has finished running, I have 2 theories and 2 corresponding questions:

  1. The blobs will eventually all get loaded into the RAM/heap automatically. But this process is running in the background after the above code has finished.
    • => How can I await that process / know when it is finished?
  2. The blobs are only referenced to the network drive and are only loaded in RAM on demand.
    • => How can I cache all images in RAM instantly?

Can someone explain exactly what’s happening there? I though awaiting the file blob object meant I await it beeing loaded into RAM. But that’s clearly not the case.

Clarification

I know there’s no way around waiting x minutes until all images are pulled from the drive.

But I’d rather pull them all at once, show a loading animation to the user and then deliver him a fluent user experience than showing him and empty grid without him knowing what’s going on.

10k images are only about 500Mb so this should definitely fit in the RAM.

Thank you all!:)

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Now I figured a way how I can force the file blobs to be in RAM, so this basically answers my question:

    After having the array of Files I read their content into an ArrayBuffer. This to my understanding than has to be in RAM. Afterwards I recreate blobs / files from those ArrayBuffers. The resulting files are then in RAM.

    function cacheFilesInRam(files: File[]): File[] {
 const arrayBufferPromises = this.files.map((file) => file.arrayBuffer());
 const arrayBuffers = await Promise.all(arrayBufferPromises);
 const blobs = arrayBuffers.map(
   (buffer) => new Blob([buffer], { type: "image/jpeg" })
 );
 return blobs.map((blob, idx) => new File([blob], files[idx].name));
}

    The important part for me is, that after the code as finished, I am sure all images are completely in RAM. So after the initial waiting time the application runs super fast and there are no loading times any more.

    The question is: Is this bad practice?. Or are there any arguments speaking against this practice? Appreciate any comments!


  2. 0
    1. Load images sequentially. If you load images one by one rather then 1000 at once, 5-10 threads is optimal for http images (browser uses 8 threads iirc), dunno about drive but don’t use over 20 unleast you have really high ping
    for (let [index, file] of files.entries()) imgs[i] = await load(file)

    1. Create actual image element document.createElement('img').src = dataScc; img.onload = resolve(img)
      so you when image is loaded into memory

    2. Use window.caches if you want to seamlessly store the data on user’s PC
      (in case you want it to be fast without)

    3. Try do display images immediately – highly prioritize loading visible images
      If the loading of 10’000 images takes 100s of seconds, loading of 100 images on the screen will take just seconds

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search