Question 119549 in Firebase Question posted in
An extensive official documentation can be found here.

Firebase – Large arrays in Firestore Database (Best practices)

I am populating a series of dates and temperatures that I was thinking of storing in a Firestore Database to later be consumed by the front-end with the following structure:

{
   date: ['1920-01-01', '1920-01-02', '1920-01-03', '1920-01-04', '1920-01-05', ...],
   values: [20, 18, 19.5, 20.5, ...]
}

The array may consider a lot of years, so it turns huge, with thousands of entries. Firestore database started complaining about returning the too many index entries for entity error, and even if I get the data uploaded, the user interface Firebase -> Firebase Database -> Panel View collapses. That happens even with less than 3000 entries array.

The fact is that the data is consumed in the front-end with an array structure very similar to the one described above (I want to plot it using Echarts library). This way, I found this structure to be the more natural way, as any other alternative will require reversing the structure to arrays in the front-end.

Nevertheless, I see that Firestore Database very clearly does not like this structure. What should I do? What is the best practice for dealing with this kind of data in Firestore?

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Firestore is not the best tool to deal with time series. The best solution I found in Firestore was creating an independent document for each day in my data. Nevertheless, that raises the number of documents I need to fetch from the front-end side and, therefore, the costs. By using large arrays in Firestore, you easily reach the index limit, and you are forced to remove the index, which I feel is a big red flag, suggesting checking another tool.

    The solution I found, in case is useful for anyone, was building my API in Flask using MongoDB as a database. Although it takes more effort than just using Firestore, it deals better with time series and brings more flexibility.


  2. 0

    The indexes required for the most basic queries in Firestore are automatically created for you. However, there are some limits involved. So you’re getting the following error:

    too many index entries for entity

    Because you hit the maximum number of index entries for a document, which is 40,000. If you add too many elements into an array or you add too many fields to a document, then you can reach the maximum limit.

    So most likely the number of elements that exist in the date array + the number of elements that exist in the values array is bigger than 40k, hence the error.

    To solve this, you might consider creating two separate documents, one for each array. If you still hit the maximum limit, then you might consider creating a document for each hour, and not for an entire day. In this way, you’ll drastically reduce the number of elements that exist in an array.

    If you don’t find these solutions useful, then you have to set some "Single-field index exemptions" to avoid the above error.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search