I’m currently developing an admin dashboard using Firebase, where I need to fetch data for all members, including their related payments and vehicles.
Here is the data model I’m working with – overview of data schema using Typescript types
export type VehicleDocType = {
// Vehicle document fields
owner: string; // Member id
...
};
export type PaymentDocType = {
// Payment document fields
recipient: string; // Member id
...
};
export type MemberDocType = {
uid: string;
...
/* Note! Arrays are not coming from Firestore and not even part of Schema.
it is used to map relations on the client side
*/
payments: Array<PaymentDocType> | null;
vehicles: Array<VehicleDocType> | null;
};
Data structure in Firebase:
----------------------------------------------------------
| Collections | Documents | Fields |
----------------------------------------------------------
| members | n docs | { uid: doc.id, ...} |
| payments | n docs | { recipient: member.id, ... } |
| vehicles | n docs | { owner: member.id, ... } |
----------------------------------------------------------
Note: vehicles
collection is the same as payments
.
For the member page, I want to display all members along with their associated payments and vehicles.
This is my current approach for fetching data:
Firebase SDK: "firebase": "^9.22.0"
Exact queries I want to run:
const fetchCustomers = async () => {
// Fetch member documents from the 'membersCol' collection
const memberDocs = (await getDocs(membersCol)).docs.map((d) => d.data());
// Fetch payment documents from the 'paymentsCol' collection
const paymentDocs = (await getDocs(paymentsCol)).docs.map((d) => d.data());
// Fetch vehicle documents from the 'vehiclesCol' collection
const vehicleDocs = (await getDocs(vehiclesCol)).docs.map((d) => d.data());
const members = memberDocs.map((member) => ({
...member,
payments: paymentDocs.filter((p) => p.recipient === member.uid),
vehicles: vehicleDocs.filter((v) => v.owner === member.uid),
}));
// Further processing and rendering of the data
};
While this approach is functional, I have encountered a few problems:
- Loading large datasets can result in slow data retrieval times.
- When implementing Firebase pagination queries, I’m unsure how to efficiently map the relations (vehicles and payments) to each member.
- I’m looking for best practices to fetch the data with minimal document reads and optimize the overall loading time.
Any suggestions or recommendations on how to improve the data fetching process, optimize document reads, and reduce loading time would be greatly appreciated. Furthermore, if someone have any recommendations on how to implement real-time updates in this situation.
2
Answers
There is no singular best way to model data in most NoSQL databases. Instead it all depends on the use-cases of your app, and you’ll regularly need to adapt your data model as you add/change use-cases.
Your current data model seems highly normalized. There is nothing wrong with that, but (given that Firestore doesn’t support server-side joins) it does mean that you end up loading lots of individual documents. So your write operations are simple and singular, but your load operations get more complex and slow.
The other extreme is if you’d replicate all data you need for the member page into a single document. This affects multiple write operations, as they’ll all need to write data to multiple locations – so that’s more complex and slower. On the other hand, the read operation will now be much simpler and scaleable, as you need to load a single document.
Any data model you pick will be somewhere between these extremes, and depend on your use-cases and your experience/comfort level around data duplication and fan-out operations.
To learn more about this, I recommend reading NoSQL data modeling and watching Get to know Cloud Firestore.
Based on your current approach, you can marginally optimize the sorting of payments and vehicles by attaching each payment and vehicle directly to the member.
For an array of 10 (
M
) members, 20 (V
) vehicles and 30 (P
) payments:filter
callbacks a total of 500 times (M * (V + P)
). Additionally, the filter callbacks need to be initialised again inside each map callback increasing the processing time.forEach
approach below will invoke theforEach
callbacks only 50 times (V + P
).As you can see from the above calculation, it is
M
times more efficient to use the latter approach.As your database grows, your queries will become significantly wasteful. So to account for this, you can elect to show a handful of members at a time instead. Below, the method has been tweaked so that it now fetches 10 members at a time, along with only their payments and vehicles. Note: to increase this number beyond 10, you’ll need to break the memberUserIdsForPage array into smaller arrays of up to 10 IDs to account for querying limitations.
Depending on your user interface, it may be desirable to get the members for the current page, display them with spinners for payments and vehicles, then fetch the payments and vehicles, and fill the data into the interface.