I need to back up my prod server Firestore DB hourly. I know about exportDocuments but it incurs one read operation per document exported. I have more than 3 million and these are increasing day by day.
Is it possible to export docs that are added/updated in a given period like the last 1 hour?
I already have Cloud Scheduler + Cloud Pub/Sub + function-based backup system. It is backing up all the docs. It is costing too much.
2
Answers
If you need to schedule some operations in Firestore, you can consider using Cloud Scheduler, which allows you to schedule HTTP requests or Cloud Pub/Sub messages to Cloud Functions for Firebase that you deploy.
If you need to get the documents that are added/updated in a given period of time, like the last 1 hour, then don’t forget to add a timestamp field to your documents. In this way, you can query based on that timestamp field.
To get docs that are added/updated in a given period like in the last 1 hour, add a field to the document, say
lastUpdated, and keep its value current with every insert/update.Then query for incremental documents like,
where("lastUpdated", ">", "lastExportTimestamp")from the backup function,lastExportTimestampbeing the time of last export (and may be stored in a separate collection).See an example here.
Hope this clarifies, else leave a comment.
P.S. Please be advised that this approach may still need a full periodic backup (say daily), for ease of restore process, if/when required.