Im curious to what would be the best way if i wanted to export all Customers or Orders that meet a certain criteria as json or csv etc., considering the rate limit and pagination factors. I am guessing it would be some kind of looping script that would query a max of 250 records and repeat that until the pagination is done. Is there any code out there that shows this, i feel like with a lot of apps this would be useful or needed to do certain things. It’s more or less the export feature in shopify, but doing it in your own app.
2
Answers
Shopify has a nice API module for Node.js. It offers easy pagination handling and rate limiting. From the perspective of app, you may decide to use some queue mechanism. As these processes may take longer to complete based on number of orders or customers, whenever a client requests a record export, push the job in a queue and notify once done.
For fetching orders or customers using API while taking care of rate limiting, following code from Shopify API Node Docs should suffice.
Once done, you can write them as csv or JSON or anything else as needed. For queues, you may use something like Bull or Agenda.
The superior way to get the data you want is to use GraphQL and the Bulk Queries. With those you specify what you want in terms of Customer or Order data. Then you ask for the data with your query, and Shopify then packages it all up in a complete JSONL file. You it around and wait till they complete the package, at which point you get a download URL where you can have ALL the data.
Paging is a crappy way to go about exporting data. It works sure, but it is slow, and subject to network errors as anyone who has worked much with Shopify will tell you.