I am using an API request and this API has a limit of 2 calls a second and 250 records per request. That’s the gist.
I have created this background job that also has the option of a second background job. This may be overkill.
Flow:
- Order webhooks on creates from shopify
- Cron job once per day for that days orders in case webhooks fail.
Goal for request:
If there are >= 250
records/orders in the first API request, to then create a second background job in a new worker to fetch page 2 in about 3 minutes, and if page 2 has >= 250
, to then create a new background job in the same worker has page 2 (after 2 completes) 3 minutes after page 2 jobs started fetch page 3, and so on.. I use n
for the page and add 1
to n
if the 250 statement is true.
Cron Job:
shops = Shop.all
shops.map do |shop|
if shop.company.present?
ShopifyOrderUpdatesWorker.perform_later(shop)
end
end
Background job 1: (for first API call)
def perform(shop)
n = 1
orders = ShopifyAPI::Order.find(:all, params: {created_at_min: 1.day.ago.beginning_of_day.iso8601}, limit: 250, page: n )
while (orders.count >= 250) || n == 1
unless n =< 1
while n > 1 && orders.count >= 250
orders = ShopifyAPI::Order.find(:all, params: {created_at_min: 1.day.ago.beginning_of_day.iso8601 }, limit: 250, page: n)
#while orders.count >= 250 || n == 2
t = 3
ShopifyOrderUpdatesLimitWorker.delay_for(t.minutes).perform_later(orders)
n += 1 #add page to API call request
t += +3 #add 3 minutes for each loop to buffer the api call queue to avoid api limits to be safe
#end
end
end
if n == 1
orders.map do |order|
#code here
end
end
n += 1
end
end
Background job 2: (for any API call after the first)
def perform(orders)
orders.map do |order|
#code here
end
end
This way, all shops can update “quickly” without being in queue behind other shops. Shops that have a lot of orders will wait the same time they would in either case of doing all of this in one action or in 2.
Is this overkill? Done right for code
In reality, it is probably very rare a webhook will fail so the chances of the second background job being called is slim.
Any possible improvements or suggestions for the code?
This may not be the right place to ask this question, but if anyone has experience with shopify or similar api situations, what are you doing?
2
Answers
When planning your capacity and rate limiting – at first you should estimate your data size, because strategy for couple shops with rarely over couple of pages each and thousands of shops with multiple pages is very different.
Because of Shopify’s rate limiting is based on “Leaky bucket” algorithm with default bucket size of 40 (at the time of writing, see official docs) – you can make a burst of up to 40 api calls in a row without any rate limiting, that’ll suffice for a few shops with a couple of pages each.
If you have/plan to have more than that (or just want to be polite) – easiest way is to make a separate queue for these tasks with a single worker, so that tasks do not run in parallel. Then add a
sleep(0.5)
(or more) after each api call – and you should not saturate your limit even if you have other arbitrary calls somewhere else in your app. But be ready to receive a rare429 Too Many Requests
, in which case just wait a little more and repeat that call.add this in your loop