Shopify Question posted in
The Shopify help center can be found here.

How to make concurrent HTTP requests with basic authentication – Shopify

My goal is to import a client’s history of orders from Shopify. Shopify only allows me to import 250 orders per request, but my client has thousands.

Here’s (basically) my current working solution using httr which is very slow

fetchedList <- list()

# Loop through pages of orders and collect them
for(pg in 1:10){

  requestURL <- paste0("https://abc-store.myshopify.com/admin/orders.json?page=", p)

  fetched <- httr::GET(
    url = requestURL,
    httr::add_headers(Accept = "application/json"),
    httr::authenticate(user = "foo", password = "bar")
  )

  # Append the fetched response to fetchedList 
  fetchedList <- c(fetchedList, list(fetched))
}

# Process the results...

I would like to speed this up by making multiple concurrent requests. How can I achieve this? It seems curl and RCurl both have support for this but I’m fairly new to HTTP and couldn’t get either solution working.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Thanks to @Jeroen who pointed me to the crul package. At the time, crul actually didn't have this functionality set up, but I spoke to the maintainer and he implemented it. So, as of v 0.5.2.9100 I can do

    dd <- Async$new(urls = c(
  'https://abc-store.myshopify.com/admin/orders.json?page=1',
  'https://abc-store.myshopify.com/admin/orders.json?page=2',
  'https://abc-store.myshopify.com/admin/orders.json?page=3'
))
res <- dd$get(auth = auth(user = "foo", pwd = "bar"))
vapply(res, function(z) z$status_code, double(1))
vapply(res, function(z) z$success(), logical(1))
lapply(res, function(z) z$parse("UTF-8"))

  2. 0

    You should use the multi api to do concurrent requests. See the manual page for ?multi_run or the section about async requests in the vignette.

    There are also packages that wrap the multi api to try and make it easier. The crul package (note crul is not a typo 🙂 or more if you want to get real fancy the async package.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search