Shopify Question posted in
The Shopify help center can be found here.

Shopify API – Get all products (60k products) Got Request Time Out or socket hang up

I am trying to get all of products but I got Request timed out while trying to get 60k products for inventory management app.

I am using nodejs to loop into 200 pages, each page limit to 250 products. I limited 2 requests ever 10 seconds for my calls (5 seconds/1 request)

sometime I got these errors one some pages. Sometimes not

  • read ECONNRESET

  • Request timed out

  • socket hang up

Could any one please tell me what is the problem? I would appreciate your help.

for (var i = 1; i<= totalPage;i++)

{

  var promise = shopify.product.list({limit:limit,page:i,fields:fields})

  .then(products =>{

  // do some thing here when got products list
    // loop through each product then save to DB
    // ShopifyModel.updateOne(.....)

  }).catch(error=>{

  // some time it fired error here 

  })

}

I also tried to rewrite a function to get products of 1 page:

const request = require('request-promise');

var getProductOnePage = function (Url_Page,headers,cb){
    request.get(productUrl, { headers: headers,gzip:true })
    .then((ListProducts) => {
        console.log(" Got products list of one page");
        cb(ListProducts);
    })
    .catch(err=>{
        // Got All Error Here when try to put into for loop or map or forEach with promise.all
        console.log("Error Cant get product of 1 page: ",err.message);
    });
}

EDIT:
I found some problem similar to my case here:
https://github.com/request/request/issues/2047
https://github.com/twilio/twilio-node/issues/312

Tags:

2

Answers


  1. 0

    You have to understand the concept of rate limiting. With any public API like Shopify, you can only make so many calls before they put you on hold. So when you get a response back from Shopify, you can check the header for how many requests you can make. If it is zero, you’ll get back a 429 if you try a request.

    So when you get a 0 for credits, or a 429 back, you can set yourself a little timeout and wait to make your next call.

    If on the other hand, as you say, you are only doing 2 calls every 10 seconds (not at all clear from your code how you do that, and why??) and you’re getting timeouts, then your Internet connection to Shopify is probably the problem.

    Login or Signup to reply.
  2. 0

    • ECONNRESET and Request timed out errors are there mostly due to network problem. Check if you have a stable internet connection.

    • If you’re using shopify api node package, then use autoLimit property. It will take care of rate limiting.

    eg:

    const shopify = new Shopify({
    shopName: shopName,
    apiKey: api_key,
    password: password,
    autoLimit : { calls: 2, interval: 1000, bucketSize: 30 }
});

    Edit: Instead of writing then catch inside a for loop, use async await. Because weather you implement request and wait approach or not, for loop will send all requests. But if you use await, it will process one request at a time.

    let getProducts = async () => {
    for (var i = 1; i<= totalPage;i++)
    {
        try {
            let products = await shopify.product.list({limit:limit,page:i,fields:fields});
            if(!products.length) {
                // all products have been fetched
                break;
            }
            // do you stuffs here
        } catch (error) {
            console.log(error);
        }
    }
}
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search