Question 160427 in Javascript Question posted in
A very good W3school tutorial can be found here.

Can I use Javascript/Promises/Fetch/Axios to fetch data from an unknown # of urls?

I am struggling to get my head around async/promises using Javascript.

This issue is: I have an unknown number of pages to fetch. This only becomes apparent when there is no data returned when attempted.

In summary:

There is a defined base url (postedUrl)

The ‘page’ number parameter increments by 1 every time a fetch has been completed (and contains data)

Using axios (or what ever) to fetch the page and store the data in a global array – accessible later on

The issue I am having is, being able to successfully run the fetch an unknown amount of times (till there is no more data returned).

Note: The page number is not supposed to be used in the do/while loop. I could not get it to correctly alter isWorking to 0, and stop itself. So I used a temp solution so I could test out the code.

Can anyone advise the best way to do this?

const getHostname = (url) => {
    return new URL(url).hostname;
}

let isWorking = 1;
let postedUrl = "https://example.com/"
let parsedUrl = getHostname(postedUrl)
let page_number = 110;

let total_products = 0;

let productList = [];


async function fetchData() {

    axios.get(postedUrl,
        {
            params: {
                page: page_number
            }
        })
        .then((response) => {
            console.log(response);
            if (response.data.length == 0) {
                isWorking = 0;
                alert("is triggering");

            }

        })
        .catch((err) => {
            console.log(err);

        });


    console.log(isWorking);
    page_number++;

}


async function startWork() {
    do {
        await fetchData();
        console.log(isWorking);
    } while (page_number < 115)
}

startWork();
Tags:

2

Answers


  1. 0

    Recursion could be the answer here. Just try

    async function startWork() {
    await fetchData()
    if(isWorking) startWork()
}
    Login or Signup to reply.
  2. 0

    If the pages are independent of each other, you can get all of them at the same time using Promise.allSettled(), which returns a Promise that fulfills when all Promises passed to it has settled:

    const pagesToFetch = [110, 111, 112, 113, 114, 115];
// or Array.from(Array(115 - 110 + 1).keys(), e => e + 110);

const productList = await Promise.allSettled(
  pagesToFetch.map(page => axios.get(/* your url/params/etc. here  */))
);

    Try it:

    console.config({ maximize: true });

const url = 'https://picsum.photos';
const sizesToFetch = [100, 200, 300, 400, 500, 600];

let responses;

(async function() {
  responses = await Promise.allSettled(
    sizesToFetch.map(page => {
      return fetch(`${url}/${page}`);
    })
  );
  console.log(responses);
})();
    <script src="https://gh-canon.github.io/stack-snippet-console/console.min.js"></script>
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search