Javascript Question posted in
A very good W3school tutorial can be found here.

How to reach an element created with JavaScript while scrapping via puppeteer?

I am using Puppeteer to reach a button (unfollow button) in my GitHub followings page. Because this button does not exist when I scrape this page I think it has been created by user-side JS code. Then I used waitForSelector command in order to wait for this button, but again page could not find the button. Here is my code:

const test2=async()=>{
    let url='https://github.com/nariman?tab=following'
    await browserInit()
    await page.goto(url)
    await page.waitForSelector('input.btn')
    console.log('done')
}

and it will cause this error:

TimeoutError: Waiting for selector `input.btn` failed: Waiting failed: 30000ms exceeded

You can test it in your own GitHub page. Is there any solution?

Tags:

2

Answers


  1. 0

    you can use something called MutationObserver like page.on(‘mutationover’,callback) by using which you can monitor DOM changes and wait specifically for the unfollow button insertion

    Login or Signup to reply.
  2. 0

    use timeout delay instead of waiting for a selector. It is a crutch but works fine for me. Also make sure your selector is not inside of the iframe

    const delay = (ms: number) => {
            return new Promise(resolve => setTimeout(resolve, ms));
        }        
    
    await page.goto('https://www.indeed.com/')
    await delay(2000);
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search