Question 162170 in Javascript Question posted in
A very good W3school tutorial can be found here.

Can Selenium execute_script pass a JS variable in Python?

Im sorry for literally spamming Selenium threads.

driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", element)

This is the code I use to scroll down on a dynamically generated list ( Instagram followers). It’s a finite list, so the lower you go with the scroll, the smaller that scroll gets, because at the top the list has only 12 elements inside, while at the bottom it consists of all your followers. My issue is that I have some data loss: when I scanned 2 accounts with 406, respectively 280 followers, the scraper returned only 392 and 264 usernames.
I checked the HTML sauce of the page to see if the loss was due to data gen/scraping/data processing. And it seems that the "fully" generated HTML lists had 392 , respectively 264 elements inside. So the way elements were loaded on the page was problematic. I don’t know what is causing these losses, I suppose internet hiccups combined with chunky scrolling ( around 2-3 secs of pause before and after each scroll with time.sleep(), so time to load shout not be a problem ).
Returning to

driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", element)

I want to make the scroll a little finer.
As the scroll gets progressively smaller and smaller and the scrollbar keeps the same size, I can’t go for a preset pixel amount . Obviously neither with a fixed factor to multiply the scrollheight, like 1/2 or 3/4 bcs the scroll won’t reach the end of the list. The way I see it I need a dynamic factor that increases with every iteration of the scroll loop, something like

for i in range(max_followers):
  q=i/max_followers
  driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight*q", element)

so each scroll will go down ~1 element/iteration.
But when I run a code like this I get an error:

selenium.common.exceptions.JavascriptException: Message: javascript error: q is not defined

Pretty understandable I guess, I have not defined a variable inside js.Is there any way to define q inside execute_script and pass values generated in python ( i, max_followers ) to jscript?

Tags:

2

Answers


  1. 0

    You can pass multiple arguments to execute_script.

    driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight * arguments[1]", element, q)
    Login or Signup to reply.
  2. 0

    You can format the string before passing it to Javascript. The standard way to do this in Python3 is f-strings, which uses {} to reference a variable.

    for i in range(max_followers):
  q=i/max_followers
  driver.execute_script(f"arguments[0].scrollTop = arguments[0].scrollHeight*{q}", element)

    What this does is replace q with your calculated q. You can read more about f-string syntax here: https://realpython.com/python-f-strings/.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search