Debian Question posted in
The oficial documentation can be found here.

Selenium ChromeDriver Not Locating HTML Elements – Debian

My python script is running on a Debian Google Cloud Server. I am trying to scrape news from
Forexfactory.com/news. I tested my code on my macOS and it worked beautifully. After uploading onto the server I had to add chrome option arguments to my code to help find the chromedriver in the VM. It was after debugging this that selenium was unable to find the elements required.

This is the python code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import os

opt = Options()
opt.add_argument("--no-sandbox")
opt.add_argument("--disable-dev-shm-usage")
opt.add_argument("--headless")

driver = webdriver.Chrome(chrome_options=opt, executable_path=r'/home/rehanmahmood38/chromedriver')
URL = 'https://www.forexfactory.com/news'
driver.get(URL)

driver.implicitly_wait(5) # wait for seconds
uiOuter = driver.find_element_by_id('ui-outer')
aHref = driver.find_elements_by_css_selector('div.flexposts__story-title a')
span = driver.find_elements_by_css_selector('div.flexposts__storydisplay-info')

This is the error message

Traceback (most recent call last):
  File "ffmain.py", line 16, in <module>
    uiOuter = driver.find_element_by_id('ui-outer')
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 360, in find_element_by_id
    return self.find_element(by=By.ID, value=id_)
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"ui-outer"}
  (Session info: headless chrome=83.0.4103.116)
  (Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee976b2c3e3b9f335cde7),platform=Linux 4.19.0-12-cloud-amd64 x86_64)

Any idea why selenium is unable to locate the elements

Tags:

2

Answers


  1. 0

    Instead of implicitly waiting, use selenium’s explicit wait.
    Seems like the element may have not loaded properly – hence it was not found.
    Try using this.

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

uiOuter = WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.ID, "ui-outer"))

    The docs on waits is helpful if you would like to look into it further.

    https://selenium-python.readthedocs.io/waits.html

    Login or Signup to reply.
  2. 0

    Use this for your custom window size. Probably it differs on MacOS and VM.

    driver.manage().window().setSize(new Dimension(1024,768));
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search