skip to Main Content

I am trying to scrape the site ‘stockx’ to get the information at the bottom under the 12 month historical section, for some reason I can get some elements such as the last_sale price, but I can’t get any of the elements at the bottom such as volatility, I get an error for the volatility no matter what I try to do.

This is the html, I want to get the part that says 2%

<dl>
 <dt class="chakra-stat__label css-1qtnkr4">Volatility
  </dt>
 <dd class="chakra-stat__number css-jcr674">2%</dd>
</dl>

this is my code, currently volatility returns an error but last_sale works just fine.

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0',
    'Accept-Language': 'en-US,en;q=0.5'
}

cookies = {
    'stockx_homepage': "sneakers",
}

def main(url):
    r = requests.get(url, headers=headers, cookies=cookies)
    soup = BeautifulSoup(r.text, 'lxml')
    last_sale = soup.select('.css-xfmxd4')[0]
    volatility = soup.find("dd",class_="chakra-stat__number .css-jcr674")

    print("last sale: " + last_sale.text + "n")
    print("volatility: " + volatility.text + "n")

url = https://stockx.com/adidas-yeezy-boost-350-turtle-dove-2022?-size=11
main(url)

please help, I don’t know why I cant select/find the 'dd' tag, or the particular css for that part

2

Answers


  1. Chosen as BEST ANSWER

    I found the website I was trying to scrape is javascript dynamic thus i had to implement selenium chromedriver to get data loaded in javascript


  2. try this

    volatility=soup.find("dd",{"class":"chakra-stat__number css-jcr674"}).text

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search