Question 109120 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Requesting JSON data in python for web-scraping

I use the code below to fetch the JSON data for some of the stocks (e.g. https://www.tipranks.com/stocks/tsla/forecast ):

import json
import requests
url = " https://tr-frontend-cdn.azureedge.net/bff/prod/stock/tsla/payload.json?ver=1678406987078"
req = requests.get(url)
data = req.json()

But the website also has the data for ETF’s (for example,
https://www.tipranks.com/etf/xlb/forecast )
But I don’y know that url to use in this case (I tried https://tr-frontend-cdn.azureedge.net/bff/prod/etf/xlp/payload.json?ver=1678406987078 but this does not work).

Can someone help me identify what url should I put to properly get the JSON data?

Tags:

2

Answers


  1. 0

    What I do in such case is:

    • Open my browser inspector
    • Go to the network tab
    • Filter requests by XHR
    • Navigate the request list and check the response for each of them, until I find what I want

    In your case, you did not precise what data you were looking for, but I suspect this could be the endpoint you were looking for: https://www.tipranks.com/etf/xlb/payload.json

    Login or Signup to reply.
  2. 0

    Try to set User-Agent header:

    import requests

url = 'https://www.tipranks.com/etf/xlb/payload.json'

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0'
}

data = requests.get(url, headers=headers).json()
print(data)

    Prints:

    {
    "_meta": {
        "time": 1679207354777
    },
    "forecast": {
        "isHaveData": true,
        "highestUpside": [
            {
                "type": "stock",
                "isTraded": true,
                "ticker": "ALB",
                "name": "Albemarle Corporation",
                "fullName": "Albemarle Corporation",
                "gain": {
                    "yearly": 0.05837
                },
                "sector": "basicmaterials",
                "smartScore": {
                    "value": 7
                },
                "tradeOn": "us",
                "marketCap": 24434606485,
                "price": null,
                "analystRatings": {
                    "consensus": {
                        "id": "moderateBuy",
                        "buy": 9,
                        "sell": 2,
                        "hold": 7,
                        "total": 18,
                        "priceTarget": {
                            "value": 308.69
                        }
                    },
                    "bestConsensus": null
                },
                "holdingData": {
                    "shares": 726543,
                    "ratio": 0.029,
                    "value": 155923393.23,
                    "currency": "USD"
                }
            },

...and so on.
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search