I want to build a scrapper. And I wrote this code, it keeps on running into error. I would be grateful if anyone can help.
import requests
import pandas as pd
def fetch_json(api_link, timeout=10):
try:
response = requests.get(api_link, timeout=timeout)
if response.status_code == 200:
return response.json()
else:
print(f"Failed to fetch data from API for link: {api_link}")
return None
except requests.exceptions.Timeout:
print(f"Timeout occurred while fetching data for link: {api_link}. Skipping...")
return None
except requests.exceptions.RequestException as e:
print(f"Error occurred while fetching data for link {api_link}: {e}")
return None
api_link = "https://apiv2.cricket.com.au/web/players/list?isActive=1&teamId=&jsconfig=eccn%3Atrue&format=json"
player_data= fetch_json(api_link)
name = []
if player_data:
players = player_data['players']
for player in players:
id = "".join(players['id'])
displayname = "".join(players['displayname'])
name.append({
"id" : player.get("id"),
"displayname" : player.get("displayname")
})
print(f"Player Name Retrieved")
player_name = pd.DataFrame(name)
It shows error along the id column and I can’t seem to solve this problem.
2
Answers
As mentioned in the comment, your code has some simple typos when referring to the values in the
json
data. For eachplayer
in the list ofplayers
use:player['id']
player['displayName']
This is the sample data:
{'id': 2786, 'firstName': 'Khaliq', 'lastName': 'Noori', 'middleName': 'Dad', 'displayName': 'Khaliq Dad', 'nationality': '', 'gender': '', 'teams': [{'id': 6, 'name': 'Afghanistan Men', 'isWomensTeam': False}], 'isActive': True}
As you can see, you need to change from "players" and "displayname" to "players" and "displayName" respectively. Also, there is no need to use join here.