Question 206971 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Json – error message when I try to web scrape data with python

I am trying to get data from this site.
My code to get the data is:

import requests
import pandas as pd
from bs4 import BeautifulSoup
import json

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
    }

data = {
    'columns[0][name]': 'ID_Societe',
    'columns[1][name]': 'sNom',
    'columns[2][name]': 'sURL',
    'columns[3][name]': 'sGroupe',
    'columns[4][name]': 'nFonds',
    'columns[5][name]': 'nParts',
    'columns[6][name]': 'nEncours',
    'columns[7][name]': 'dtEncours',
    'columns[8][name]': 'sAwards',
    'columns[9][name]': 'nAAA',
    'columns[10][name]': 'nAA',
    'columns[11][name]': 'nA',
    'columns[12][name]': 'dtAwards',
    'order[0][column]': '2',
    'order[0][dir]': 'asc',
    'start': '0',
    'length': '10',
}

response = requests.post('https://www.quantalys.com/SocieteGestion/ListeData', headers=headers, data=data)
df_list_sdg = pd.DataFrame(response.json()['data'])

df_list_sdg

but I had an error message which is:

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I don’t understand why I get this error message.

Any help will be helpful.

Tags:

2

Answers


  1. 0

    The response of your request is

    print(response.text)

    '<!DOCTYPE html>rn<html>rn    <head>rn        <title>Runtime Error</title>rn        <meta name="viewport" content="width=device-width" />rn        <style>rn         body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} rn         p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}rn         b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}rn         H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }rn         H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }rn         pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}rn         .marker {font-weight: bold; color: black;text-decoration: none;}rn         .version {color: gray;}rn         .error {margin-bottom: 10px;}rn         .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:pointer; }rn         @media screen and (max-width: 639px) {rn          pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }rn         }rn         @media screen and (max-width: 479px) {rn          pre { width: 280px; }rn         }rn        </style>rn    </head>rnrn    <body bgcolor="white">rnrn            <span><H1>Server Error in '/' Application.<hr width=100% size=1 color=silver></H1>rnrn            <h2> <i>Runtime Error</i> </h2></span>rnrn            <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">rnrn            <b> Description: </b>An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine.rn            <br><br>rnrn            <b>Details:</b> To enable the details of this specific error message to be viewable on remote machines, please create a &lt;customErrors&gt; tag within a &quot;web.config&quot; configuration file located in the root directory of the current web application. This &lt;customErrors&gt; tag should then have its &quot;mode&quot; attribute set to &quot;Off&quot;.<br><br>rnrn            <table width=100% bgcolor="#ffffcc">rn               <tr>rn                  <td>rn                      <code><pre>rnrn&lt;!-- Web.Config Configuration File --&gt;rnrn&lt;configuration&gt;rn    &lt;system.web&gt;rn        &lt;customErrors mode=&quot;Off&quot;/&gt;rn    &lt;/system.web&gt;rn&lt;/configuration&gt;</pre>                      </code>rnrn                  </td>rn               </tr>rn            </table>rnrn            <br>rnrn            <b>Notes:</b> The current error page you are seeing can be replaced by a custom error page by modifying the &quot;defaultRedirect&quot; attribute of the application&#39;s &lt;customErrors&gt; configuration tag to point to a custom error page URL.<br><br>rnrn            <table width=100% bgcolor="#ffffcc">rn               <tr>rn                  <td>rn                      <code><pre>rnrn&lt;!-- Web.Config Configuration File --&gt;rnrn&lt;configuration&gt;rn    &lt;system.web&gt;rn        &lt;customErrors mode=&quot;RemoteOnly&quot; defaultRedirect=&quot;mycustompage.htm&quot;/&gt;rn    &lt;/system.web&gt;rn&lt;/configuration&gt;</pre>                      </code>rnrn                  </td>rn               </tr>rn            </table>rnrn            <br>rnrn            </font>rnrn    </body>rn</html>rn'

    which is html and not json

    Login or Signup to reply.
  2. 0

    It seems that you need to send more parameters to get correct repsonse:

    import requests
import pandas as pd
from bs4 import BeautifulSoup
import json

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}

data = {
    "draw": "2",
    "columns[0][data]": "ID_Societe",
    "columns[0][name]": "ID_Societe",
    "columns[0][searchable]": "true",
    "columns[0][orderable]": "true",
    "columns[0][search][value]": "",
    "columns[0][search][regex]": "false",
    "columns[1][data]": "1",
    "columns[1][name]": "checkbox",
    "columns[1][searchable]": "true",
    "columns[1][orderable]": "false",
    "columns[1][search][value]": "",
    "columns[1][search][regex]": "false",
    "columns[2][data]": "sNom",
    "columns[2][name]": "sNom",
    "columns[2][searchable]": "true",
    "columns[2][orderable]": "true",
    "columns[2][search][value]": "",
    "columns[2][search][regex]": "false",
    "columns[3][data]": "sURL",
    "columns[3][name]": "sURL",
    "columns[3][searchable]": "true",
    "columns[3][orderable]": "true",
    "columns[3][search][value]": "",
    "columns[3][search][regex]": "false",
    "columns[4][data]": "sGroupe",
    "columns[4][name]": "sGroupe",
    "columns[4][searchable]": "true",
    "columns[4][orderable]": "true",
    "columns[4][search][value]": "",
    "columns[4][search][regex]": "false",
    "columns[5][data]": "nFonds",
    "columns[5][name]": "nFonds",
    "columns[5][searchable]": "true",
    "columns[5][orderable]": "true",
    "columns[5][search][value]": "",
    "columns[5][search][regex]": "false",
    "columns[6][data]": "nParts",
    "columns[6][name]": "nParts",
    "columns[6][searchable]": "true",
    "columns[6][orderable]": "true",
    "columns[6][search][value]": "",
    "columns[6][search][regex]": "false",
    "columns[7][data]": "nEncours",
    "columns[7][name]": "nEncours",
    "columns[7][searchable]": "true",
    "columns[7][orderable]": "true",
    "columns[7][search][value]": "",
    "columns[7][search][regex]": "false",
    "columns[8][data]": "dtEncours",
    "columns[8][name]": "dtEncours",
    "columns[8][searchable]": "true",
    "columns[8][orderable]": "true",
    "columns[8][search][value]": "",
    "columns[8][search][regex]": "false",
    "columns[9][data]": "sAwards",
    "columns[9][name]": "sAwards",
    "columns[9][searchable]": "true",
    "columns[9][orderable]": "true",
    "columns[9][search][value]": "",
    "columns[9][search][regex]": "false",
    "columns[10][data]": "nAAA",
    "columns[10][name]": "nAAA",
    "columns[10][searchable]": "true",
    "columns[10][orderable]": "true",
    "columns[10][search][value]": "",
    "columns[10][search][regex]": "false",
    "columns[11][data]": "nAA",
    "columns[11][name]": "nAA",
    "columns[11][searchable]": "true",
    "columns[11][orderable]": "true",
    "columns[11][search][value]": "",
    "columns[11][search][regex]": "false",
    "columns[12][data]": "nA",
    "columns[12][name]": "nA",
    "columns[12][searchable]": "true",
    "columns[12][orderable]": "true",
    "columns[12][search][value]": "",
    "columns[12][search][regex]": "false",
    "columns[13][data]": "dtAwards",
    "columns[13][name]": "dtAwards",
    "columns[13][searchable]": "true",
    "columns[13][orderable]": "true",
    "columns[13][search][value]": "",
    "columns[13][search][regex]": "false",
    "order[0][column]": "2",
    "order[0][dir]": "asc",
    "start": "0",
    "length": "10",
    "search[value]": "",
    "search[regex]": "false",
    "sNom": "",
    "sNotation": "",
    "sSignePerf": "ge",
    "nPerf": "",
    "sType": "",
    "treeSDG": "",
    "treeNationalite": "",
    "treeAwards": "",
    "treeQuantaAwards": "",
}

response = requests.post(
    "https://www.quantalys.com/SocieteGestion/ListeData", headers=headers, data=data
)
df_list_sdg = pd.DataFrame(response.json()["data"])

print(df_list_sdg)

    Prints:

       ID_Societe  ID_CatSociete  ID_MaisonMere                                 sNom                           sURL  nFonds  nParts sPays      nEncours   dtEncours sAwards  nAwards  nA  nAA  nAAA    dtAwards   sGroupe  bGroupe  bMaisonMere
0        1829              3            NaN                                    -  /societegestion/synthese/1829      83      83    FR  4.409222e+09  2023-07-12    None        0   0    0     0        None      None    False        False
1        1321              3            NaN              123 Investment Managers  /societegestion/synthese/1321      48      53    FR  5.607430e+08  2023-02-28    None        0   0    0     0        None      None    False        False
2        6536              3            NaN                          21Shares AG  /societegestion/synthese/6536      10      10    CH           NaN        None    None        0   0    0     0        None      None    False        False
3        6426              3            NaN                           2Xideas AG  /societegestion/synthese/6426       1       4    CH           NaN        None    None        0   0    0     0        None      None    False        False
4         558              3            NaN                       A Plus Finance   /societegestion/synthese/558      32      41    FR  2.201705e+08  2023-07-12    None        0   0    0     0        None      None    False        False
5         380              3         5548.0             Abeille Asset Management   /societegestion/synthese/380       8      14    FR  4.712152e+08  2023-03-31    None        0   1    1     1  2022-10-31     Aviva    False        False
6        1850              3         6697.0                   Abeille Assurances  /societegestion/synthese/1850      12      12    FR           NaN        None    None        0   0    0     0        None      Aema    False        False
7        4923              3         5539.0   Aberdeen Asset Management Asia Ltd  /societegestion/synthese/4923       5      30    SG           NaN        None    None        0   0    0     0        None  Aberdeen    False        False
8        5961              1         5539.0      Aberdeen Asset Managers Limited  /societegestion/synthese/5961       8      67    GB           NaN        None    None        0   0    0     0        None  Aberdeen    False        False
9        4925              3         5539.0  Aberdeen Standard Fund Managers Ltd  /societegestion/synthese/4925      21     125    GB  2.666350e+08  2020-03-31    None        0   0    0     0        None  Aberdeen    False        False
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search