Open the url https://hnx.vn/en-gb/cophieu-etfs/chung-khoan-ny.html with browser,then click page 2 ,you can see that the content in response is a json.
I build a request with urllib to get the json:
import urllib.request
import ssl,json
base_url = "https://hnx.vn/ModuleIssuer/List/ListSearch_Datas"
proxy_handler = urllib.request.ProxyHandler({'https': '127.0.0.1:10801'})
opener = urllib.request.build_opener(proxy_handler)
urllib.request.install_opener(opener)
headers={
"Host": "hnx.vn",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0",
"Accept": "*/*",
"Accept-Language": "en-GB,en;q=0.5",
"Accept-Encoding": "gzip, deflate, br",
"Content-Type": "application/x-www-form-urlencoded",
"__RequestVerificationToken": "mKy3wwtZXopsLE5cIKMSnVxK934KPWzUievs0TWY7W9C9Hgu61EKzt1KTaxOGGTX5zvXIs0V_fPwWmEuDXLvXSfcSQNjY6uzwNCAeaQgyu01",
"X-Requested-With": "XMLHttpRequest",
"Content-Length": "113",
"Origin": "https://hnx.vn",
"Connection": "keep-alive",
"Referer": "https://hnx.vn/en-gb/cophieu-etfs/chung-khoan-ny.html",
"Sec-Fetch-Dest": "empty",
"Sec-Fetch-Mode": "cors",
"Sec-Fetch-Site": "same-origin"
}
data = {
"p_issearch":"0",
"p_keysearch":"",
"p_market_code":"",
"p_orderby":"STOCK_CODE",
"p_ordertype":"ASC",
"p_currentpage":"2",
"p_record_on_page":"10",
}
postdata = bytes(json.dumps(data), "utf-8")
req=urllib.request.Request(url=base_url,headers=headers,data=postdata,method='POST')
ssl._create_default_https_context = ssl._create_unverified_context
response = opener.open(req)
data = response.read()
We get the response ,but it is not a json,list the header of the data
data[0:50]
b'<!DOCTYPE html>rn<html>rnrn<script>rn '
The first 50 characters in json (responsed data) are as below:
'{"Content":"rnu003cstyleu003e.STT,.td_action{min-width:40p
Why can’t get the right response?How to fix my code?
2
Answers
To add proxy based on Andrej Kesely's code :
Although i get the json,the above code is based on requests library ,i can't accept it as a formal answer,my question remains,why my code with urllib library can't get the json?
I'll receive the resolution revised on my previous code，it is your chance to get the points.
To load the data from the URL you can use next example:
Prints: