Question posted in Json
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

Not able to parse JSON data from text file using python script

Loki
January 13, 2023
279 views
0 votes
2 Answers

I have a ‘.txt’ file that contains JSON data like below.

[{'tradable': True, 'mode': 'full', 'instrument_token': 4708097, 'last_price': 178.65, 'last_traded_quantity': 5, 'average_traded_price': 180.1, 'volume_traded': 4581928, 'total_buy_quantity': 1282853, 'total_sell_quantity': 1673842, 'ohlc': {'open': 181.95, 'high': 181.95, 'low': 177.8, 'close': 181.0}, 'change': -1.2983425414364609, 'last_trade_time': datetime.datetime(2023, 1, 12, 13, 4, 58), 'oi': 0, 'oi_day_high': 0, 'oi_day_low': 0, 'exchange_timestamp': datetime.datetime(2023, 1, 12, 13, 5, 1), 'depth': {'buy': [{'quantity': 653, 'price': 178.6, 'orders': 8}, {'quantity': 2408, 'price': 178.55, 'orders': 15}, {'quantity': 6329, 'price': 178.5, 'orders': 22}, {'quantity': 9161, 'price': 178.45, 'orders': 24}, {'quantity': 7775, 'price': 178.4, 'orders': 17}], 'sell': [{'quantity': 5726, 'price': 178.7, 'orders': 8}, {'quantity': 4099, 'price': 178.75, 'orders': 11}, {'quantity': 23951, 'price': 178.8, 'orders': 25}, {'quantity': 7446, 'price': 178.85, 'orders': 21}, {'quantity': 11379, 'price': 178.9, 'orders': 21}]}}, {'tradable': True, 'mode': 'full', 'instrument_token': 871681, 'last_price': 972.55, 'last_traded_quantity': 1, 'average_traded_price': 973.85, 'volume_traded': 411290, 'total_buy_quantity': 152925, 'total_sell_quantity': 214765, 'ohlc': {'open': 971.75, 'high': 978.6, 'low': 969.0, 'close': 967.75}, 'change': 0.4959958667011061, 'last_trade_time': datetime.datetime(2023, 1, 12, 13, 4, 53), 'oi': 0, 'oi_day_high': 0, 'oi_day_low': 0, 'exchange_timestamp': datetime.datetime(2023, 1, 12, 13, 5, 4), 'depth': {'buy': [{'quantity': 6, 'price': 972.15, 'orders': 2}, {'quantity': 3, 'price': 972.1, 'orders': 2}, {'quantity': 15, 'price': 972.05, 'orders': 3}, {'quantity': 455, 'price': 972.0, 'orders': 16}, {'quantity': 14, 'price': 971.95, 'orders': 2}], 'sell': [{'quantity': 6, 'price': 972.5, 'orders': 3}, {'quantity': 49, 'price': 972.55, 'orders': 2}, {'quantity': 10, 'price': 972.6, 'orders': 1}, {'quantity': 27, 'price': 972.65, 'orders': 2}, {'quantity': 10, 'price': 972.7, 'orders': 1}]}}]

This data was written to a .txt file after it was recieved from zerodha websocket. Now, I want to read the data from the .txt file using my python script and want to load it as a json. But the json.loads() method in python throws the below error.
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 3 (char 2)

I have tried eval and ast.literal_eval methods in python as well but it didn’t solve my problem. All I want is to be able to read the above data as a JSON to my python script. Any leads would be of great help.

Answers

Your JSON is not valid. You can check that by using an online JSON validator like this one: https://jsonformatter.org/

After entering the “JSON”, you can see it is not in a valid format.
You could either export it right, which I would highly recommend, or you could replace the wrong chars.

Currently, you have three issues:

You are using single quotes instead of double quotes
You are not parsing your datetime object, it looks like you just insert the object, you have to serialize it
You are writing true values as True, but that is the python way and not the JSON way. You either have to write it as true, or you have to pass it as a string. I would recommend the first one.

It could look like this (but I didnt parse datetime right, I just stringified it):

[{"tradable": true, "mode": "full", "instrument_token": 4708097, "last_price": 178.65, "last_traded_quantity": 5, "average_traded_price": 180.1, "volume_traded": 4581928, "total_buy_quantity": 1282853, "total_sell_quantity": 1673842, "ohlc": {"open": 181.95, "high": 181.95, "low": 177.8, "close": 181.0}, "change": -1.2983425414364609, "last_trade_time": "datetime.datetime(2023, 1, 12, 13, 4, 58)", "oi": 0, "oi_day_high": 0, "oi_day_low": 0, "exchange_timestamp": "datetime.datetime(2023, 1, 12, 13, 5, 1)", "depth": {"buy": [{"quantity": 653, "price": 178.6, "orders": 8}, {"quantity": 2408, "price": 178.55, "orders": 15}, {"quantity": 6329, "price": 178.5, "orders": 22}, {"quantity": 9161, "price": 178.45, "orders": 24}, {"quantity": 7775, "price": 178.4, "orders": 17}], "sell": [{"quantity": 5726, "price": 178.7, "orders": 8}, {"quantity": 4099, "price": 178.75, "orders": 11}, {"quantity": 23951, "price": 178.8, "orders": 25}, {"quantity": 7446, "price": 178.85, "orders": 21}, {"quantity": 11379, "price": 178.9, "orders": 21}]}}, {"tradable": true, "mode": "full", "instrument_token": 871681, "last_price": 972.55, "last_traded_quantity": 1, "average_traded_price": 973.85, "volume_traded": 411290, "total_buy_quantity": 152925, "total_sell_quantity": 214765, "ohlc": {"open": 971.75, "high": 978.6, "low": 969.0, "close": 967.75}, "change": 0.4959958667011061, "last_trade_time": "datetime.datetime(2023, 1, 12, 13, 4, 53)", "oi": 0, "oi_day_high": 0, "oi_day_low": 0, "exchange_timestamp": "datetime.datetime(2023, 1, 12, 13, 5, 4)", "depth": {"buy": [{"quantity": 6, "price": 972.15, "orders": 2}, {"quantity": 3, "price": 972.1, "orders": 2}, {"quantity": 15, "price": 972.05, "orders": 3}, {"quantity": 455, "price": 972.0, "orders": 16}, {"quantity": 14, "price": 971.95, "orders": 2}], "sell": [{"quantity": 6, "price": 972.5, "orders": 3}, {"quantity": 49, "price": 972.55, "orders": 2}, {"quantity": 10, "price": 972.6, "orders": 1}, {"quantity": 27, "price": 972.65, "orders": 2}, {"quantity": 10, "price": 972.7, "orders": 1}]}}]

Let me start off with THIS IS A BAD IDEA.

The comments on the question call out that this is not a JSON object in a file, rather the result of some other Python process printing the result and putting that in a file. The correct solution would be to modify the producer to use json.dumps() instead.

That aside, here’s a DANGEROUS way to read that source file into a Python object.

import datetime  # needed for `eval()`

with open('textfile.txt', 'r') as f:
    data = eval(f.read())

from pprint import pprint
pprint(data)

This will produce the following output from that input:

[{'average_traded_price': 180.1,
  'change': -1.2983425414364609,
  'depth': {'buy': [{'orders': 8, 'price': 178.6, 'quantity': 653},
                    {'orders': 15, 'price': 178.55, 'quantity': 2408},
                    {'orders': 22, 'price': 178.5, 'quantity': 6329},
                    {'orders': 24, 'price': 178.45, 'quantity': 9161},
                    {'orders': 17, 'price': 178.4, 'quantity': 7775}],
            'sell': [{'orders': 8, 'price': 178.7, 'quantity': 5726},
                     {'orders': 11, 'price': 178.75, 'quantity': 4099},
                     {'orders': 25, 'price': 178.8, 'quantity': 23951},
                     {'orders': 21, 'price': 178.85, 'quantity': 7446},
                     {'orders': 21, 'price': 178.9, 'quantity': 11379}]},
  'exchange_timestamp': datetime.datetime(2023, 1, 12, 13, 5, 1),
  'instrument_token': 4708097,
  'last_price': 178.65,
  'last_trade_time': datetime.datetime(2023, 1, 12, 13, 4, 58),
  'last_traded_quantity': 5,
  'mode': 'full',
  'ohlc': {'close': 181.0, 'high': 181.95, 'low': 177.8, 'open': 181.95},
  'oi': 0,
  'oi_day_high': 0,
  'oi_day_low': 0,
  'total_buy_quantity': 1282853,
  'total_sell_quantity': 1673842,
  'tradable': True,
  'volume_traded': 4581928},
 {'average_traded_price': 973.85,
  'change': 0.4959958667011061,
  'depth': {'buy': [{'orders': 2, 'price': 972.15, 'quantity': 6},
                    {'orders': 2, 'price': 972.1, 'quantity': 3},
                    {'orders': 3, 'price': 972.05, 'quantity': 15},
                    {'orders': 16, 'price': 972.0, 'quantity': 455},
                    {'orders': 2, 'price': 971.95, 'quantity': 14}],
            'sell': [{'orders': 3, 'price': 972.5, 'quantity': 6},
                     {'orders': 2, 'price': 972.55, 'quantity': 49},
                     {'orders': 1, 'price': 972.6, 'quantity': 10},
                     {'orders': 2, 'price': 972.65, 'quantity': 27},
                     {'orders': 1, 'price': 972.7, 'quantity': 10}]},
  'exchange_timestamp': datetime.datetime(2023, 1, 12, 13, 5, 4),
  'instrument_token': 871681,
  'last_price': 972.55,
  'last_trade_time': datetime.datetime(2023, 1, 12, 13, 4, 53),
  'last_traded_quantity': 1,
  'mode': 'full',
  'ohlc': {'close': 967.75, 'high': 978.6, 'low': 969.0, 'open': 971.75},
  'oi': 0,
  'oi_day_high': 0,
  'oi_day_low': 0,
  'total_buy_quantity': 152925,
  'total_sell_quantity': 214765,
  'tradable': True,
  'volume_traded': 411290}]

Again, I will restate, THIS IS A BAD IDEA.

Read more on the specifics of eval here: https://realpython.com/python-eval-function/

Please signup or login to give your own answer.

Click here to cancel reply.