Question 109410 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

convert list of multiple json files into a dataframe pandas

This how I parsed multiple json files in a single list

base_dir = 'jsons_final_folder/'
data_list = []
for file in os.listdir(base_dir):

    if 'json' in file:
        json_path = os.path.join(base_dir, file)
        json_data = pd.read_json(json_path, lines=True)
        data_list.append(json_data)

And I got a list that look like this

print(data_list)

output:

[                                                   0

0  {"general":{"key":"value","q":"...,                                          0

0  {"general":{"key":"value","q":"...,                                          0

0  {"general":{"key":"value","q":"...,                                          0

0  {"general":{"key":"value","q":"...,                                          0

0  {"general":{"key":"value","q":"...,                                          0

0  {"general":{"key":"value","q":"...,                                          0

0  {"general":{"key":"value","q":"...,]                                         0

So this is my code to convert df

with open("f.csv","w") as f:
    wr = csv.writer(f)
    wr.writerow(data_list)

But I get a df that type pandas.core.frame.DataFrame like this:

{"general":{"key":"value","q":"…, {"general":{"key":"value","q":"…, {"general":{"key":"value","q":"…, {"general":{"key":"value","q":"…,

with shape of n columns and 0 rows

What I am trying to do here is to make a df out of this list that contains only jsons with specific queries but i don’t what’s the problem.

I also tried to add dilimiter

I wanted the final shape be look like this

json
{"general":{"key":"value","q":"…,
{"general":{"key":"value","q":"…,

Thank you

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Guys I found the solution from This video

    function to return files

    def get_files(filepath):
   all_files = []
   for root, dirs, files in os.walk(filepath):
      files = glob.glob(os.path.join(root,'*.json'))
      for f in files:
          all_files.append(os.path.abspath(f))
   return all_files


j_files = get_files("../path goes here/")

    Here we read each file and import it into list

    j_files_list = []
for j_file in j_files:
    with open(j_file) as doc:
        exp = json.load(doc)
        j_files_list.append(exp)

    here we convert it into a df

    df = pd.DataFrame(j_files_list)

    And then save list to csv

    df.to_csv('json_files_in_df.csv')

    Thanks for helping out


  2. 0

    Have you tried df = pd.DataFrame({'json': data_list}) ?

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search