Use json_normalize with extraneous dictionary layer with single key?

In the posted question, the dataframe dfAIS contains a Message field and a MetaData field, as well as a MessageType field that I described as redundant.

Each of the Message and MessageType fields contains a (possibly nested) dictionary. I want the fields of the inner most dictionaries to be the columns of the dataframe. Here is an example record of the source JSON file:

{"Message": {"ShipStaticData": {"AisVersion": 1,
                                "CallSign": "CQUR   ",
                                "Destination": "DELBC               ",
                                <...snip...>
                               }
            },
 "MessageType": "ShipStaticData",
 "MetaData": {"MMSI": 255814000,
              "MMSI_String": 255814000,
              "ShipName": "CORONA SEA          ",
              "latitude": 53.89848666666667,
              "longitude": 10.781958333333334,
              "time_utc": "2024-04-15 23:00:24.770932839 +0000 UTC"
             }
}

The ShipStaticData and MetaData fields above contain the two inner-most dictionaries whose fields I want to become the new columns of the dataframe. The challenging dictionary is ShipStaticData because (i) it is nested within a 1-item Message dictionary, i.e., more deeply than the MetaData dictionary and (ii) it goes by names other than ShipStaticData. I just want the fields and values of the dictionary -- I don't care whether it is called ShipStaticData, nor that it sometimes goes by different names. Nor do I care about the name MetaData -- I just want the dictionaries fields and values.

In my case, the JSON file contains multiple lines/records and furthermore, is gzipped. The first half of the code below reads all the records into a list of dictionaries while the 2nd half takes the fields of the ShipStaticData and MetaData dictionaries and turns the fields into dataframe columns:

AISmsgs = [] # List of dictionaries, 1 per record
fpINgz="data/20240415/ais_messages_185928.662241.json.gz"
with gzip.open(fpINgz,'rt',encoding='UTF-8') as zipfile:
   for JsonObj in zipfile:
      AISmsg1 = json.loads(JsonObj) # 1 record
      AISmsgs.append(AISmsg1)

dfMsg = pd.DataFrame([
   next(iter( rec1['Message'].values() )) | rec1['MetaData']
   for rec1 in AISmsgs # Each record
])

The | operator concatenates the ShipStaticData and MetaData dictionaries. Expressions rec1['Message'] and rec1['MetaData'] simply accesses the two dictionaries for each record. The Message dictionary requires more processing it is the inner ShipStaticData dictionary that I want:

• rec1['Message'] returns a dictionary in which ShipStaticData is the sole key
• rec1['Message'].values() returns the "value" for the ShipStaticData key, which itself is one of the two innermost dictionaries whose keys I want to turn into dataframe columns dictionary
• However, rec1['Message'].values() is of <class 'dict_values'>, "which does not support indexing [so I can't actually access the value]. But it's iterable, it can be converted to list, which supports indexing" (see here)

The following has the effect of converting rec1['Message'].values() to a single-item list containing the value corresponding to the key ShipStaticData, i.e., the dictionary whose fields I want to become dataframe columns:

list( AISmsgs[0]['Message'].values() )[0] # Simplest
next(iter( AISmsgs[0]['Message'].values() )) # Most efficient

Here is a sanity check of the correctness of converting data from the list of nested dictionaries AISmsgs to dataframe dfMsg based on the first record:

AISmsgs[0]

   {'Message': {'ShipStaticData': {'AisVersion': 1,
      'CallSign': 'CQUR   ',
      'Destination': 'DELBC               ',
      'Dimension': {'A': 26, 'B': 161, 'C': 13, 'D': 13},
      'Dte': False,
      'Eta': {'Day': 15, 'Hour': 16, 'Minute': 0, 'Month': 4},
      'FixType': 1,
      'ImoNumber': 9357597,
      'MaximumStaticDraught': 6.3,
      'MessageID': 5,
      'Name': 'CORONA SEA          ',
      'RepeatIndicator': 0,
      'Spare': False,
      'Type': 71,
      'UserID': 255814000,
      'Valid': True}},
    'MessageType': 'ShipStaticData',
    'MetaData': {'MMSI': 255814000,
     'MMSI_String': 255814000,
     'ShipName': 'CORONA SEA          ',
     'latitude': 53.89848666666667,
     'longitude': 10.781958333333334,
     'time_utc': '2024-04-15 23:00:24.770932839 +0000 UTC'}}

dfMsg.iloc[0]

   AisVersion                                          1.0
   CallSign                                        CQUR
   Destination                        DELBC
   Dimension         {'A': 26, 'B': 161, 'C': 13, 'D': 13}
   Dte                                               False
   Eta    {'Day': 15, 'Hour': 16, 'Minute': 0, 'Month': 4}
   FixType                                             1.0
   ImoNumber                                     9357597.0
   MaximumStaticDraught                                6.3
   MessageID                                           5.0
   Name                               CORONA SEA
   RepeatIndicator                                     0.0
   Spare                                             False
   Type                                               71.0
   UserID                                      255814000.0
   Valid                                              True
   MMSI                                          255814000
   MMSI_String                                   255814000
   ShipName                           CORONA SEA
   latitude                                      53.898487
   longitude                                     10.781958
   time_utc        2024-04-15 23:00:24.770932839 +0000 UTC

   ApplicationID                                       NaN
   ApplicationIDValid                                  NaN
   CommunicationState                                  NaN
    <...snip...>
   AltFromBaro                                         NaN
   Altitude                                            NaN
   Text                                                NaN
   Name: 0, dtype: object

At the very bottom is a long series of mysterious fields containing NaN. The first record in the source data file contains none of these fields. I tracked this down to the fact that not all of the records follow the data model shown at the top of this answer. To avoid this inconsistency, I need to find a way to add a column to the original data set containing MessageID, which I will filter on. That has strong prospects of leaving records that have the same data model.