I need to flatten JSON file so that I can get output in table format.Ihavetried but not getting the output that I want This is my JSON file :- { "records": [ { "name": "A", "last_name": "B", "special_values": [ {…
I have a Spark application and I want to access an Azure Blob container by writing the event log into the blob container. I want to authenticate using a SAS token. The SAS token generated by the Azure portal works…
I'm new to Spark and I'm running spark job on EMR cluster. It takes about one hour for job to finish. In job, there is a code for reading the file from S3 (file size is 10MB) and writing it's…
I have a view in databricks and in one column the content is a json, example: [{"ECID":100017056,"FIRST_NAME":"Ioannis","LAST_NAME":"CHATZIZYRLIS","TITLE":"Mr","GENDER":"M","DATE_OF_BIRTH":"1995-04-14","PLACE_OF_BIRTH":"Greece","COUNTRY_OF_BIRTH":"GR","NATIONALITY":"GR","RESIDENCE":"GR","ADDRESS":[{"TYPE":1,"STREET_1":"Gizi 6","CITY":"Agios Doe","POSTAL_CODE":"111 34","COUNTRY":"GR","MOBILE":"0000000","EMAIL":" "}]}] I want to retrieve the content of that column so that I can post it to an api. I…
Using PySpark 2.4.7 My goal is to write PySpark DataFrame into specific number of parquet files into AWS S3. Say I want to write PySpark DataFrame into 10 parquet files. This is my approach df.repartition(10).write.mode("append").partitionBy(partition_cols).saveAsTable(<db.table>) This writes 10 parquet files…
I have created a Glue catalog in my account. It has 1 DB and 1 table. I followed this guide from AWS and created my EMR cluster. However, when I run spark-shell and try to access Glue catalog, I am…
I want to read the Parquet files I get from the readSteam() method from a Kafka topic. However, when I read these Parquet files, I get a dataframe that I don't want instead of real values. import glob parquet_files =…
I have a log in below format(json/ion) where the value of time_of_initial_request is in timestamp format but without double quotes. {post_handler_type:"trace", default_marketplace:"prod_iad", request_id:"5MTB3656X0JRL38R6LJY", rest_uri:"productApi/feature", client_logging_id:"PredictionMetadata", time_of_initial_request:2023-05-24T15:00:00.577Z } Since the field(key) name does not have quotes, I am using the allowUnquotedFieldNames…
I have a mongo db with multiple collections (let's say 3). I want to join them all so that I can do some aggregations. I was able to connect to a local db and one collection using documentation found oneline…
I have a JSON file in the following structure: { "10712": { "id": "10712", "age": 27, "gender": "male" }, "217": { "id": "217", "age": 60, "gender": "female" } } This causes a problem when importing using spark.read.json, because of the…