Suppose I have two columns (StringType): JSON 1 JSON 2 {"key1":"value1"} {"key2":"value2"} {"key3":"value3"} {"key4":"value4"} I need to concat these columns to have a LIST of JSON in the third column (also string), something like this: JSON 1 JSON 2 JSON…
I am using databricks for a specific workload. This workload involves an approx of 10 to 200 dataframes that are read and written to a storage location. This workload can benefit from parallelism. The constraint i have is cost optimization.…
When I load data from MongoDB into a data frame using spark, there is one field with ObjecType and ArrayType whose value is null due to missing data, when I df.show() I get an error: Cannot cast ARRAY into a…
I'm looking to use spark in a standalone VS code java project (ie no maven or gradle). Is this possible? I've download library from https://spark.apache.org/downloads.html, extracted and copied the contents of Jars folder into my vs code lib folder. I…
I am using the Databricks VSCode extension for development in an IDE. The basic functionalities are all working well. I connected to an Azure Databricks workspace with Unity Catalog enabled, selected an active cluster (DBR 13.2) and configured the sync…
I have a JSON file and I need to convert it into tabular form by using only Pyspark. My JSON file :- { "records": [ { "name": "Priya", "last_name": "Munjal", "special_values": [ { "name": "adress", "value": "some adress" }, {…
I am new to Pyspark and trying to flatten JSON file using Pyspark but not getting desired output. Here is my JSON file :- { "events": [ { "event_name": "start", "event_properties": ["property1", "property2", "property3"], "entities": ["entityI", "entityII", "entityIII"], "event_timestamp": "2022-05-01…
I need to flatten JSON file so that I can get output in table format.Ihavetried but not getting the output that I want This is my JSON file :- { "records": [ { "name": "A", "last_name": "B", "special_values": [ {…
I have a Spark application and I want to access an Azure Blob container by writing the event log into the blob container. I want to authenticate using a SAS token. The SAS token generated by the Azure portal works…
I'm new to Spark and I'm running spark job on EMR cluster. It takes about one hour for job to finish. In job, there is a code for reading the file from S3 (file size is 10MB) and writing it's…