i'm trying to play around with the platforms, creating some sort of data lake/warehouse. So i have the minio instance on localhost:9000, and can i use the ui and upload files. great. I create the network named spark network and…
The task I would like to solve: I have a constant influx of files in a specific folder held on azure storage. I would like to periodically list the files in this folder in order to copy them to a…
Suppose I have two columns (StringType): JSON 1 JSON 2 {"key1":"value1"} {"key2":"value2"} {"key3":"value3"} {"key4":"value4"} I need to concat these columns to have a LIST of JSON in the third column (also string), something like this: JSON 1 JSON 2 JSON…
I am using databricks for a specific workload. This workload involves an approx of 10 to 200 dataframes that are read and written to a storage location. This workload can benefit from parallelism. The constraint i have is cost optimization.…
When I load data from MongoDB into a data frame using spark, there is one field with ObjecType and ArrayType whose value is null due to missing data, when I df.show() I get an error: Cannot cast ARRAY into a…
I'm looking to use spark in a standalone VS code java project (ie no maven or gradle). Is this possible? I've download library from https://spark.apache.org/downloads.html, extracted and copied the contents of Jars folder into my vs code lib folder. I…
I am using the Databricks VSCode extension for development in an IDE. The basic functionalities are all working well. I connected to an Azure Databricks workspace with Unity Catalog enabled, selected an active cluster (DBR 13.2) and configured the sync…
I have a JSON file and I need to convert it into tabular form by using only Pyspark. My JSON file :- { "records": [ { "name": "Priya", "last_name": "Munjal", "special_values": [ { "name": "adress", "value": "some adress" }, {…
I am new to Pyspark and trying to flatten JSON file using Pyspark but not getting desired output. Here is my JSON file :- { "events": [ { "event_name": "start", "event_properties": ["property1", "property2", "property3"], "entities": ["entityI", "entityII", "entityIII"], "event_timestamp": "2022-05-01…
I need to flatten JSON file so that I can get output in table format.Ihavetried but not getting the output that I want This is my JSON file :- { "records": [ { "name": "A", "last_name": "B", "special_values": [ {…