I have a JSON string substitutions as a column in dataframe which has multiple array elements that I want to explode and create a new row for each element present in that array. There are other columns present in the…
I am new to PySpark and I am struggling with flattening a nested json file into a PySpark data frame. I need to define the schema for the JSON data. I know how to define schema for regular json files…
My Data set looks like this : I am using this filter : df = df.filter(trim(col("AGE"))!="" & trim(col("PHONE"))!="") I am getting an empty dataframe, I want the data without the record having name =G3 Any help is appreciated.
I have an Excel file in the azure datalake ,I have read the excel file like the following ddff=spark.read.format("com.crealytics.spark.excel") .option("header", "true") .option("sheetName","__all__") .option("inferSchema","true") .load("abfss://[email protected]/file.xlsx") Now Iam confused how to get just the sheetnames from that Excel file,is there any direct…
I have been able to write to console the json file I want to work on to console. Please, how do I separate the 'value' column into columns of data as in the json and write to delta lake for…
I have a DF column which is a long strings with comma separated values, like: 2000,2001,2002:a,2003=b,2004,100,101,500,20 101,102,20 What I want to do is to create a new Array<Int> column out of it where: only values starting with 2 are included…
I have a .csv file (in Azure Data Lake Storage), which looks approximately like this -> I want to create a notebook (PySpark (Python)), which could be implemented in the synapse analytics (integrate -> pipeline) in one of the pipelines.…
I want to parse a JSON request and create multiple columns out of it in pyspark as follows: { "ID": "abc123", "device": "mobile", "Ads": [ { "placement": "topright", "Adlist": [ { "name": "ad1", "subtype": "placeholder1", "category": "socialmedia", }, { "name":…
What is the best way to explode a comma separated column based on specific conditions? I have some data in the following format: ID col1 1 a100,a101,b100,c100 2 a105,b100 3 b101, c104 what I want to achieve is to: grab…
I have a JSON file with a field named "AdditionalFields" as below- "additionalFields": [ { "fieldName":"customer_name", "fieldValue":"ABC" }, { "fieldName":"deviceid", "fieldValue":"1234" }, { "fieldName":"txn_id", "fieldValue":"2" }, { "fieldName":"txn_date", "fieldValue":"2017-08-14T18:17:37" }, { "fieldName":"orderid", "fieldValue":"I126101" } ] How to parse this as…