My Data set looks like this : I am using this filter : df = df.filter(trim(col("AGE"))!="" & trim(col("PHONE"))!="") I am getting an empty dataframe, I want the data without the record having name =G3 Any help is appreciated.
I have an Excel file in the azure datalake ,I have read the excel file like the following ddff=spark.read.format("com.crealytics.spark.excel") .option("header", "true") .option("sheetName","__all__") .option("inferSchema","true") .load("abfss://[email protected]/file.xlsx") Now Iam confused how to get just the sheetnames from that Excel file,is there any direct…
I have been able to write to console the json file I want to work on to console. Please, how do I separate the 'value' column into columns of data as in the json and write to delta lake for…
I have a DF column which is a long strings with comma separated values, like: 2000,2001,2002:a,2003=b,2004,100,101,500,20 101,102,20 What I want to do is to create a new Array<Int> column out of it where: only values starting with 2 are included…
I have a .csv file (in Azure Data Lake Storage), which looks approximately like this -> I want to create a notebook (PySpark (Python)), which could be implemented in the synapse analytics (integrate -> pipeline) in one of the pipelines.…
I want to parse a JSON request and create multiple columns out of it in pyspark as follows: { "ID": "abc123", "device": "mobile", "Ads": [ { "placement": "topright", "Adlist": [ { "name": "ad1", "subtype": "placeholder1", "category": "socialmedia", }, { "name":…
What is the best way to explode a comma separated column based on specific conditions? I have some data in the following format: ID col1 1 a100,a101,b100,c100 2 a105,b100 3 b101, c104 what I want to achieve is to: grab…
I have a JSON file with a field named "AdditionalFields" as below- "additionalFields": [ { "fieldName":"customer_name", "fieldValue":"ABC" }, { "fieldName":"deviceid", "fieldValue":"1234" }, { "fieldName":"txn_id", "fieldValue":"2" }, { "fieldName":"txn_date", "fieldValue":"2017-08-14T18:17:37" }, { "fieldName":"orderid", "fieldValue":"I126101" } ] How to parse this as…
I'm starting a project to adjust the data lake for the specific purge of data, to comply with data privacy legislation. Basically the owner of the data opens a call requesting the deletion of records for a specific user, and…
I have a json string column in a dataframe that looks like this. {"columns":["ApplicationNum","eads59Us01S","HouseDeal_flag","Liability_Asset_Ratio","CBRAvailPcnt","CMSFairIsaacScore","OweTaxes_or_IRAWithdrawalHistry","eads14Fi02S","GuarantorCount","CBRRevMon","CBRInstalMon","CMSApprovedToRequested","SecIncSource","eads59Us01S_4","Liability_Asset_Ratio_40_90","CBRAvailPcnt_20_95","CMSFairIsaacScore_Fund","eads14Fi02S_2","InstalMonthlyPayments_400_3k","RevolvingMonthlyPayments_1k_cap","ApprovedToRequested_0_100","NoSecIncome","coef_eads59Us01S_4","coef_HouseDeal_flag","coef_Liability_Asset_Ratio_40_90","coef_CBRAvailPcnt_20_95","coef_CMSFairIsaacScore_Fund","coef_OweTaxes_or_IRAWithdrawalHistry","coef_eads14Fi02S_2","coef_GuarantorCount","coef_RevolvingMonthlyPayments_1k_cap","coef_InstalMonthlyPayments_400_3k","coef_ApprovedToRequested_0_100","coef_NoSecIncome","coef_Intercept"],"data":[[569325.0,2,0.0,1,92,825,0.0,4,1.0,74,854,0.51,2,2.0,0.9,92.0,825.0,4.0,854.0,1000.0,0.51,0.0,0.11716245,0.299528064,0.392119645,-0.010826643,-0.004957868,0.339407077,0.061509795,0.3685047,0.000167603,0.000225742,0.902205454,-0.371734864,2.788087559]]} I have a columns tag in there with a list of column values, and a data tag in there with the corresponding list of values for…