My Data set looks like this : I am using this filter : df = df.filter(trim(col("AGE"))!="" & trim(col("PHONE"))!="") I am getting an empty dataframe, I want the data without the record having name =G3 Any help is appreciated.
i was able to get row values from delta table using foreachWriter in spark-shell and cmd but while writing the same code in azure databricks it doesn't work. val process_deltatable=read_deltatable.writeStream.foreach(new ForeachWriter[Row] { def process(value: Row):Unit ={ val Telemetery= value.getString(0) println(Telemetery)…
... <property> <name>hive.metastore.warehouse.dir</name> <value>hdfs://spark-master-01:9000/skybluelee/skybluelee_warehouse_mysql_5.7</value> <description>location of default database for the warehouse</description> </property> ... the code is a part of /user/spark3/conf/hive-site.xml At first the value was hdfs://spark-master-01:9000/kikang/skybluelee_warehouse_mysql_5.7 And I changed the value hdfs://spark-master-01:9000/skybluelee/skybluelee_warehouse_mysql_5.7 Below there is a code and result println(spark.conf.get("spark.sql.warehouse.dir"))…
I have a spark dataset that has fields: "identifier_id", "inner_blob" , "json_blob" "inner_blob": { "identifier_id": 2.0, "name": "test1", "age": 30.0 }, "identifier_id": 2.0, "json_blob": { "identifier_id": 2.0, "order_id": 2.0, "inner_blob": [ { "item_id": 23.0, "item_name": "airpods2", "item_price": 300.0 }, {…
I have been able to write to console the json file I want to work on to console. Please, how do I separate the 'value' column into columns of data as in the json and write to delta lake for…
I have a DF column which is a long strings with comma separated values, like: 2000,2001,2002:a,2003=b,2004,100,101,500,20 101,102,20 What I want to do is to create a new Array<Int> column out of it where: only values starting with 2 are included…
I want to parse a JSON request and create multiple columns out of it in pyspark as follows: { "ID": "abc123", "device": "mobile", "Ads": [ { "placement": "topright", "Adlist": [ { "name": "ad1", "subtype": "placeholder1", "category": "socialmedia", }, { "name":…
What is the best way to explode a comma separated column based on specific conditions? I have some data in the following format: ID col1 1 a100,a101,b100,c100 2 a105,b100 3 b101, c104 what I want to achieve is to: grab…
here is my dataset processed_on|status | ------------+-------+ 2023-01-01 |Success| 2023-01-02 |Success| 2023-01-03 |Success| 2023-01-04 |Fail | 2023-01-05 |Fail | 2023-01-06 |Success| 2023-01-07 |Fail | 2023-01-08 |Success| 2023-01-09 |Success| expected output is -------------------------- start_date|end_date|status -------------------------- 2023-01-01|2023-01-03|Success 2023-01-04|2023-01-05|Fail 2023-01-06|2023-01-06|Success 2023-01-07|023-01-07|Fail 2023-01-08|2023-01-09|Success i…
I have a scala spark code that writes a json file with file name as part-*.json for example (part-00000-14732361-f017-468a-b948-22d3b6d460dc-c000.json). I want to do s3.doesObjectExist(buckey, key) where bucket = xyz and key = abc/def/part-*.json. Looks like s3 doesn't support wildcard search.…