My Data set looks like this : I am using this filter : df = df.filter(trim(col("AGE"))!="" & trim(col("PHONE"))!="") I am getting an empty dataframe, I want the data without the record having name =G3 Any help is appreciated.
... <property> <name>hive.metastore.warehouse.dir</name> <value>hdfs://spark-master-01:9000/skybluelee/skybluelee_warehouse_mysql_5.7</value> <description>location of default database for the warehouse</description> </property> ... the code is a part of /user/spark3/conf/hive-site.xml At first the value was hdfs://spark-master-01:9000/kikang/skybluelee_warehouse_mysql_5.7 And I changed the value hdfs://spark-master-01:9000/skybluelee/skybluelee_warehouse_mysql_5.7 Below there is a code and result println(spark.conf.get("spark.sql.warehouse.dir"))…
I have a spark dataset that has fields: "identifier_id", "inner_blob" , "json_blob" "inner_blob": { "identifier_id": 2.0, "name": "test1", "age": 30.0 }, "identifier_id": 2.0, "json_blob": { "identifier_id": 2.0, "order_id": 2.0, "inner_blob": [ { "item_id": 23.0, "item_name": "airpods2", "item_price": 300.0 }, {…
I want to parse a JSON request and create multiple columns out of it in pyspark as follows: { "ID": "abc123", "device": "mobile", "Ads": [ { "placement": "topright", "Adlist": [ { "name": "ad1", "subtype": "placeholder1", "category": "socialmedia", }, { "name":…
What is the best way to explode a comma separated column based on specific conditions? I have some data in the following format: ID col1 1 a100,a101,b100,c100 2 a105,b100 3 b101, c104 what I want to achieve is to: grab…
Say I have a table, table1. And table1 has 2 columns, "Vegetables" and "Fruits" Using the values from table 1, how can I get the distinct of these 2 columns and use them to create a new column in a…
we are trying to read a table from Bigquery to spark dataframe. Strucute of the table is Following pyspark code is used for reading the data. from google.oauth2 import service_account from google.cloud import bigquery import json import base64 as bs…
I have a json that looks like this: [ { "event_date": "20221207", "user_properties": [ { "key": "user_id", "value": { "set_timestamp_micros": "1670450329209558" } }, { "key": "doc_id", "value": { "set_timestamp_micros": "1670450329209558" } } ] }, { "event_date": "20221208", "user_properties": [ {…
I have a Main pipeline in Synapse workspace which has 2 activities: 1st - Notebook activity 2nd - If Condition activity For the 1st one (Synapse notebook, spark pool, pyspark), I have a SQL cell like the following: It has…
When I am trying to create a sparksession I get this error: spark = SparkSession.builder.appName("Practice").getOrCreate() py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM This is my code: import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder.appName("Practice").getOrCreate() What am I doing…