pyspark Questions

Converting pyspark.sql.Rowtype data to Json string eliminating values in Azure Databricks NB

October 9, 2023
Saswat Ray
2 Answers

I have below pyspark row type data: indv_msg = [Row(cbm_json_output=Row(country_code='USA', date='06-10-2023', date_epoch='1696550400', id='USA-001535-1696550400', interfaceVersion='1.0.0', opmode_car_door=Row(health_category='GREEN', msg_id='1', num_yellow_preds_in_last_14_days=0, reason=None, reasonDetail=None), opmode_landing_door=Row(health_category='GREEN', msg_id='1', reason=None, reasonDetail=None), sensor=Row(component_type=None, health_category=None, landing_priority=None, msg_id='1', num_yellow_preds_in_last_14_days=None, reason=None, reasonDetail=None), unit_id='001535')) While trying to convert to json string, it is…

VIEW QUESTION

Azure – Import Python file in databricks notebook

October 6, 2023
Harshit Mahajan
2 Answers

I am trying to import python files from databricks notebook. I am able to import the function but it keep giving me error - NameError: name 'col' is not defined. I have a file /Workspace/Shared/Common/common.py. The code in that file…

VIEW QUESTION

Docker – Which combination of pyspark and delta versions should I use?

October 2, 2023
Boris
2 Answers

I'm using the jupyter/pyspark-notebook docker image to develop a spark script. My dockerfile looks like this: FROM jupyter/pyspark-notebook USER root COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt && rm requirements.txt # this is a default user and the…

VIEW QUESTION

Concat two JSON string columns with PySpark

September 26, 2023
joao pedro imamura
2 Answers

Suppose I have two columns (StringType): JSON 1 JSON 2 {"key1":"value1"} {"key2":"value2"} {"key3":"value3"} {"key4":"value4"} I need to concat these columns to have a LIST of JSON in the third column (also string), something like this: JSON 1 JSON 2 JSON…

VIEW QUESTION

How to access a subcolumn in JSON file in Pandas/Spark

September 25, 2023
Another.Chemist
2 Answers

I have a JSON file and I want to access the "country" column What I have done so far is resumed in this image:

VIEW QUESTION

Azure – Optimizing Delta Lake Partitioning for Timestamp Range Queries

September 21, 2023
elyptikus
2 Answers

I'm working with a Delta table in Databricks that is partitioned by a Year column, and I have a timestamp column in the table. I want to optimize my queries to fetch data within a specific timestamp range (e.g., between…

VIEW QUESTION

Fetch the data from mysql db and run the data in pyspark

September 20, 2023
snk
2 Answers

i had a table named xyz in mysql Aurora DB, which has Table_Name, Table_ID,QuerySeq,sql_query,view_name. Now i need to read the table in pyspark, run the sql queries(using spark.sql) present in sql_query according to QuerySeq and create a view for the…

VIEW QUESTION

Azure – PySpark Tabula-Py Read_PDF (ERROR: No module named 'org.apache.commons')

September 18, 2023
mohamadmaarouf_
2 Answers

I've been runnning a pipeline in Azure for 4 months and it suddenly broke last night. I have the following code: !pip install tabula-py from tabula.io import read_pdf import tabula df = tabula.io.read_pdf(BytesIO(pdf_content), pandas_options={'header': None}, pages=3, stream=True)[0] I got this…

VIEW QUESTION

Amazon web services – How to control databricks autoscaling from the driver node

August 31, 2023
The indian guy
2 Answers

I am using databricks for a specific workload. This workload involves an approx of 10 to 200 dataframes that are read and written to a storage location. This workload can benefit from parallelism. The constraint i have is cost optimization.…

VIEW QUESTION

Mongodb – Load dataframe using spark error: Cannot cast ARRAY into a StructType

August 31, 2023
Mai Nhật Nam
2 Answers

When I load data from MongoDB into a data frame using spark, there is one field with ObjecType and ArrayType whose value is null due to missing data, when I df.show() I get an error: Cannot cast ARRAY into a…

VIEW QUESTION