I'm trying to filter out data from the current date to last 3 years and trying to use this for Pyspark dataframe. below sql query needs to convert into Pyspark dataframe format date >= dateadd(month,-4,current_date) How to write above sql…
I am building a data lake on Databricks. Basically, it consists of a pipeline of Databricks jobs, written in Python (and using PySpark for some). Now, I was wondering how could I test those notebooks. Integration test As I have…
I have use case where after inserting a row in mysql table i want to fetch back the id of the last inserted row. I wanted to run a MYSQL query in aws glue for the same. I am new…
i'm having trouble trying to read a json just like below: { "cats":{ "1":{ "name":"meow", "favorite_food":"milk" }, "2":{ "name":"honey", "favorite_food":"fish" }, "3":{ "name":"cutie", "favorite_food":"milk" }, "4":{ "name":"puss", "favorite_food":"chicken" }, "n"...... } } I want to output a table_cats like this:…
I have a json file just like below, and I need to read it and generate a table with the attributes of the person. { "person":[ [ "name", "Guy" ], [ "age", "25" ], [ "height", "2.00" ] ] }…
I have a function that takes a dataframe as a parameter and calculates the NULL value counts and NULL value percentage and returns a dataframe with column_name, null value count and null percentages. How can I register it as UDF…
DynamoDB Table: +------------------+---------+---------------------+ | id |Column A | Column B| +------------------+---------+---------------------+ |1 | "155.2"| 400 | |2 | 100| 200 | |3 | "455.2"| 305.5 | |4 | "312.3"| 350 | +------------------+---------+---------------------+ Notice that in Column A. We have numbers…
I have below JSON string as a column in a pyspark dataframe. { "result":{ "version":"1.2", "timeStamp":"2023-08-14 14:00:12", "description":"", "data":{ "DateTime_Received":"2023-08-14T14:01:10.4516457+01:00", "DateTime_Actual":"2023-08-14T14:00:12", "OtherInfo":null, "main":[ { "Status":0, "ID":111, "details":null } ] }, "tn":"aaa" } } I want to explode the above one…
I have below pyspark row type data: indv_msg = [Row(cbm_json_output=Row(country_code='USA', date='06-10-2023', date_epoch='1696550400', id='USA-001535-1696550400', interfaceVersion='1.0.0', opmode_car_door=Row(health_category='GREEN', msg_id='1', num_yellow_preds_in_last_14_days=0, reason=None, reasonDetail=None), opmode_landing_door=Row(health_category='GREEN', msg_id='1', reason=None, reasonDetail=None), sensor=Row(component_type=None, health_category=None, landing_priority=None, msg_id='1', num_yellow_preds_in_last_14_days=None, reason=None, reasonDetail=None), unit_id='001535')) While trying to convert to json string, it is…
I am trying to import python files from databricks notebook. I am able to import the function but it keep giving me error - NameError: name 'col' is not defined. I have a file /Workspace/Shared/Common/common.py. The code in that file…