How to convert a pyspark data frame like below to a json array structure OrderID field fieldValue itemSeqNo 123 Date 01-01-23 1 123 Amount 10.00 1 123 description Pencil 1 123 Date 01-02-23 2 123 Amount 11.00 2 123 description…
In Pyspark, I have 2 dataframe. 1st dataframe say df1 is empty dataframe created from a schema. 2nd dataframe df2 is non-empty dataframe filled from a csv file. Now I want to merge such that all below scenarios are covered.…
I'm trying to filter out data from the current date to last 3 years and trying to use this for Pyspark dataframe. below sql query needs to convert into Pyspark dataframe format date >= dateadd(month,-4,current_date) How to write above sql…
I am building a data lake on Databricks. Basically, it consists of a pipeline of Databricks jobs, written in Python (and using PySpark for some). Now, I was wondering how could I test those notebooks. Integration test As I have…
I have use case where after inserting a row in mysql table i want to fetch back the id of the last inserted row. I wanted to run a MYSQL query in aws glue for the same. I am new…
i'm having trouble trying to read a json just like below: { "cats":{ "1":{ "name":"meow", "favorite_food":"milk" }, "2":{ "name":"honey", "favorite_food":"fish" }, "3":{ "name":"cutie", "favorite_food":"milk" }, "4":{ "name":"puss", "favorite_food":"chicken" }, "n"...... } } I want to output a table_cats like this:…
I have a json file just like below, and I need to read it and generate a table with the attributes of the person. { "person":[ [ "name", "Guy" ], [ "age", "25" ], [ "height", "2.00" ] ] }…
I have a function that takes a dataframe as a parameter and calculates the NULL value counts and NULL value percentage and returns a dataframe with column_name, null value count and null percentages. How can I register it as UDF…
DynamoDB Table: +------------------+---------+---------------------+ | id |Column A | Column B| +------------------+---------+---------------------+ |1 | "155.2"| 400 | |2 | 100| 200 | |3 | "455.2"| 305.5 | |4 | "312.3"| 350 | +------------------+---------+---------------------+ Notice that in Column A. We have numbers…
I have below JSON string as a column in a pyspark dataframe. { "result":{ "version":"1.2", "timeStamp":"2023-08-14 14:00:12", "description":"", "data":{ "DateTime_Received":"2023-08-14T14:01:10.4516457+01:00", "DateTime_Actual":"2023-08-14T14:00:12", "OtherInfo":null, "main":[ { "Status":0, "ID":111, "details":null } ] }, "tn":"aaa" } } I want to explode the above one…