I have a Databricks DataFrame with Columns : tno,data_value Output of first Databricks run: tno, data_value 1,hdjsjsjnsns 2,dhjdjdjsnsn 3,jdjsjsjsjsjjs When I run again the same notebook after some time it should generate as: tno, data_value 4,hdjsjsjnsns 5,dhjdjdjsnsn 6,jdjsjsjsjsjjs Just like…
I'm working on a SQL solution to squeeze the number of rows when there are matching "start" and "close" values across records. Here is an example input and the required output Input name start close A 120 130 A 130…
I am trying to run a query to read data from redshift table through AWS Glue job. When it reads data from table, it creating dataframe with only 1 partition and it is taking lot of time to read the…
I'm trying to find a explode_percent column by dividing a column to sum of the bucket. Below is my query - SELECT amount, sum(amount) over ( partition by col1, col2, col3, col4 order by col2, col3, col4 ) as explode_amount,…
"events": { "a": { "eventName": "a", "eventIdentifier": "abc" }, "b": { "eventName": "b", "eventIdentifier": "xyz" } } I want to split events "a" and "b" into 2 different records using spark sql only and above json should not in array.…
I am trying to add a new column called timestamp in AWS Glue. My upstream data keys have capital letters. However, after adding the column timestamp, the keys of the remaining columns got lowercased. Experiment 1: Transform - SQL Query…
I have this tableConfig.json inside a ADLS container based location . It has table specific details. { "tableName":"employee", "databaseName": "dbo", "location" : "/mnt/clean/demo", "colsList" : ["emp_id","emp_name","emp_city"] } Now I want to read that tableConfig.json in azure databricks python notebooks. and…
Using PySpark 2.4.7 My goal is to write PySpark DataFrame into specific number of parquet files into AWS S3. Say I want to write PySpark DataFrame into 10 parquet files. This is my approach df.repartition(10).write.mode("append").partitionBy(partition_cols).saveAsTable(<db.table>) This writes 10 parquet files…
I want to read the Parquet files I get from the readSteam() method from a Kafka topic. However, when I read these Parquet files, I get a dataframe that I don't want instead of real values. import glob parquet_files =…
I have a JSON file of the form: { 42 : {"name": "MeowBark", "id": 42, "category": "pet store"}, 67 : {"name": "Chef's Kiss", "id": 67, "category": "restaurant"}, } I have to parse this in Pyspark and am using the following…