I have a column with data coming in as an string representation of an array I tried to type cast it to an array type but the data is getting modified. I tried to use regex as well to remove…
within a UDF I want to read a delta table into a dataframe, based on its content update the row of the actual dataframe, on which the UDF is applied, and then update the delta table. I would use the…
I have multiple zip files called f1.zip, f2.zip, ... f7.zip each contain around 200k xml files and I am using this code to multiprocess to unzip them in parallel, but even if each file is very small, the read/write process…
I am trying to run a query to read data from redshift table through AWS Glue job. When it reads data from table, it creating dataframe with only 1 partition and it is taking lot of time to read the…
i want to send this simple pyspark dataframe to kafka and i done everything but i always get an error. i try it with simple python producer script and it works and the pyspark read stream work my only problem…
I have 2 JSON files, one is like: { "a":{ "a1":"xxx" }, "b":"xxx" } Another one is like: { "a":{ "a1":"xxx", "a2":"xxx" }, "b":"xxx" } And I want to read these two JSON files into one Dataframe in Spark. I…
Context : Sorry to bother you with I am struggling to install a Cassandra-Spark connector. My goal is to install it to use Spark-SQL since Cassandra has strong limitations to do requests. I have : Spark 3.1.1, Cassandra 4.0.11 Scala…
I have a Spark Dataframe which contains in every single row two items: a file name (with an extension, like for instance .jpg), and the content of the file in bytes. I would like to write a process that takes…
I tried different ways to get this done, but unable to do so. Please help here. I am trying to explode a nested json list using python pd.json_normalize. After exploding the output is coming in two rows causing duplicate PKeys.…
in Azure databricks, need to connect to a table in odbc sql server and create a dataframe and create exactly the same table in databricks I have already completed the below codes and I can see as attached. How to…