I would like to process the data 24/7 using databricks structured streaming. But before I process all the data I need to apply some transformations on it. therefore I am using foreach. Below is my def from pyspark.sql import DataFrame,…
I have followed the official documentation to set up Apache Spark on my local Windows 11 machine. This setup includes: Proper installation of Apache Spark, setting up the env variables etc. Creation of a virtual env specifically for Python 3.9…
Let assume I have two parquet file aka main and updates with some data as shown below. I need to implement upsert kind of operation in duck db. parquet file main: id, name, city data: id name city 1 a…
I have a column with data coming in as an string representation of an array I tried to type cast it to an array type but the data is getting modified. I tried to use regex as well to remove…
I am trying to create a Apache Spark docker container, mainly for self learning purposes. So, I know (not enough details) but there are vulnerability issues with log4j1.2.17 jar. Having said that, I have the following Dockerfile(some contents have been…
I am trying to run a query to read data from redshift table through AWS Glue job. When it reads data from table, it creating dataframe with only 1 partition and it is taking lot of time to read the…
I have 2 JSON files, one is like: { "a":{ "a1":"xxx" }, "b":"xxx" } Another one is like: { "a":{ "a1":"xxx", "a2":"xxx" }, "b":"xxx" } And I want to read these two JSON files into one Dataframe in Spark. I…
Context : Sorry to bother you with I am struggling to install a Cassandra-Spark connector. My goal is to install it to use Spark-SQL since Cassandra has strong limitations to do requests. I have : Spark 3.1.1, Cassandra 4.0.11 Scala…
i'm trying to play around with the platforms, creating some sort of data lake/warehouse. So i have the minio instance on localhost:9000, and can i use the ui and upload files. great. I create the network named spark network and…
The task I would like to solve: I have a constant influx of files in a specific folder held on azure storage. I would like to periodically list the files in this folder in order to copy them to a…