I am working on a project in Databricks using Apache Spark, I was doing some data manipulation, during which I encountered this error basically stating "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE". The code snippet is as follows: player_match_df = player_match_df.withColumn( "years_since_debut", (year(current_date()) - (col("season_year"))) )…
I am connecting to an on prem SQL Server database using the code below and it works as expected. I have wrapped the query I want to use in a view and I called the view to read the data…
Problem Description: We tried to start Apache Zeppelin 0.11.1 on the following server. We followed the installation instructions on the official Apache Zepplin website. Apache Zeppelin can be started and the notebooks open. When executing a paragraph, the attached error…
I would like to process the data 24/7 using databricks structured streaming. But before I process all the data I need to apply some transformations on it. therefore I am using foreach. Below is my def from pyspark.sql import DataFrame,…
I have followed the official documentation to set up Apache Spark on my local Windows 11 machine. This setup includes: Proper installation of Apache Spark, setting up the env variables etc. Creation of a virtual env specifically for Python 3.9…
Let assume I have two parquet file aka main and updates with some data as shown below. I need to implement upsert kind of operation in duck db. parquet file main: id, name, city data: id name city 1 a…
I have a column with data coming in as an string representation of an array I tried to type cast it to an array type but the data is getting modified. I tried to use regex as well to remove…
I am trying to create a Apache Spark docker container, mainly for self learning purposes. So, I know (not enough details) but there are vulnerability issues with log4j1.2.17 jar. Having said that, I have the following Dockerfile(some contents have been…
I am trying to run a query to read data from redshift table through AWS Glue job. When it reads data from table, it creating dataframe with only 1 partition and it is taking lot of time to read the…
I have 2 JSON files, one is like: { "a":{ "a1":"xxx" }, "b":"xxx" } Another one is like: { "a":{ "a1":"xxx", "a2":"xxx" }, "b":"xxx" } And I want to read these two JSON files into one Dataframe in Spark. I…