pyspark Questions

Amazon web services – Pyspark error: " Class org.apache.hadoop.fs.s3a.S3AFileSystem not found" in EMR 7.0.0

November 15, 2024
TripleH
2 Answers

I am using EMR 7.0.0 version, which has python 3.9, spark 3.5.0, Hadoop 3.3.6 in AWS. I got the error: File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 740, in csv File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__ File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py",…

VIEW QUESTION

Amazon web services – List all the files which are inside a folder in a s3 bucket using pyspark

October 28, 2024
Divyansh Rathi
2 Answers

I have a s3 bucket "bucket1" inside which I have a directory named "dir1". Inside this directory there are multiple files. I just want to create a list of all the file names in this directory in my pyspark code…

VIEW QUESTION

Mongodb – Py4JJava Error on Azure Databricks notebook

I am trying to read data from mongodb in batch My compute cluster configurations are: Spark Version > 3.5.0 Scala Version > 2.12 I have installed mongodb connector library version is(through maven): org.mongodb.spark:mongo-spark-connector_2.12:10.4.0 I have my schemas defined in a…

VIEW QUESTION

How convert CSV file into hierarchical json

September 12, 2024
Julio
2 Answers

today i have a modification from another question that i did yesterday. The challenge is: How convert a CSV into a hierarchical json but the keys and values are different. It must be done with PYTHON (pandas and json library)…

VIEW QUESTION

How convert CSV table structure to JSON using Python?

September 11, 2024
Julio
2 Answers

Today I have a challenge at my school, it is convert a CSV file to a JSON file. This CSV has a table structure (which means contains the information from an oracle table in this example). So i have to…

VIEW QUESTION

Phpmyadmin – Pyspark stream kafka debezium topic Error format, ETL

August 21, 2024
NV0C
2 Answers

I have successfully created a mariadb database connection using debezium and kafka When I tried to stream the topic using pyspark this is the output that I get ------------------------------------------- Batch: 0 ------------------------------------------- +------+--------------------------------------------------------------------------------------------------------------------------+ |key |value | +------+--------------------------------------------------------------------------------------------------------------------------+ ||MaxDoe1.4.2.Finalnmysqlmariadbbtruebasecampemployees mysql-bin.000032�r�ȯݭd |…

VIEW QUESTION

Amazon web services – Join Two 100k table taking longer than half hours

August 16, 2024
TripleH
2 Answers

I am using pyspark to join two tables with 100k rows for each (so not skewed join). It takes longer than 30mins even an hour which I think something is wrong here. The code is just regular join a =…

VIEW QUESTION

Spark Code Completion in Visual Studio Code

August 13, 2024
Bob
2 Answers

I am trying to use Visual Studio Code for Spark development. My PySpark code all runs fine, but there is no code completion/hints. How can I add these features for VS Code? I have the Python and Pylance extensions installed.…

VIEW QUESTION

Dynamically infer schema of JSON data using Pyspark

August 6, 2024
RushHour
3 Answers

I have a MongoDB server from which I load the data in a PySpark Dataframe. However, the problem is due to some differences between the systems(I also tried legacy data types) I can't load the data directly from MongoDB. So,…

VIEW QUESTION

Azure – How to create a continuous sequence id irrespective of the runs in Databricks

July 24, 2024
Rocking Surya
2 Answers

I have a Databricks DataFrame with Columns : tno,data_value Output of first Databricks run: tno, data_value 1,hdjsjsjnsns 2,dhjdjdjsnsn 3,jdjsjsjsjsjjs When I run again the same notebook after some time it should generate as: tno, data_value 4,hdjsjsjnsns 5,dhjdjdjsnsn 6,jdjsjsjsjsjjs Just like…

VIEW QUESTION

Page 1
Page 2
Page 3
Page 4
…
Page 13
Next