Amazon Web Services Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – Pyspark error: " Class org.apache.hadoop.fs.s3a.S3AFileSystem not found" in EMR 7.0.0

I am using EMR 7.0.0 version, which has python 3.9, spark 3.5.0, Hadoop 3.3.6 in AWS.
I got the error:

  File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 740, in csv
  File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco
  File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o45.csv.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found

I searched the solution, and suggested to org.apache.hadoop:hadoop-aws = 3.2.0.
However, it still keeps failing. The code to build spark

builder = pyspark.sql.SparkSession.builder.appName("stock-daily-ingestion") 
    .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.0") 
    .config('spark.sql.shuffle.partitions', 2500) 
    .config("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") 
    .config("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")
sc = builder.getOrCreate()

Did anyone meet this error? Help will be appreciated. Thanks.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    The issue is resolved. When running in spark, adding the following in the spark-submit command

    --conf spark.hadoop.io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec 
--packages org.apache.hadoop:hadoop-aws:3.2.0

  2. 0

    Spark cannot find the class responsible for accessing S3 storage within the Hadoop ecosystem.The class is part of the hadoop-aws library.
    Sample code to include hadoop-aws library is as follows:

    spark-submit 
    --master yarn 
    --conf spark.yarn.queue=your-queue 
    --packages org.apache.hadoop:hadoop-aws:3.3.6 
    your_spark_script.py
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search