I have multiple zip files called f1.zip, f2.zip, ... f7.zip each contain around 200k xml files and I am using this code to multiprocess to unzip them in parallel, but even if each file is very small, the read/write process…
I am trying to create a emr-serverless application through the EmrServerlessCreateApplicationOperator but I keep facing the error botocore.exceptions.NoRegionError: You must specify a region. I am passing the region like below: create_app = EmrServerlessCreateApplicationOperator( task_id="create_spark_app", job_type="SPARK", release_label="emr-6.6.0", config={"aws_access_key_id":args["aws_access_key_id"], "aws_secret_access_key": args["aws_secret_access_key"], "aws_session_token":…
Using emr-5.33.1 and python3.7.16. Goal is to add petastorm==0.12.1 into EMR. These are the steps to install it in EMR (worked until now) Add all required dependencies of petastorm and itself into s3 folder copy paste all libraries from s3…
I'm new to Spark and I'm running spark job on EMR cluster. It takes about one hour for job to finish. In job, there is a code for reading the file from S3 (file size is 10MB) and writing it's…
I have created a Glue catalog in my account. It has 1 DB and 1 table. I followed this guide from AWS and created my EMR cluster. However, when I run spark-shell and try to access Glue catalog, I am…
In our place, we use AWS services for all our data infrastructure and services needs. Our hive tables are external tables and the actual data files are stored in S3. We use Apache Spark for Data ingestion and transformation. We…
I have a pyspark job that is distributed in multiple code files in this structure: flexible_clendar - Cache - redis_main.py - Helpers - helpers.py - Spark - spark_main.py - main.py In the 'main.py' I'm using the functions from 'helpers.py', 'redis_main.py',…
I'm running a Spark application in AWS EMR. The code is like this: with SparkSession.builder.appName(f"Spark App").getOrCreate() as spark: dataframe = spark.read.format('jdbc').options( ... ).load() print("Log A") max_date_result = dataframe.agg(max_(date_format('date', 'yyyy-MM-dd')).alias('max_date')).collect()[0] print("Log B") This application always gets stuck for a long time…