I run my AWS Glue jobs locally in a docker container (AWS Glue lib 4.0) and want to convert/write a pandas dataframe to delta format. I added spark = SparkSession.builder .appName("YourAppName") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate() sys.argv += ['--datalake-formats', 'delta']…
I am getting "Error loading data" in Job run monitoring page on AWS Glue. I can able to see the data when I logged in as "Admin" user but I cannot see the data while I am logging in as…
I have a terraform script to deploy an AWS Glue Job with it's sourcecode (ET script). When I run terraform apply it's updating the infrastructure. But not the source code. What should I do to update the code as well??…
I am trying to add a new column called timestamp in AWS Glue. My upstream data keys have capital letters. However, after adding the column timestamp, the keys of the remaining columns got lowercased. Experiment 1: Transform - SQL Query…
I know that athena store every query results in the bucket and query data will just accumulate over time. I want to know whether retaining previous query results in S3 would make an impact to performance of my queries. For…
I'm unable to alter the column name of Hudi table . spark.sql("ALTER TABLE customer_db.customer RENAME COLUMN subid TO subidentifier") unbable to change the column name. A clear and concise description of the problem. I'm unable to alter the column name…
I have created a Glue catalog in my account. It has 1 DB and 1 table. I followed this guide from AWS and created my EMR cluster. However, when I run spark-shell and try to access Glue catalog, I am…
Imagine you had a set of R scripts that form an ETL pipeline that you wanted to run as an AWS Glue job. AWS Glue supports Python and Scala. Is it possible to call an R as a Python subprocess…
I have an S3 bucket that contains multiple different files by date with different schemas (they are not in sub-folders). For example: FileTypeA-032323 FileTypeA-032423 FileTypeA-032523 FileTypeB-032323 FileTypeB-032423 FileTypeC-032323 FileTypeD-032423 FileTypeD-032423 FileTypeD-032523 I need to merge all of the files for…
I am trying to create a custom visual transformation in a visual etl job in order to add a column to the output. The name of the column is something I would like to parameterize and pass at runtime using…