I am getting "Error loading data" in Job run monitoring page on AWS Glue. I can able to see the data when I logged in as "Admin" user but I cannot see the data while I am logging in as…
I have a terraform script to deploy an AWS Glue Job with it's sourcecode (ET script). When I run terraform apply it's updating the infrastructure. But not the source code. What should I do to update the code as well??…
I am trying to add a new column called timestamp in AWS Glue. My upstream data keys have capital letters. However, after adding the column timestamp, the keys of the remaining columns got lowercased. Experiment 1: Transform - SQL Query…
I know that athena store every query results in the bucket and query data will just accumulate over time. I want to know whether retaining previous query results in S3 would make an impact to performance of my queries. For…
I'm unable to alter the column name of Hudi table . spark.sql("ALTER TABLE customer_db.customer RENAME COLUMN subid TO subidentifier") unbable to change the column name. A clear and concise description of the problem. I'm unable to alter the column name…
I have created a Glue catalog in my account. It has 1 DB and 1 table. I followed this guide from AWS and created my EMR cluster. However, when I run spark-shell and try to access Glue catalog, I am…
Imagine you had a set of R scripts that form an ETL pipeline that you wanted to run as an AWS Glue job. AWS Glue supports Python and Scala. Is it possible to call an R as a Python subprocess…
I have an S3 bucket that contains multiple different files by date with different schemas (they are not in sub-folders). For example: FileTypeA-032323 FileTypeA-032423 FileTypeA-032523 FileTypeB-032323 FileTypeB-032423 FileTypeC-032323 FileTypeD-032423 FileTypeD-032423 FileTypeD-032523 I need to merge all of the files for…
I am trying to create a custom visual transformation in a visual etl job in order to add a column to the output. The name of the column is something I would like to parameterize and pass at runtime using…
My Data set looks like this : I am using this filter : df = df.filter(trim(col("AGE"))!="" & trim(col("PHONE"))!="") I am getting an empty dataframe, I want the data without the record having name =G3 Any help is appreciated.