I am trying to create a connection between AWS Glue and the Redshift database in Glue. Currently, I am getting an error: Connection creation is failed. Create connection failed during validating credentials. Please validate connection inputs and VPC connectivity to…
Using AWS Firehose to ingest data into an Iceberg table managed by AWS Glue, I'm unable to insert timestamp data. Firehose I'm trying to insert data using the following script: json_data = json.dumps( { "ADF_Record": { "foo": "bar", "baz": "2024-09-04T18:56:15.114"…
As mentioned in title, I am trying to fetch list of databases from AWS Glue. I have plenty of databases listed in Glue. I am using a policy with limited resource access so that I can only see specific type…
I have a Event Bridge Trigger set on a s3 bucket and everytime we upload an object, it triggers a NotifyEvent in Cloud Trail. I am trying to extract the bucket name and key from the payload
I am trying to read the schema from a text file under the same package as the code but cannot read that file using the AWS glue job. I will use that schema for creating a dataframe using Pyspark. I…
A Python Shell job cannot use more than one DPU. This means that it has a limit of 16 GB of memory. Earlier today, I wired what I considered to be a modest ETL task to AWS Glue with 1…
I know that this has been asked before. But I have spent hours trying to get this to work. I have a directory structure like: - datalake --- datasets ----- foo ------- 00001.json ------- 00002.json ------- latest.json ----- bar -------…
We have a dynamodb database and would like to export the data to a new redshift database staging table on a nightly basis. Ideally I think it be better if we only export inserts or updates since last load otherwise…
I am trying to run a query to read data from redshift table through AWS Glue job. When it reads data from table, it creating dataframe with only 1 partition and it is taking lot of time to read the…
Here's my setup: using AWS Organizations to segregate different application stacks; each Organizational Unit has separate AWS accounts to separate Dev and Prod. one "Shared services" AWS account with a CodeArtifact repository containing private Python packages, including common packages for…