AWS glue is new to me. I'm working with AWS Glue and I'm trying to read from RDS database table and write the data to an s2 bucket as a single csv file. I've setup an AWS Glue job using…
I have use case where after inserting a row in mysql table i want to fetch back the id of the last inserted row. I wanted to run a MYSQL query in aws glue for the same. I am new…
DynamoDB Table: +------------------+---------+---------------------+ | id |Column A | Column B| +------------------+---------+---------------------+ |1 | "155.2"| 400 | |2 | 100| 200 | |3 | "455.2"| 305.5 | |4 | "312.3"| 350 | +------------------+---------+---------------------+ Notice that in Column A. We have numbers…
I'm creating a serverless application in AWS that needs to store some data gathered trough an external API into a DynamoDB instance. I'm trying to achieve it through Lambda functions, but the data are quite heavy, so everytime I try…
I have a psql RDS on the same AWS account where I am trying to set up a glue connection to it. I used the RDS option and selected my existing RDS, then I set up the network to be…
I am using Athena to query an S3 bucket and using Athena partition projection as well. Here is my bucket structure: bucket-name/node=[A-Z0-9]{4}/date={yyyy}-{MM}-{dd}/{uuid}.parquet e.g: bucket-name/node=NODE/date=2023-10-10/123.parquet Glue table schema: [ { "Name": "data", "Type": "string" }, { "Name": "effective_time_start", "Type": "bigint" },…
I am trying to execute an aws glue start-job-run command in a bash script where I need to pass multiple arguments in the --arguments parameter. Here is an example of what I am expecting to do: aws glue start-job-run --job-name…
I have a parquet datalake on S3 that I wish to query from. To optimize performance, I aim to compact my files periodically using below script: import boto3 import datetime import math from awsglue.utils import getResolvedOptions import sys args =…
I have a use case as follows: An IoT device is uploading many small files into an AWS S3 input bucket. Upon upload, every single file is processed by an AWS Lambda function to decode the data and write a…
I run my AWS Glue jobs locally in a docker container (AWS Glue lib 4.0) and want to convert/write a pandas dataframe to delta format. I added spark = SparkSession.builder .appName("YourAppName") .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") .getOrCreate() sys.argv += ['--datalake-formats', 'delta']…