I use boto3 to generate a presigned URL for an S3 object S3_CLIENT = boto3.client('s3') PRESIGNED_URL_EXPIRY_SECONDS = 7*24*3600 def generate_presigned_url(bucket_name, object_key, expiration): """ Generates a pre-signed URL for an S3 object. Args: bucket_name (str): The name of the S3 bucket.…
I have a requirement to trigger an AWS Lambda function when there some process pushes the message to Amazon SQS. I have following CloudFormation configuration. The other process is able to push the message to SQS but it is not…
Im trying to extract an array using array_position to return the position of a specific string. Unfortunately, redshift doesn't have this function.
We could ingest 2 datasets (dataset A and dataset B) daily, dataset A may or may not show up everyday. They both trigger separate Step Functions. step function B with dataset B cant run until step function A with dataset…
I am having a problem with the behavior of a Lambda function. Ultimately, I want my lambda function to read some information about json files that I upload to my S3 bucket. Here is a test function: import boto3 import…
I am creating a step function for running an ETL job when ever files are uploaded into DB. The step function involves tasks like glue jobs, crawler runs lambdas etc. I have different clients that upload data to their respective…
Created a new RDS MySQL t3.micro server to use for development purposes and testing. Total cost is supposed to be $0.026 per hour for a Standard, Single-AZ instance of this type in my selected region which means around $18.72 per…
I'm trying to get ec2 instance id and assign it into environment variable with a script. I found some command that supposed to give me the id but they don't work. The curl command returns nothing: curl command, I tried…
I have configured the MSK cluster and allowed public access through SASL/SCRAM authentication method. Now I am facing the issue where I do not have the necessary permissions when using these credentials (specified in the Secrets Manager created with a…
I'm starting a project to adjust the data lake for the specific purge of data, to comply with data privacy legislation. Basically the owner of the data opens a call requesting the deletion of records for a specific user, and…