Php – AWS ISO-8859-1 Header
I'm creating a download link to an item that is stored on AWS / S3. As I am building out this link, I've confirmed that the data is encoded in UTF-8, but when the user goes to download it, they…
Amazon web services – Pyspark error: " Class org.apache.hadoop.fs.s3a.S3AFileSystem not found" in EMR 7.0.0
I am using EMR 7.0.0 version, which has python 3.9, spark 3.5.0, Hadoop 3.3.6 in AWS. I got the error: File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 740, in csv File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__ File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py",…
Amazon web services – Fetch multiple filename from S3 and append in a list to print the same in output
I have multiple files uploaded in S3 bucket and I want to print the names of the files in the output in a list through lamda python . Lets say I have files S1,S2,S3 ... and so on and I…
Amazon web services – How to programmatically create IAM Roles Anywhere sessions with AWS SDK for Java V2 in a Spring Boot service
I'm working on a Spring Boot backend service where I need to access certain S3 buckets programmatically using IAM Roles Anywhere. I found the following documentation for RolesAnywhereClient in the AWS SDK for Java V2: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/rolesanywhere/RolesAnywhereClient.html However, I’m unsure how…
Amazon web services – ytdl-core is returning a stream with nothing in it when I try to download a video
I am trying to use ytdl-core in an AWS lambda function to upload an .mp4 of a youtube video to an S3 bucket. However, I cannot get ytdl to create a stream with any data in it, so nothing is…
Amazon web services – List all the files which are inside a folder in a s3 bucket using pyspark
I have a s3 bucket "bucket1" inside which I have a directory named "dir1". Inside this directory there are multiple files. I just want to create a list of all the file names in this directory in my pyspark code…
Amazon web services – Restrict public access to S3 bucket, but allow access for Lambdas
I'm trying define an S3 bucket which has public access disabled, but my other defined Lambdas can access it. I keep receiving an error on deployment about "Invalid principal in policy". This is my resource definition... ResultsBucketPolicy: Type: AWS::S3::BucketPolicy Properties:…
Amazon web services – How to share objects in a S3 folder and get links to them?
I upload files to a folder in an S3 bucket via the S3 JavaScript SDK. Now I want to share all objects in that folder (maybe there are more objects later) therefore I want to get a list of all…
Amazon web services – Process large CSV from amazon s3 into postgres database table
I need to process a large CSV from AWS S3 to Postgres DB (AWS RDS). I am new to AWS and need some suggestions on the architecture for this. Below are the requirements. We receive a CSV file daily in…
Amazon web services – s3 storage not aging out
I have an s3 bucket which isn't deleting old versions. The bucket is used to store backups - each backup is two files 30GB and 1GB. We keep backups for 14 days and then have a script which deletes the…