Question 301313 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – aws s3 AccessDenied when calling the GetObject operation: Access Denied

I have a python code to use pandas to read s3 files located at

s3://bucket_name/table_name/file.parquet

If I provide the pull path with file name like above, my python works.
But if I provide the directory path as the prefix to list all objects and then get each object, I get the access denied error:

bucket=bucket_name
key=table_name/
objects=s3.list_objects(Bucket=bucket,Prefix=key)['Contents']
print(objects)
dfs=[]
for obj in objects:
    file_key=obj['Key']
    obj=s3.get_object(Bucket=bucket,Key=file_key)
    df_single=pd.read_parquet(BytesIO(obj['Body'].read()))
    dfs.append(df_single)
df=pd.concat(dfs, ignore_index=True)

I got the error in the title.

I checked the IAM policy for the bucket. we allow for s3.ListBucket and s3.getObject for the bucket "s3://bucket_name*/*"

UPDATE

updated the policy by adding the resource without ‘/*’ as shown below.
and attached this policy to the role that I use as the executing role of the lambda function (which runs the script)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketTagging",
                "s3:ListBucketVersions",
                "s3:ListBucket",
                "s3:GetBucketAcl",
                "s3:BypassGovernanceRetention",
                "s3:GetBucketPolicy",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ObjectOwnerOverrideToBucketOwner",
                "s3:DescribeJob",
                "s3:PutObjectVersionAcl",
                "s3:PutBucketAcl",
                "s3:PutBucketPolicy",
                "s3:DeleteAccessPointPolicy",
                "s3:DeleteObject",
                "s3:PutAccessPointPolicy",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::bucket_name",
                "arn:aws:s3:::bucket_name*/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListStorageLensConfigurations",
                "s3:GetAccessPoint",
                "s3:ListAllMyBuckets",
                "s3:ListAccessPoints",
                "s3:ListJobs",
                "s3:PutStorageLensConfiguration",
                "s3:CreateJob"
            ],
            "Resource": "*"
        }
    ]
}

run the script:

s3_resource=boto3.resource('s3')
bucket = s3_resource.Bucket(bucket_name)
for obj in bucket.objects.all():
    print(obj.key)

get the error

[ERROR] ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
Tags:

2

Answers


  1. 0

    Please note that ListBucket requires permissions on the bucket (without /*) while GetObject applies at the object level and can use * wildcards.

    Also, if you are granting access to an IAM User or IAM Role in the same AWS Account, it is better to grant permissions via an IAM Policy on the IAM User/Role instead of using a Bucket Policy.

    This IAM Policy can be applied to the IAM User/IAM Role and grant access to the whole bucket to that user:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::BUCKET-NAME"
        },
        {
            "Action": "s3:GetObject",
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::BUCKET-NAME/*"
        }
    ]
}

    If you were to use a Bucket Policy (not advisable), you would need to add a Principal to identify who receives access:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::BUCKET-NAME",
            "Principal": "arn:aws:iam::123456789012:user/USERNAME"
        },
        {
            "Action": "s3:GetObject",
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::BUCKET-NAME/*",
            "Principal": "arn:aws:iam::123456789012:user/USERNAME"
        }
    ]
}

    These policies can be further restricted to only grant access to a specific directory if desired. See: Bucket policy examples – Amazon Simple Storage Service

    Login or Signup to reply.
  2. 0

    To reproduce your situation, I did the following:

    • Created a new Amazon S3 bucket with default settings
    • Uploaded a few files to the bucket, with default settings
    • Created a new IAM User and added an inline policy (substituting BUCKET-NAME for the actual bucket name):
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::BUCKET-NAME"
        },
        {
            "Action": "s3:GetObject",
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::BUCKET-NAME/*"
        }
    ]
}
    • Created an Access Key + Secret Key for this user in the IAM Console and stored the credentials using aws configure --profile stack
    • Tested with aws s3 ls --profile stack — it successfully listed the contents of my bucket
    • Tested with a different bucket name — was not able to list the bucket (as expected)
    • Ran this Python code (substituting my-bucket-name for the actual bucket name):
    import boto3

BUCKET_NAME = 'my-bucket-name'

session = boto3.Session(profile_name='stack')
s3_resource = session.resource('s3')

bucket = s3_resource.Bucket(BUCKET_NAME)
for obj in bucket.objects.all():
    print(obj.key)
    • It successfully listed the objects I previously uploaded to the bucket

    If your results are different, you might want to compare the above steps to what you have done.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search