We are trying to optimise AWS S3. The cost explorer shows very high cost of ListBuckets calls.
To get the source of these calls, we enabled the CloudTrails with configuration to log all the management and data eventtype logs for S3. In the cloudtrail logs, there is not even 0.0001% of the calls which are reflected in the cost explorer. What can be the possible reasons for this?
Is the ListBuckets in cost explorer related to some other API in CloudTrail?
Or are the ListBuckets events not getting logged in CloudTrail becuase am I doing something wrong while configuring the CloudTrails?
2
Answers
According to Actions, resources, and condition keys for Amazon S3 – Service Authorization Reference:
I always find this naming confusing. The
ListBucket
operation lists the contents of buckets, not to be confused withListBuckets
permission that is associated with theListAllMyBuckets
action. Confusing indeed!So, I think it is actually charging you for listing the contents of buckets. In boto3 this is called
list_objects()
. So, look for anything in Cost Explorer that might talk about listing objects within a bucket and it will probably show more data.Cost Explorer does not (to the best of my knowledge, as of today) track costs at an API level. The "API Operation" filter in Cost Explorer appears to just select the "Usage Type"(s) associated with the API call.
In the case of "ListBuckets", this appears to be "Requests-Tier1", which includes "PUT, COPY, POST, LIST requests" per the S3 pricing page (search for "Requests & data retrievals").
So, it’s far more likely that your charges are caused by the
PutObject
API call.TBH, I think that the "API Operation" selector is buggy. I used to to examine an account where I had uploaded several million objects in the past month, yet it showed $0.00 for the "PutObject" API call.