I have an s3 bucket which isn’t deleting old versions.
The bucket is used to store backups – each backup is two files 30GB and 1GB.
We keep backups for 14 days and then have a script which deletes the older backups.
Viewing the bucket objects shows that this script is working as only the expected 14 days of objects are visible.
The bucket is object locked with versioning enabled.
We have a lifecycle rule which is designed to cleanup stuff our script missed – this should never actually happen – but is included as a failsafe.
The lifecycle delete rule is:
- Apply to all objects in bucket
- Permanently delete noncurrent versions
- Delete expired object delete markers or incomplete multipart uploads
- Days after objects become noncurrent: 90
number of versions: not set.
It might be worth noting that each backup file only ever has a single version.
If I run:
aws s3api list-object-versions --bucket <bucket name>
I see:
Versions": [
{
"ETag": ""XXXXX"",
"Size": 5183943176,
"StorageClass": "STANDARD_IA",
"Key": "backup/autobackup/2023-11-25_04-05-41/volumes.zip.gpg",
"VersionId": "XXX.XXX",
"IsLatest": false,
"LastModified": "2023-11-24T17:06:09+00:00",
"Owner": {
"DisplayName": "support",
"ID": "XXXXXX"
}
},
As you can see this version is almost 12 months old so clearly outside the 90 day delete rules.
So what am I doing wrong here?
Edit:
The retention period on the object is:
{
"Retention": {
"Mode": "COMPLIANCE",
"RetainUntilDate": "2023-12-24T17:06:08.312000+00:00"
}
}
Note: I’ve update the example object as I was able to delete it from the CLI. The retain until date relates to the updated example object.
2
Answers
There could be some conflict between object lock and lifecycle policies. Best way to debug would be to check retention period for an object version
If the retention period is set for a time beyond the 90-day lifecycle window, the object will not be deleted until that retention period has expired.
Reference – https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html
I feel the issue is with the object lock which is over ridding the object life cycle rule.