I have a large AWS CF template that I have worked on for about a year with hundreds of deploys.
Today I changed a the key on a DynamoDB table and the full stack went into Rollback_complete. The only solution was to delete the stack and manually delete a lot of resources as well, including an S3 bucket with loads of data. The account is only for dev environment so no real harm happened, but I really don’t want it to happen again.
Should the CF stack have other parameters for “on fail” or are there other solutions for this?
I still can’t understand why a small change like this could ruin the whole stack…
2
Answers
You changed a key in DynamoDB, means resource has to be deleted and created again, if table has data CF failed to delete it. Means whole rollback and nothign changed.
FYI: It is only limitation with KEY, You can do it with non-key attribute.
Changing key of dynamo db table "with data" and "without recreation" is not supported using CF or any other like Terraform. It is limitation with dynamo DB not CloudaFormation or Terraform.
AWS documentation provides an article to handle these situations where CloudFormation cannot be updated via re-deployment and as mentioned in other answers deletion of complete stack should be only used when nothing works (Not applicable for production). You can refer article: Handling CloudFormation Rollback