Question 19595 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Terraform: Codepipeline not triggered on upload to s3 source bucket

I am trying to trigger the codepipeline on upload to s3 using terraform.

Use case – So a terraform code for various resources will be pushed as a zip file to the source bucket which will trigger a pipeline. This pipeline will run terraform apply for the zip file. So in order to run the pipeline I am setting up a trigger

Here is what I have done.

  • Create source s3 bucket
  • Create code pipeline
  • Created cloudwatch events rule for s3 events fro cloudtrail
  • Created cloudTrail Manually, and added data event to log source bucket write events. , all previous steps were done using terraform.

After doing all this still, my pipeline is not triggered on upload of new bucket.

I was reading this docs and it had particular statement about sending trail events to eventbridge rule which I think is the cause but I can’t find the option to add through console.

AWS CloudTrail is a service that logs and filters events on your Amazon S3 source bucket. The trail sends the filtered source changes to the Amazon CloudWatch Events rule. The Amazon CloudWatch Events rule detects the source change and then starts your pipeline.

https://docs.aws.amazon.com/codepipeline/latest/userguide/create-cloudtrail-S3-source.html

Here is my event ridge rule

resource "aws_cloudwatch_event_rule" "xxxx-pipeline-event" {
  name        = "xxxx-ci-cd-pipeline-event"
  description = "Cloud watch event when zip is uploaded to s3"

  event_pattern = <<EOF
{
  "source": ["aws.s3"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["s3.amazonaws.com"],
    "eventName": ["PutObject", "CompleteMultipartUpload", "CopyObject"],
    "requestParameters": {
      "bucketName": ["xxxxx-ci-cd-zip"],
      "key": ["app.zip"]
    }
  }
}
EOF
}

    resource "aws_cloudwatch_event_target" "code-pipeline" {
  rule      = aws_cloudwatch_event_rule.XXXX-pipeline-event.name
  target_id = "SendToCodePipeline"
  arn       = aws_codepipeline.cicd_pipeline.arn
  role_arn  = aws_iam_role.pipeline_role.arn
}

Event bridge role permissions terraform code

data "aws_iam_policy_document" "event_bridge_role" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"
    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }
  }

}

resource "aws_iam_role" "pipeline_event_role" {
  name               = "xxxxx-pipeline-event-bridge-role"
  assume_role_policy = data.aws_iam_policy_document.event_bridge_role.json
}

data "aws_iam_policy_document" "pipeline_event_role_policy" {
  statement {
    sid       = ""
    actions   = ["codepipeline:StartPipelineExecution"]
    resources = ["${aws_codepipeline.cicd_pipeline.arn}"]
    effect    = "Allow"
  }
}

resource "aws_iam_policy" "pipeline_event_role_policy" {
  name   = "xxxx-codepipeline-event-role-policy"
  policy = data.aws_iam_policy_document.pipeline_event_role_policy.json
}

resource "aws_iam_role_policy_attachment" "pipeline_event_role_attach_policy" {
  role       = aws_iam_role.pipeline_event_role.name
  policy_arn = aws_iam_policy.pipeline_event_role_policy.arn
}
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    The problem was with CLoudtrail filter. The filter was set for bucket and write actions.

    I had to modify filter by adding prefix to it.Because my event bridge is looking for my-app.zip so it was not triggered if I used only bucket level prefix

    bucket/prefix and write action

    Docs :https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html


  2. 0

    I am using this to accomplish a similar goal but I’m having issues with aws_s3_object not always triggering the notification:

    s3.tf

    # Create S3 bucket for the tenant's sauce account (mount point for s3fs)
resource "aws_s3_bucket" "this" {
  bucket        = random_pet.generator.id
  force_destroy = true
}

# Set the public access block for the sauce bucket
resource "aws_s3_bucket_public_access_block" "sauce" {
  bucket                  = aws_s3_bucket.this.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# Set the ACL for the sauce bucket to make it private
resource "aws_s3_bucket_acl" "this" {
  bucket = aws_s3_bucket.this.id
  acl    = "private"

}

# Push S3 notifications to EventBridge
resource "aws_s3_bucket_notification" "this" {
  bucket      = aws_s3_bucket.this.id
  eventbridge = true
}

# Create S3 object for aws_codebuild_project.on_success Terraform package
resource "aws_s3_object" "this" {
  bucket                 = aws_s3_bucket.this.id
  key                    = "on_success.zip"
  content_type           = "application/zip"
  source                 = data.archive_file.on_success.output_path
  etag                   = data.archive_file.on_success.output_md5
  server_side_encryption = "AES256"
  force_destroy          = true
  depends_on = [
    aws_s3_bucket_notification.this
  ]
}

    cloudwatch.tf

    # Create an EventBridge trigger to send terraform plans to aws_codebuild_project.on_success
resource "aws_cloudwatch_event_rule" "on_success" {
  name        = "on-success-event-trigger"
  event_pattern = jsonencode(
    {
      "source" : ["aws.s3"],
      "detail-type" : ["Object Created"],
      "account" : [data.aws_caller_identity.this.account_id],
      "region" : [var.AWS_DEFAULT_REGION],
      "detail" : {
        "bucket" : {
          "name" : [aws_s3_bucket.this.id]
        },
        "object" : {
          "key" : ["on_success.zip"]
        },
      }
    }
  )
}

# Create an EventBridge target to deploy terraform with aws_codebuild_project.on_success
resource "aws_cloudwatch_event_target" "on_success" {
  rule      = aws_cloudwatch_event_rule.on_success.name
  arn       = aws_codebuild_project.on_success.arn
  role_arn  = aws_iam_role.this.arn
  target_id = aws_codebuild_project.on_success.name
}

    codebuild.tf

    # Create a CodeBuild project to apply Terraform plans
resource "aws_codebuild_project" "on_success" {
  name          = "on-success"
  service_role  = aws_iam_role.this.arn
  environment {
    compute_type = "BUILD_GENERAL1_SMALL"
    image        = "hashicorp/terraform:latest"
    type = "LINUX_CONTAINER"
  }

  source {
    type      = "NO_SOURCE"
    buildspec = file("buildspec.yml")
  }

  artifacts {
    type = "NO_ARTIFACTS"
  }

  # ISSUE: This is a workaround for the CodeBuild projects in GovCloud
  # https://github.com/hashicorp/terraform-provider-aws/issues/22473
  lifecycle {
    ignore_changes = [project_visibility]
  }

  # This file will be available to the buildspec.yml phases
  secondary_sources {
    source_identifier = "local"
    type              = "S3"
    location          = "${aws_s3_object.this.bucket}/${aws_s3_object.this.key}"
  }
}
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search