Question 231106 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – How to update the Source Code as well with terraform apply?

I have a terraform script to deploy an AWS Glue Job with it’s sourcecode (ET script). When I run terraform apply it’s updating the infrastructure. But not the source code. What should I do to update the code as well??

Folder structure.

--glue-script-tf.py
--main.tf

Below is my terraform script(main.tf).

provider "aws" {
  region     = "us-east-1"
}

variable "job-language" {
  default = "python"
}
    
variable "bucket-name" {
  default = "glue-poc"
}

variable "glue-role-arn" {
  default = "arn:aws:iam::#####:role/dp-da-glue-poc-role-dev"
}

variable "job-name" {
  default = "glue-job-from-terraform"
}

variable "file-name" {
  default = "glue-script-tf.py"
}

#===========================================
    
resource "aws_s3_bucket_object" "upload-glue-script" {
  bucket = "${var.bucket-name}"
  key    = "scripts/${var.file-name}"
  source = "${var.file-name}"
}
    
resource "aws_glue_job" "glue-job" {
  name        = "${var.job-name}"
  role_arn    = "${var.glue-role-arn}"
  description = "This is script to create large files from small files "
  max_retries = "1"
  timeout     = 2880
  command {
    script_location = "s3://${var.bucket-name}/scripts/${var.file-name}"
    python_version = "3"
  }
  execution_property {
    max_concurrent_runs = 2
  }
  glue_version = "4.0"
}
Tags:

2

Answers


  1. 0

    Terraform does not automatically detect the code changes in your glue-script-tf.py. It only detects infrastructure changes.

    When you run terraform apply Terraform will try to determine whether any changes are necessary based on the resources you’ve defined in your configuration. If you modify the Glue script and then run terraform apply, Terraform might not automatically detect the change and update the Glue Job with the new script.

    In summary, whatever you edit in your Python file it will not be taken as infrastructure change and it will show no changes – your current infrastructure matches the configuration.

    Alternatively, you can use the file function to read the Glue script content and upload it to S3. This approach ensures that changes in your python file be detected and take action.

    resource "aws_s3_object" "upload-glue-script" {
  bucket = "${var.bucket-name}"
  key = "scripts/${var.file-name}"
  content = file("${var.file-name}")
}
    Login or Signup to reply.
  2. 0

    What you could do (depending on the terraform version you are using) is to use the filemd5 built-in function with the etag argument:

    (Optional) Triggers updates when the value changes. The only meaningful value is filemd5("path/to/file") (Terraform 0.11.12 or later) or ${md5(file("path/to/file"))} (Terraform 0.11.11 or earlier).

    So, if you are using newer terraform versions, you could do the following:

    resource "aws_s3_bucket_object" "upload-glue-script" {
  bucket = var.bucket-name
  key    = "scripts/${var.file-name}"
  source = var.file-name
  etag   = filemd5("${var.file-name}") # or ${md5(file("${var.file-name}"))}
}

    Note also that if the file is bigger than 16MB, you should default to using source_hash argument (it also helps if you enable encryption with SSE-KMS).

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search