Question 19709 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

How to use CloudFormation to update AWS Glue Jobs

We have many AWS Glue jobs and we are only updating the job code, which are scripts stored in S3.
The problem is CloudFormation couldn’t tell when and when not to update our Glue jobs because all CloudFormation template parameters remain the same after script changes, even the script location is pointing to the same S3 object.

Tags:

2

Answers


  1. 0

    As this is similar to Lambda code package, you can use parameterize the Glue script path and then have different versions of Glue script file

    Glue CFT has "Command" parameter taking in "JobCommand" Type value which includes "ScriptLocation" attribute, make this as a CFT parameter and have the script dynamic

    {
  "Name" : String,
  "PythonVersion" : String,
  "ScriptLocation" : String
}

    You can probably setup a CI/CD pipeline using AWS CodePipeline or your 3rd party CI/CD tool with the below steps

    • Pull the new code from your SCM like Github to deploy S3
    • Update CloudFormation stack with new S3 script path (with versions like v1, v2 etc…)
    Login or Signup to reply.
  2. 0

    You can use the CloudFormation package command. This enables you to reference local files in your git repository as scripts for Glue Jobs. Every time before you deploy to CloudFormation you just run the package command.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search