Question 95429 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Execute only one Glue job at a time / sequential Glue job execution – Amazon web services

Currently, we have the following AWS setup for executing Glue jobs. An S3 event triggers a lambda function execution whose python logic triggers 10 AWS Glue jobs.

S3 -> Trigger -> Lambda -> 1 or more Glue Jobs.

With this setup, we see that at a time, multiple different Glue jobs run in parallel. How can I make it so that at any point in time, only one Glue job runs? And any Glue jobs sent for execution wait in a queue until the currently running Glue job is finished?

Tags:

2

Answers


  1. 0

    You can use step function and in each steps specify job you want to run so you will have control to run jobs and once step one complete then call step 2 jobs etc

    Login or Signup to reply.
  2. 0

    If you are looking for having some job queues to have the Glue jobs trigger in sequence, you may consider using a combination of SQS->lambda->Glue jobs? Please refer this SO for details

    AWS Step function is also another option as suggested by Vaquar Khan

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search