skip to Main Content

Airflow GlueJobOperator not working with Workertype and NumberofWorkers – Amazon web services

I am trying to create a glue job with this configuration 'NumberOfWorkers': 2, 'WorkerType': 'G.1X'. Here's my code for job creation. job_name = "glue_job" submit_glue_job = AwsGlueJobOperator( task_id="glue_job", job_name=job_name, wait_for_completion=True, # num_of_dpus=10, retry_limit=0, script_location=f"s3://bucket/etl.py", s3_bucket=GLUE_EXAMPLE_S3_BUCKET, iam_role_name=GLUE_CRAWLER_ROLE.split("/")[-1], create_job_kwargs={ 'GlueVersion': '3.0', 'NumberOfWorkers':…

VIEW QUESTION
Back To Top
Search