Question posted in
The official documentation can be found here.

Redis – How to execute a task in Airflow in another server?

I have already set up the following infrastructure:

  • host (main):
    airflow (with all the services)

  • slave server
    airflow (redis, postgres, worker)

Cannot see any tutorial or couldn’t find any clue about how to do it, but i don’t have any idea about how to connect the task with the parameter queue=node1 to execute it in the slave server, thank you!

Tags:

2

Answers


  1. 0

    To execute a task in Airflow on another server, you can utilize the SSHOperator or set up a remote worker configuration. With SSHOperator, you can execute commands on a remote server directly. Alternatively, setting up a remote worker configuration allows tasks to be distributed across multiple servers. Check out here to learn more about servers.

    Login or Signup to reply.
  2. 0

    You can use the following example to redirect the task:

    train_model = PythonOperator(
  task_id='train_model',
  python_callable=train_model_flights
  queue='<QUEUE_NAME>'
)
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search