skip to Main Content

I am running spark workers using docker, replicated using a docker-compose setup:

version: '2'

services:
  spark-worker:
    image: bitnami/spark:latest
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://1.1.1.1:7077
    deploy:
      mode: replicated
      replicas: 4

When I run docker-compose exec spark-worker ls, for example, it only runs on usually the first replica. Is there a way to broadcast these commands to all of the replicas?

docker-compose version 1.29.2, build 5becea4c

Docker version 20.10.7, build f0df350

2

Answers


  1. There’s no built-in facility for this, but you could construct something fairly easily. For example:

    docker ps -q --filter label=com.docker.compose.service=spark-worker |
    xargs -ICID docker exec CID ls
    
    Login or Signup to reply.
  2. I would propose a simpler command than what @larsks proposed, which leverages the docker-compose command itself.

    SERVICE_NAME=spark-worker
    for id in $(docker-compose ps -q $SERVICE_NAME); do
        docker exec -t $id "echo" "hello"
    done;
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search