I am running spark workers using docker, replicated using a docker-compose setup:
version: '2'
services:
spark-worker:
image: bitnami/spark:latest
environment:
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://1.1.1.1:7077
deploy:
mode: replicated
replicas: 4
When I run docker-compose exec spark-worker ls
, for example, it only runs on usually the first replica. Is there a way to broadcast these commands to all of the replicas?
docker-compose version 1.29.2, build 5becea4c
Docker version 20.10.7, build f0df350
2
Answers
There’s no built-in facility for this, but you could construct something fairly easily. For example:
I would propose a simpler command than what @larsks proposed, which leverages the
docker-compose
command itself.