Question 80619 in Docker Question posted in
The official documentation can be found here.

Docker compose doesn't work correctly with yaml config

Recently I’ve added the following command in command section of the docker-compose config file:

  livy:
    image: ecp/spark:${TAG:-latest}
    container_name: spark-livy
    ports:
      - "${LIVY_PORT:-8998}:8998"
    command: ["sh", "-c", "/opt/bitnami/livy/bin/livy-server && sed -i 's/r$//' /opt/bitnami/livy/conf/livy-env.sh && sed -i 's/r$//' /opt/bitnami/livy/conf/livy.conf && sed -i 's/r$//' /opt/bitnami/livy/conf/log4j.properties &&  sed -i 's/r$//' /opt/bitnami/spark/conf/hbase-site.xml"]
    volumes:
      - ./conf/livy-env.sh:/opt/bitnami/livy/conf/livy-env.sh
      - ./conf/livy.conf:/opt/bitnami/livy/conf/livy.conf
      - ./conf/log4j.properties:/opt/bitnami/livy/conf/log4j.properties
      - ./conf/hbase-site.xml:/opt/bitnami/spark/conf/hbase-site.xml
      - ../../spark:/opt/bitnami/spark/app # pySpark scripts
    networks:
      - default_network
    depends_on:
      - spark

After that, when I try to rebuild or remove container with commands like "docker-compose build" or "docker-compose down && docker-compose rm" it just prints:

pavel@HP-ProBook:~/dp/docker/spark$ docker-compose down
$//' /opt/bitnami/spark/conf/hbase-site.xml" &&  sed -i 's/ service "livy": "/opt/bitnami/livy/bin/livy-server && sed -i 's/
pavel@HP-ProBook:~/dp/docker/spark$ docker-compose rm
$//' /opt/bitnami/spark/conf/hbase-site.xml" &&  sed -i 's/ service "livy": "/opt/bitnami/livy/bin/livy-server && sed -i 's/

and does nothing.

If I remove this part:

&& sed -i 's/r$//' /opt/bitnami/livy/conf/livy-env.sh && sed -i 's/r$//' /opt/bitnami/livy/conf/livy.conf && sed -i 's/r$//' /opt/bitnami/livy/conf/log4j.properties &&  sed -i 's/r$//' /opt/bitnami/spark/conf/hbase-site.xml

and leave only

command: ["sh", "-c", "/opt/bitnami/livy/bin/livy-server"]

the script works normally.

What’s the matter?

Tags:

2

Answers


  1. 0

    It’s r You should change it to \r.

    Try to change the command to this:

        command: ["sh", "-c", "/opt/bitnami/livy/bin/livy-server && sed -i 's/\r$//' /opt/bitnami/livy/conf/livy-env.sh && sed -i 's/\r$//' /opt/bitnami/livy/conf/livy.conf && sed -i 's/\r$//' /opt/bitnami/livy/conf/log4j.properties &&  sed -i 's/\r$//' /opt/bitnami/spark/conf/hbase-site.xml"]

    How ever I don’t think other commands execute! because if /opt/bitnami/livy/bin/livy-server be execute never exits to let other command run.

    you should run them first or use a entrypoint script.

    Login or Signup to reply.
  2. 0

    When you stop the container, livy-server function exits. If it stops without error (exit code 0),only then does the remainder of && commands get executed…

    The better way to do this is to create your own Dockerfile

    ARG TAG
FROM ecp/spark:${TAG}

WORKDIR /opt/bitnami/

COPY conf/livy.conf livy/conf/livy.conf
RUN ["sed",...]
# repeat for each 

ENTRYPOINT ["/opt/bitnami/livy/bin/livy-server"]

    And

    livy:
  build: 
    context: . 
    args:
     - TAG=${TAG:-latest}

    However, none of your files should end with carriage returns anyway, so you really shouldn’t be doing that specific sed command from compose or a Dockerfile. Use a text editor on your host, before mounting each file, to save the files with proper line endings

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search