Question 81775 in Docker Question posted in
The official documentation can be found here.

How to automatically move all DAG files to Airflow Docker and not just have only the latest added file moved and renamed to 'example_python_operator'?

PROBLEM

  • How to automatically move all DAG files to Airflow Docker
  • and not just have only the latest added file moved
  • and renamed to ‘example_python_operator’?

Guides That I have Tried

1.

docker run -d -p 8080:8080 -v /home/user/airflow/dags:/usr/local/airflow/dags puckel/docker-airflow webserver

  • Is this my actual local path -> :/usr/local/airflow/dags ?
  • Is this a standard parth :/usr/local/airflow/dags?
  • Is this the container name of the webserver or some internal routing puckel/docker-airflow webserver?
    docker run -d -p 8080:8080 -v /MY_ACTUAL_PATH_BASED_ON_PWD/dags:/usr/local/airflow/dags my_IMAGE_name_of_webserver

2.

I can not put this to the config file because I use docker-compose so I dont get one after the 1st start.
dags_folder = /usr/local/airflow/dags

3.

This just plane doesn’t pushes my DAGs in to docker docker-compose up -d --build

#4.
added the volume but again I dont have a config file generated.
1.

webserver: image: puckel/docker-airflow:1.10.0-2 … volumes: – ./airflow:/usr/local/airflow
2.edit the dags folder configuration in the airflow configuration file(it do not needs edit by default, as it is under the airflow folder)
3.every time, check if the dag name appeared by following command:airflow list_dags

My question ask how to add them automatically. I dont what to write commands for it every time. Also non of this answers even gets the file to the folder. – Airflow in Docker: how to add DAGs to Airflow?

All my code

COMMANDS

docker build -t my38 .
docker-compose up airflow-init
docker-compose up -d

FILES

airflow/Dockerfile

FROM apache/airflow:latest-python3.8
COPY requirements.txt .
RUN pip install -r requirements.txt

airflow/requirements.txt

apache-airflow==2.4.0
pandas==1.4.2
numpy==1.20.3
pendulum==2.1.2

airflow/docker-compose.yml (THIS IS FROM THE OFFICIAL AIRFLOW SITE)

--- version: '3' x-airflow-common:   &airflow-common   # In order to add custom dependencies or upgrade provider packages you can use your extended image.   # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml   # and uncomment the "build" line below, Then run `docker-compose build` to build the images.   image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:latest-python3.8}   # build: .   environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    # For backward compatibility, with Airflow <2.3
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth'
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}   volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins   user: "${AIRFLOW_UID:-50000}:0"   depends_on:
    &airflow-common-depends-on
    redis:
      condition: service_healthy
    postgres:
      condition: service_healthy

services:   postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5
    restart: always

  redis:
    image: redis:latest
    expose:
      - 6379
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 30s
      retries: 50
    restart: always

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 8080:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-worker:
    <<: *airflow-common
    command: celery worker
    healthcheck:
      test:
        - "CMD-SHELL"
        - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
      interval: 10s
      timeout: 10s
      retries: 5
    environment:
      <<: *airflow-common-env
      # Required to handle warm shutdown of the celery workers properly
      # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
      DUMB_INIT_SETSID: "0"
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-triggerer:
    <<: *airflow-common
    command: triggerer
    healthcheck:
      test: ["CMD-SHELL", 'airflow jobs check --job-type TriggererJob --hostname "$${HOSTNAME}"']
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

  airflow-init:
    <<: *airflow-common
    entrypoint: /bin/bash
    # yamllint disable rule:line-length
    command:
      - -c
      - |
        function ver() {
          printf "%04d%04d%04d%04d" $${1//./ }
        }
        airflow_version=$$(AIRFLOW__LOGGING__LOGGING_LEVEL=INFO && gosu airflow airflow version)
        airflow_version_comparable=$$(ver $${airflow_version})
        min_airflow_version=2.2.0
        min_airflow_version_comparable=$$(ver $${min_airflow_version})
        if (( airflow_version_comparable < min_airflow_version_comparable )); then
          echo
          echo -e "33[1;31mERROR!!!: Too old Airflow version $${airflow_version}!e[0m"
          echo "The minimum Airflow version supported: $${min_airflow_version}. Only use this or higher!"
          echo
          exit 1
        fi
        if [[ -z "${AIRFLOW_UID}" ]]; then
          echo
          echo -e "33[1;33mWARNING!!!: AIRFLOW_UID not set!e[0m"
          echo "If you are on Linux, you SHOULD follow the instructions below to set "
          echo "AIRFLOW_UID environment variable, otherwise files will be owned by root."
          echo "For other operating systems you can get rid of the warning with manually created .env file:"
          echo "    See: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#setting-the-right-airflow-user"
          echo
        fi
        one_meg=1048576
        mem_available=$$(($$(getconf _PHYS_PAGES) * $$(getconf PAGE_SIZE) / one_meg))
        cpus_available=$$(grep -cE 'cpu[0-9]+' /proc/stat)
        disk_available=$$(df / | tail -1 | awk '{print $$4}')
        warning_resources="false"
        if (( mem_available < 4000 )) ; then
          echo
          echo -e "33[1;33mWARNING!!!: Not enough memory available for Docker.e[0m"
          echo "At least 4GB of memory required. You have $$(numfmt --to iec $$((mem_available * one_meg)))"
          echo
          warning_resources="true"
        fi
        if (( cpus_available < 2 )); then
          echo
          echo -e "33[1;33mWARNING!!!: Not enough CPUS available for Docker.e[0m"
          echo "At least 2 CPUs recommended. You have $${cpus_available}"
          echo
          warning_resources="true"
        fi
        if (( disk_available < one_meg * 10 )); then
          echo
          echo -e "33[1;33mWARNING!!!: Not enough Disk space available for Docker.e[0m"
          echo "At least 10 GBs recommended. You have $$(numfmt --to iec $$((disk_available * 1024 )))"
          echo
          warning_resources="true"
        fi
        if [[ $${warning_resources} == "true" ]]; then
          echo
          echo -e "33[1;33mWARNING!!!: You have not enough resources to run Airflow (see above)!e[0m"
          echo "Please follow the instructions to increase amount of resources available:"
          echo "   https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#before-you-begin"
          echo
        fi
        mkdir -p /sources/logs /sources/dags /sources/plugins
        chown -R "${AIRFLOW_UID}:0" /sources/{logs,dags,plugins}
        exec /entrypoint airflow version
    # yamllint enable rule:line-length
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
      _PIP_ADDITIONAL_REQUIREMENTS: ''
    user: "0:0"
    volumes:
      - .:/sources

  airflow-cli:
    <<: *airflow-common
    profiles:
      - debug
    environment:
      <<: *airflow-common-env
      CONNECTION_CHECK_MAX_COUNT: "0"
    # Workaround for entrypoint issue. See: https://github.com/apache/airflow/issues/16252
    command:
      - bash
      - -c
      - airflow

  # You can enable flower by adding "--profile flower" option e.g. docker-compose --profile flower up   # or by explicitly targeted on the command line e.g. docker-compose up flower.   # See: https://docs.docker.com/compose/profiles/   flower:
    <<: *airflow-common
    command: celery flower
    profiles:
      - flower
    ports:
      - 5555:5555
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully

volumes:   
    postgres-db-volume:

Tried to solve the problem

1.

  • I have tried adding to x-airflow-common environment
  • AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags
  • I have deleted all my images and containers.
  • Rebuilt all images and docker-compse up the project
  • but it still did not works.
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I had to change the dag_id in the actual DAG files

    dag_id="hello_world", # <-- THIS HAVE TO BE UNIQUE

    Ex.:

    import pendulum

from airflow import DAG
from airflow.decorators import task

with DAG(
    dag_id="hello_world", ############################### <-- THIS HAVE TO BE UNIQUE
    schedule=None,
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
    tags=["example"],) as dag:

    @task()
    def print_array():
        """scipy."""
        from scipy.special import exp10   # <- THIS IS HOW IMPOTS SHOULD BE IMPORTED IN THIS CASE
        #define exp10 function and pass value in its
        exp = exp10([1,10])
        print(exp)

    print_array()

  2. 0
    Point (1): About puckel/docker-airflow image questions.

    • :/usr/local/airflow/dags this is the standard path for DAGs in this image.

    • webserver is not container name, it’s for entrypoint, By default entrypoint option in this image is webserver.

      docker run -d -p 8080:8080 -v /LOCAL_PATH/dags:/usr/local/airflow/dags IMAGE_NAME COMMAND_ENTRYPOINT[OPTIONAL]

    Point (2):

    Why you need change path for dags folder in container ?! but you can do that By,

    • After running containers using docker-compose up, you can execute docker exec -it <CONTAINER ID> bash. Then open airflow.cfg under path /opt/airflow/, You will need any text editor like VIM for editing the file.
    Point (3):
    • Check your volumes that mounted in docker-compose file, it should be
        volumes:
      - LOCAL_PATH/dags:/opt/airflow/dags
      - LOCAL_PATH/logs:/opt/airflow/logs
      - LOCAL_PATH/plugins:/opt/airflow/plugins
    • Change image name to your new image in docker-compose file, In your case AIRFLOW_IMAGE_NAME will be my38
        image: ${AIRFLOW_IMAGE_NAME:-my38:latest}
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search