Amazon Web Services Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – Lifecycle Configuration File fails to auto-terminate amazon sagemaker studio instance

I am trying to configure a Lifecycle Configuration (LCC) file in order to implement an auto-terminate feature upon reaching an idle threshold for sagemaker’s code editor. I have followed the solution in this question. I am able to see and select the LCC file in the amazon sagemaker studio UI after successful completion of the steps but the instances do not shutdown after 1 hour of idle time (example below is 60 seconds for testing).

This is the LCC file:

# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: MIT-0

#!/bin/bash
set -eux
ASI_VERSION=0.3.0

# User variables [update as needed]
IDLE_TIME_IN_SECONDS=60  # in seconds, change this to desired idleness time before app shuts down

# System variables [do not change if not needed]
CONDA_HOME=/opt/conda/bin
LOG_FILE=/var/log/apps/app_container.log # Writing to app_container.log delivers logs to CW logs.
SOLUTION_DIR=/var/tmp/auto-stop-idle # Do not use /home/sagemaker-user
PYTHON_PACKAGE=sagemaker_code_editor_auto_shut_down-$ASI_VERSION.tar.gz
PYTHON_SCRIPT_PATH=$SOLUTION_DIR/sagemaker_code_editor_auto_shut_down/auto_stop_idle.py

# Installing cron
sudo apt-get update -y
sudo sh -c 'printf "#!/bin/shnexit 0" > /usr/sbin/policy-rc.d'
sudo apt-get install -y cron

# Creating solution directory.
sudo mkdir -p $SOLUTION_DIR

# Downloading autostop idle Python package.
echo "Downloading autostop idle Python package..."
curl -LO --output-dir /var/tmp/ https://github.com/aws-samples/sagemaker-studio-apps-lifecycle-config-examples/releases/download/v$ASI_VERSION/$PYTHON_PACKAGE
sudo $CONDA_HOME/pip install -U -t $SOLUTION_DIR /var/tmp/$PYTHON_PACKAGE

# Touch file to ensure idleness timer is reset to 0
echo "Touching file to reset idleness timer"
touch /opt/amazon/sagemaker/sagemaker-code-editor-server-data/data/User/History/startup_timestamp

# Setting container credential URI variable to /etc/environment to make it available to cron
sudo /bin/bash -c "echo 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI' >> /etc/environment"

# Add script to crontab for root.
echo "Adding autostop idle Python script to crontab..."
echo "*/2 * * * * /bin/bash -ic '$CONDA_HOME/python $PYTHON_SCRIPT_PATH --time $IDLE_TIME_IN_SECONDS --region $AWS_DEFAULT_REGION >> $LOG_FILE'" | sudo crontab -
Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Apparently there is a bug in the latest sagemaker distribution image that prevented cron from installing.

    Replacing this:

    sudo apt-get install -y cron

    with this:

    # Check if cron needs to be installed
status="$(dpkg-query -W --showformat='${db:Status-Status}' "cron" 2>&1)"
if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
    # Fixing invoke-rc.d: policy-rc.d denied execution of restart.
    sudo /bin/bash -c "echo '#!/bin/sh
    exit 0' > /usr/sbin/policy-rc.d"

    # Installing cron.
    echo "Installing cron..."
    sudo apt install cron
else
    echo "Package cron is already installed."
    sudo cron
fi

    ...solved my problem.


  2. 0

    Your general approach is correct if you intend to use CodeEditor.

    In your last question, you’re referring to JupyterLab. I’d suggest to validate that you’re actually using CodeEditor and not JupyterLab. SageMaker Studio provides both as editors, and each of these editors need different LCCs registered.

    For JupyterLab, you can find the LCC script and instructions on how to register the script here: https://github.com/aws-samples/sagemaker-studio-apps-lifecycle-config-examples/tree/main/jupyterlab/auto-stop-idle

    If your intention is to use CodeEditor and the LCC is not functioning, I’d suggest to check your CloudWatch logs. The LCC script establishes a cron job that checks the idle state every two minutes and logs this to CloudWatch.

    You also have the option to manually invoke the script in the CodeEditor app. Open a Terminal in CodeEditor (Terminal > New Terminal) and execute:

    # this example uses 1800 seconds as idle time threshold
/opt/conda/bin/python /var/tmp/auto-stop-idle/sagemaker_code_editor_auto_shut_down/auto_stop_idle.py --time 1800 --region $AWS_DEFAULT_REGION

    This should provide an output like:

    2024-04-16T06:16:55.359295z - [auto-stop-idle] - Logging time difference between current time and time files were last changed 80.18277621269226.
2024-04-16T06:16:55.359445z - [auto-stop-idle] - Logging time difference between current time and time files were last changed inf.
2024-04-16T06:16:55.359476z - [auto-stop-idle] - SageMaker Code Editor app is not idle. Passing check.

    If the idle time is exceeded, running the script will actually terminate your CodeEditor app. This should be a quick way to test if the script is working in your environment.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search