Question 85444 in Docker Question posted in
The official documentation can be found here.

AWX all jobs stop processing and hang indefinitely — why – Docker

Problem

We’ve had a working Ansible AWX instance running on v5.0.0 for over a year, and suddenly all jobs stop working — no output is rendered. They will start "running" but hang indefinitely without printing out any logging.

The AWX instance is running in a docker compose container setup as defined here: https://github.com/ansible/awx/blob/5.0.0/INSTALL.md#docker-compose

Observations

Standard troubleshooting such as restarting of containers, host OS, etc. hasn’t helped. No configuration changes in either environment.

Upon debugging an actual playbook command, we observe that the command to run a playbook from the UI is like the below:

ssh-agent sh -c ssh-add /tmp/awx_11021_0fmwm5uz/artifacts/11021/ssh_key_data && rm -f /tmp/awx_11021_0fmwm5uz/artifacts/11021/ssh_key_data && ansible-playbook -vvvvv -u ubuntu --become --ask-vault-pass -i /tmp/awx_11021_0fmwm5uz/tmppo7rcdqn -e @/tmp/awx_11021_0fmwm5uz/env/extravars playbook.yml

That’s broken down into three commands in sequence:

  1. ssh-agent sh -c ssh-add /tmp/awx_11021_0fmwm5uz/artifacts/11021/ssh_key_data
  2. rm -f /tmp/awx_11021_0fmwm5uz/artifacts/11021/ssh_key_data
  3. ansible-playbook -vvvvv -u ubuntu --become --ask-vault-pass -i /tmp/awx_11021_0fmwm5uz/tmppo7rcdqn -e @/tmp/awx_11021_0fmwm5uz/env/extravars playbook.yml

You can see in part 3, the -vvvvv is the debugging argument — however, the hang is happening on command #1. Which has nothing to do with ansible or AWX specifically, but it’s not going to get us much debugging info.

I tried doing an strace to see what is going on, but for reasons given below, it is pretty difficult to follow what it is actually hanging on. I can provide this output if it might help.

Analysis

So one natural question with command #1 — what is ‘ssh_key_data’?

Well it’s what we set up to be the Machine credential in AWX (an SSH key) — it hasn’t changed in a while and it works just fine when used in a direct SSH command. It’s also apparently being set up by AWX as a file pipe:

prw------- 1 root root 0 Dec 10 08:29 ssh_key_data

Which starts to explain why it could be potentially hanging (if nothing is being read in from the other side of the pipe).

Running a normal ansible-playbook from command line (and supplying the SSH key in a more normal way) works just fine, so we can still deploy, but only via CLI right now — it’s just AWX that is broken.

Conclusions

So the question then becomes "why now"? And "how to debug"? I have checked the health of awx_postgres, and verified that indeed the Machine credential is present in an expected format (in the main_credential table). I have also verified that can use ssh-agent on the awx_task container without the use of that pipe keyfile. So it really seems to be this piped file that is the problem — but I haven’t been able to glean from any logs where the other side of the pipe (sender) is supposed to be or why they aren’t sending the data.

Tags:

5

Answers


  1. 0

    Had the same issue starting this Friday in the same timeframe as you. Turned out that Crowdstrike (falcon sensor) Agent was the culprit. I’m guessing they pushed a definition update that is breaking or blocking fifo pipes. When we stopped the CS agent, AWX started working correctly again, with no issues. See if you are running a similar security product.

    Login or Signup to reply.
  2. 0

    For users of Crowdstrike, the problem is likely related to a policy change implemented by your organization over the weekend:

    crowdstrike released version 6.32, which was adopted by many organizations to respond to a log4j vulnerability over the weekend, which introduced some changes around script level inspection.

    Script-Based Execution Monitoring is the culprit of the disruption. As other users have said, you can disable crowdstrike entirely and restart AWX jobs to get it working, but for security in production that may not be appropriate.

    Instead, you must contact your crowdstrike administrator who will have updated the policy of your instance profile to include Script-Based Execution Monitoring. The policy management GUI has a checkbox which can enable/disable the use of this feature (new in 6.32). Ask them to disable it and send logs to the vendor.

    Login or Signup to reply.
  3. 0

    Confirmed Crowdstrike policy update was the issue with why Ansible Tower stopped working for 48 hrs at our company as well. Disabling the monitor option allowed jobs to run successfully almost instantly.

    Login or Signup to reply.
  4. 0

    I confirm same situation even if AWX is running on container…

    Regarding existing solutions, none fitting for my case:

    1. Disable Crowdstrike service… Impossible for security reason
    2. Asking to disable "Script-Based Execution Monitoring" will be headache in my current position

    So I’ve patched ansible-runner to avoid letting shell/bash (aka Script-based) reading the file and let Python do it

    I’m using old version of AWX 15.0.1 and so the corresponding ansible-runner maybe a bit old. But I’ve checked on master method still exists (but on different file)

    I’ve moved from

    def wrap_args_with_ssh_agent(self, args, ssh_key_path, ssh_auth_sock=None, silence_ssh_add=False):
    """
    Given an existing command line and parameterization this will return the same command line wrapped with the
    necessary calls to ``ssh-agent``
    """
    if ssh_key_path:
        ssh_add_command = args2cmdline('ssh-add', ssh_key_path)
    ...

    to

    def wrap_args_with_ssh_agent(self, args, ssh_key_path, ssh_auth_sock=None, silence_ssh_add=False):
    """
    Given an existing command line and parameterization this will return the same command line wrapped with the
    necessary calls to ``ssh-agent``
    """
    if ssh_key_path:
        ssh_key_content = ""
        with open(ssh_key_path) as f:
            ssh_key_content = f.read()

        ssh_add_command = 'ssh-add - <<< "' + ssh_key_content + '"'

    ATTENTION the private key content will be exposed on log. Is simply a POC, I think we can do differently and better (maybe rewriting content on other file? and using that new clone)

    Login or Signup to reply.
  5. 0

    Crowdstrike has released sensor version 6.32.12905 to address this issue. Whomever configures Falcon in your environment can create/modify the Sensor Update Policy to push the latest version to the affected hosts.

    Release Notes:
    Summary
    The hotfix release resolves an issue with Script-Based Execution Monitoring in which the Linux sensor might hang a command line that includes a pipe file.

    Fixed
    Resolves an issue with Script-Based Execution Monitoring in which the Linux sensor might hang a command line that includes a pipe file

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search