skip to Main Content

Context

I’m asking this question because those two questions/answerse (one and two) have little context, and I’d liket o expand on it.

I’m trying to provision CentOS7 production servers with ansible client 2.8 . My environment consist one master node (NIS Server) and one compute node (NIS Client). Before provisioning live servers, I setted up a virtual lab (vlab) from VirtualBoxes, which mimics production environment. I copied my public keys to both production and vlab environment.

Problem

My playbooks work with vlab. Unfortunately, the same commands/playbooks are failing against production nodes to which I have ssh password less access and sudo rights. The only feedback that I get is this error "Timeout (7s) waiting for privilege escalation prompt: u001b[?1hu001b=rr"

Question

Does anyone know what causes this behaviour and how to fix it? I tried SeLinux permissions on home directory, I ended up disabling it (sudo setenforce 0). I’m arriving to a conclusion that there has to be something external that stops ansible but?

Debugging, logging

Ansible Config

# config file for ansible -- https://raw.githubusercontent.com/ansible/ansible/devel/examples/ansible.cfg

[defaults]
timeout = 5
inventory = ./config/hosts
remote_user = lukas

SSHD Config

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

SyslogFacility AUTHPRIV

AuthorizedKeysFile      .ssh/authorized_keys

PasswordAuthentication yes

ChallengeResponseAuthentication no

GSSAPIAuthentication yes
GSSAPICleanupCredentials no

UsePAM yes

X11Forwarding yes
UseDNS no

AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS

Subsystem       sftp    /usr/libexec/openssh/sftp-server

Ansible log

# checking kernel version
$ uname -a
Linux compute01 3.10.0-514.26.2.el7.x86_64

# pinging compute node
$ ansible -m ping compute01
123.123.123.123 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

# installing vim
$ ansible compute01 -m yum -a 'name=vim state=installed' -b -K -u lukas
ansible 2.8.4
  config file = /home/lukas/Coding/projects/nebula-provision/ansible/producion/ansible.cfg
  configured module search path = [u'/home/lukas/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609]
Using /home/lukas/Coding/projects/nebula-provision/ansible/producion/ansible.cfg as config file
BECOME password: 
setting up inventory plugins
host_list declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
script declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
auto declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
Not replacing invalid character(s) "set([u'-'])" in group name (kubernetes-master)
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be disabled by setting
 deprecation_warnings=False in ansible.cfg.
....
Parsed /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts inventory source with ini plugin
Loading callback plugin minimal of type stdout, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/minimal.pyc
META: ran handlers
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861 `" && echo ansible-tmp-1566570853.88-235437120093861="` echo /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861 `" ) && sleep 0'"'"''
<123.123.123.123> (0, 'ansible-tmp-1566570853.88-235437120093861=/home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 19: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug2: fd 3 setting O_NONBLOCKrndebug2: mux_client_hello_exchange: master version 4rndebug3: mux_client_forwards: request forwardings: 0 local, 0 remoterndebug3: mux_client_request_session: enteringrndebug3: mux_client_request_alive: enteringrndebug3: mux_client_request_alive: done pid = 12697rndebug3: mux_client_request_session: session request sentrndebug1: mux_client_request_session: master session id: 2rndebug3: mux_client_read_packet: read header failed: Broken piperndebug2: Received exit status from master 0rn')
<123.123.123.123> Attempting python interpreter discovery
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<123.123.123.123> (0, 'PLATFORMnLinuxnFOUNDn/usr/bin/pythonn/usr/bin/python2.7n/usr/bin/pythonnENDFOUNDn', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 19: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug2: fd 3 setting O_NONBLOCKrndebug2: mux_client_hello_exchange: master version 4rndebug3: mux_client_forwards: request forwardings: 0 local, 0 remoterndebug3: mux_client_request_session: enteringrndebug3: mux_client_request_alive: enteringrndebug3: mux_client_request_alive: done pid = 12697rndebug3: mux_client_request_session: session request sentrndebug1: mux_client_request_session: master session id: 2rndebug3: mux_client_read_packet: read header failed: Broken piperndebug2: Received exit status from master 0rn')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<123.123.123.123> (0, '{"osrelease_content": "NAME=\"CentOS Linux\"\nVERSION=\"7 (Core)\"\nID=\"centos\"\nID_LIKE=\"rhel fedora\"\nVERSION_ID=\"7\"\nPRETTY_NAME=\"CentOS Linux 7 (Core)\"\nANSI_COLOR=\"0;31\"\nCPE_NAME=\"cpe:/o:centos:centos:7\"\nHOME_URL=\"https://www.centos.org/\"\nBUG_REPORT_URL=\"https://bugs.centos.org/\"\n\nCENTOS_MANTISBT_PROJECT=\"CentOS-7\"\nCENTOS_MANTISBT_PROJECT_VERSION=\"7\"\nREDHAT_SUPPORT_PRODUCT=\"centos\"\nREDHAT_SUPPORT_PRODUCT_VERSION=\"7\"\n\n", "platform_dist_result": ["centos", "7.3.1611", "Core"]}n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 19: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug2: fd 3 setting O_NONBLOCKrndebug2: mux_client_hello_exchange: master version 4rndebug3: mux_client_forwards: request forwardings: 0 local, 0 remoterndebug3: mux_client_request_session: enteringrndebug3: mux_client_request_alive: enteringrndebug3: mux_client_request_alive: done pid = 12697rndebug3: mux_client_request_session: session request sentrndebug1: mux_client_request_session: master session id: 2rndebug3: mux_client_read_packet: read header failed: Broken piperndebug2: Received exit status from master 0rn')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/system/setup.py
<123.123.123.123> PUT /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaIn TO /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py
<123.123.123.123> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 '[123.123.123.123]'
<123.123.123.123> (0, 'sftp> put /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaIn /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.pyn', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 19: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug2: fd 3 setting O_NONBLOCKrndebug2: mux_client_hello_exchange: master version 4rndebug3: mux_client_forwards: request forwardings: 0 local, 0 remoterndebug3: mux_client_request_session: enteringrndebug3: mux_client_request_alive: enteringrndebug3: mux_client_request_alive: done pid = 12697rndebug3: mux_client_request_session: session request sentrndebug1: mux_client_request_session: master session id: 2rndebug2: Remote version: 3rndebug2: Server supports extension "[email protected]" revision 1rndebug2: Server supports extension "[email protected]" revision 2rndebug2: Server supports extension "[email protected]" revision 2rndebug2: Server supports extension "[email protected]" revision 1rndebug2: Server supports extension "[email protected]" revision 1rndebug3: Sent message fd 5 T:16 I:1rndebug3: SSH_FXP_REALPATH . -> /home/lukas size 0rndebug3: Looking up /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaInrndebug3: Sent message fd 5 T:17 I:2rndebug3: Received stat reply T:101 I:2rndebug1: Couldn't stat remote file: No such file or directoryrndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.pyrndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 4 32768 bytes at 0rndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:8 O:131072 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:9 O:163840 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:10 O:196608 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:11 O:229376 S:23124rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 5 32768 bytes at 32768rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 6 32768 bytes at 65536rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 7 32768 bytes at 98304rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 8 32768 bytes at 131072rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 9 32768 bytes at 163840rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 10 32768 bytes at 196608rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 11 23124 bytes at 229376rndebug3: Sent message SSH2_FXP_CLOSE I:4rndebug3: SSH2_FXP_STATUS 0rndebug3: mux_client_read_packet: read header failed: Broken piperndebug2: Received exit status from master 0rn')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'chmod u+x /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/ /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py && sleep 0'"'"''
<123.123.123.123> (0, '', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 19: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug2: fd 3 setting O_NONBLOCKrndebug2: mux_client_hello_exchange: master version 4rndebug3: mux_client_forwards: request forwardings: 0 local, 0 remoterndebug3: mux_client_request_session: enteringrndebug3: mux_client_request_alive: enteringrndebug3: mux_client_request_alive: done pid = 12697rndebug3: mux_client_request_session: session request sentrndebug1: mux_client_request_session: master session id: 2rndebug3: mux_client_read_packet: read header failed: Broken piperndebug2: Received exit status from master 0rn')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 -tt 123.123.123.123 '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=kuswqyltevcovqytnefnxinbrwvcydkq] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-kuswqyltevcovqytnefnxinbrwvcydkq ; /usr/bin/python /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
123.123.123.123 | FAILED! => {
    "msg": "Timeout (7s) waiting for privilege escalation prompt: u001b[?1hu001b=rr"

sshd log

Aug 26 13:36:19 123.123.123.123 sudo: pam_unix(sudo:auth): conversation failed
Aug 26 13:36:19 123.123.123.123 sudo: pam_unix(sudo:auth): auth could not identify password for [lukas]

3

Answers


  1. Chosen as BEST ANSWER

    Turned out that the problem relates to PAM auth module. In short, I added auth sufficient pam_permit.so line to /etc/pam.d/sudo file. In long this is how I arrived to the solution.

    Be aware that there can be a million of reasons for PAM to fail, thus this solution might not work for you.

    $ cat /etc/pam.d/sudo
    
    #%PAM-1.0
    # Fixing "auth could not identify password for [username]" ssh problem.
    auth       sufficient   pam_permit.so
    
    # Original config below
    auth       include      system-auth
    account    include      system-auth
    password   include      system-auth
    session    optional     pam_keyinit.so revoke
    session    required     pam_limits.so
    session    include      system-auth
    

  2. I believe the problem you are having is that you haven’t set the permission escalation user password. In Ansible, when we need root permissions, we set the become variable. The method for becoming root may vary, so you can set the way using the ansible_become_method.

    Regarding your problem, I think you need to set up the ansible_become_user and ansible_become_password before running your playbook. You can do it in your inventory, or wherever it feels right for your case.

    Here is the link to the variable list you can configure to modify how Ansible will connect to the hosts.

    You can get more information about privilege escalation on the “Understanding Privilege Escalation” of Ansible Docs.

    I hope it helps.

    Login or Signup to reply.
  3. Just now I had the same issue and my command was

    ./ansible-playbook playbook.yml -i hosts -b --become-user root --extra-vars "ansible_become_pass= myPass" --become-method su
    

    after removing the space in

    "ansible_become_pass=[space]myPass" --> "ansible_become_pass=myPass"
    

    I tried executing again and got

    The authenticity of host 'localhost (::1)' can't be established.
    ECDSA key fingerprint is SHA256:[random string].
    Are you sure you want to continue connecting (yes/no/[fingerprint])?
    

    And I was like wt fun?

    and then I saw that when executing this playbook I already was a root user
    so I’ve changed to a different user and tried executing this command again and thank God it worked.

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search