Context
I’m asking this question because those two questions/answerse (one and two) have little context, and I’d liket o expand on it.
I’m trying to provision CentOS7 production servers with ansible client 2.8 . My environment consist one master node (NIS Server) and one compute node (NIS Client). Before provisioning live servers, I setted up a virtual lab (vlab) from VirtualBoxes, which mimics production environment. I copied my public keys to both production and vlab environment.
Problem
My playbooks work with vlab. Unfortunately, the same commands/playbooks are failing against production nodes to which I have ssh password less access and sudo rights. The only feedback that I get is this error "Timeout (7s) waiting for privilege escalation prompt: u001b[?1hu001b=rr"
Question
Does anyone know what causes this behaviour and how to fix it? I tried SeLinux permissions on home directory, I ended up disabling it (sudo setenforce 0
). I’m arriving to a conclusion that there has to be something external that stops ansible but?
Debugging, logging
Ansible Config
# config file for ansible -- https://raw.githubusercontent.com/ansible/ansible/devel/examples/ansible.cfg
[defaults]
timeout = 5
inventory = ./config/hosts
remote_user = lukas
SSHD Config
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTHPRIV
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
UsePAM yes
X11Forwarding yes
UseDNS no
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem sftp /usr/libexec/openssh/sftp-server
Ansible log
# checking kernel version
$ uname -a
Linux compute01 3.10.0-514.26.2.el7.x86_64
# pinging compute node
$ ansible -m ping compute01
123.123.123.123 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
# installing vim
$ ansible compute01 -m yum -a 'name=vim state=installed' -b -K -u lukas
ansible 2.8.4
config file = /home/lukas/Coding/projects/nebula-provision/ansible/producion/ansible.cfg
configured module search path = [u'/home/lukas/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609]
Using /home/lukas/Coding/projects/nebula-provision/ansible/producion/ansible.cfg as config file
BECOME password:
setting up inventory plugins
host_list declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
script declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
auto declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
Not replacing invalid character(s) "set([u'-'])" in group name (kubernetes-master)
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
....
Parsed /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts inventory source with ini plugin
Loading callback plugin minimal of type stdout, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/minimal.pyc
META: ran handlers
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861 `" && echo ansible-tmp-1566570853.88-235437120093861="` echo /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861 `" ) && sleep 0'"'"''
<123.123.123.123> (0, 'ansible-tmp-1566570853.88-235437120093861=/home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 19: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug2: fd 3 setting O_NONBLOCKrndebug2: mux_client_hello_exchange: master version 4rndebug3: mux_client_forwards: request forwardings: 0 local, 0 remoterndebug3: mux_client_request_session: enteringrndebug3: mux_client_request_alive: enteringrndebug3: mux_client_request_alive: done pid = 12697rndebug3: mux_client_request_session: session request sentrndebug1: mux_client_request_session: master session id: 2rndebug3: mux_client_read_packet: read header failed: Broken piperndebug2: Received exit status from master 0rn')
<123.123.123.123> Attempting python interpreter discovery
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<123.123.123.123> (0, 'PLATFORMnLinuxnFOUNDn/usr/bin/pythonn/usr/bin/python2.7n/usr/bin/pythonnENDFOUNDn', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 19: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug2: fd 3 setting O_NONBLOCKrndebug2: mux_client_hello_exchange: master version 4rndebug3: mux_client_forwards: request forwardings: 0 local, 0 remoterndebug3: mux_client_request_session: enteringrndebug3: mux_client_request_alive: enteringrndebug3: mux_client_request_alive: done pid = 12697rndebug3: mux_client_request_session: session request sentrndebug1: mux_client_request_session: master session id: 2rndebug3: mux_client_read_packet: read header failed: Broken piperndebug2: Received exit status from master 0rn')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<123.123.123.123> (0, '{"osrelease_content": "NAME=\"CentOS Linux\"\nVERSION=\"7 (Core)\"\nID=\"centos\"\nID_LIKE=\"rhel fedora\"\nVERSION_ID=\"7\"\nPRETTY_NAME=\"CentOS Linux 7 (Core)\"\nANSI_COLOR=\"0;31\"\nCPE_NAME=\"cpe:/o:centos:centos:7\"\nHOME_URL=\"https://www.centos.org/\"\nBUG_REPORT_URL=\"https://bugs.centos.org/\"\n\nCENTOS_MANTISBT_PROJECT=\"CentOS-7\"\nCENTOS_MANTISBT_PROJECT_VERSION=\"7\"\nREDHAT_SUPPORT_PRODUCT=\"centos\"\nREDHAT_SUPPORT_PRODUCT_VERSION=\"7\"\n\n", "platform_dist_result": ["centos", "7.3.1611", "Core"]}n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 19: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug2: fd 3 setting O_NONBLOCKrndebug2: mux_client_hello_exchange: master version 4rndebug3: mux_client_forwards: request forwardings: 0 local, 0 remoterndebug3: mux_client_request_session: enteringrndebug3: mux_client_request_alive: enteringrndebug3: mux_client_request_alive: done pid = 12697rndebug3: mux_client_request_session: session request sentrndebug1: mux_client_request_session: master session id: 2rndebug3: mux_client_read_packet: read header failed: Broken piperndebug2: Received exit status from master 0rn')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/system/setup.py
<123.123.123.123> PUT /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaIn TO /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py
<123.123.123.123> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 '[123.123.123.123]'
<123.123.123.123> (0, 'sftp> put /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaIn /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.pyn', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 19: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug2: fd 3 setting O_NONBLOCKrndebug2: mux_client_hello_exchange: master version 4rndebug3: mux_client_forwards: request forwardings: 0 local, 0 remoterndebug3: mux_client_request_session: enteringrndebug3: mux_client_request_alive: enteringrndebug3: mux_client_request_alive: done pid = 12697rndebug3: mux_client_request_session: session request sentrndebug1: mux_client_request_session: master session id: 2rndebug2: Remote version: 3rndebug2: Server supports extension "[email protected]" revision 1rndebug2: Server supports extension "[email protected]" revision 2rndebug2: Server supports extension "[email protected]" revision 2rndebug2: Server supports extension "[email protected]" revision 1rndebug2: Server supports extension "[email protected]" revision 1rndebug3: Sent message fd 5 T:16 I:1rndebug3: SSH_FXP_REALPATH . -> /home/lukas size 0rndebug3: Looking up /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaInrndebug3: Sent message fd 5 T:17 I:2rndebug3: Received stat reply T:101 I:2rndebug1: Couldn't stat remote file: No such file or directoryrndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.pyrndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 4 32768 bytes at 0rndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:8 O:131072 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:9 O:163840 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:10 O:196608 S:32768rndebug3: Sent message SSH2_FXP_WRITE I:11 O:229376 S:23124rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 5 32768 bytes at 32768rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 6 32768 bytes at 65536rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 7 32768 bytes at 98304rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 8 32768 bytes at 131072rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 9 32768 bytes at 163840rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 10 32768 bytes at 196608rndebug3: SSH2_FXP_STATUS 0rndebug3: In write loop, ack for 11 23124 bytes at 229376rndebug3: Sent message SSH2_FXP_CLOSE I:4rndebug3: SSH2_FXP_STATUS 0rndebug3: mux_client_read_packet: read header failed: Broken piperndebug2: Received exit status from master 0rn')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'chmod u+x /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/ /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py && sleep 0'"'"''
<123.123.123.123> (0, '', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 19: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug2: fd 3 setting O_NONBLOCKrndebug2: mux_client_hello_exchange: master version 4rndebug3: mux_client_forwards: request forwardings: 0 local, 0 remoterndebug3: mux_client_request_session: enteringrndebug3: mux_client_request_alive: enteringrndebug3: mux_client_request_alive: done pid = 12697rndebug3: mux_client_request_session: session request sentrndebug1: mux_client_request_session: master session id: 2rndebug3: mux_client_read_packet: read header failed: Broken piperndebug2: Received exit status from master 0rn')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 -tt 123.123.123.123 '/bin/sh -c '"'"'sudo -H -S -p "[sudo via ansible, key=kuswqyltevcovqytnefnxinbrwvcydkq] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-kuswqyltevcovqytnefnxinbrwvcydkq ; /usr/bin/python /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
123.123.123.123 | FAILED! => {
"msg": "Timeout (7s) waiting for privilege escalation prompt: u001b[?1hu001b=rr"
sshd log
Aug 26 13:36:19 123.123.123.123 sudo: pam_unix(sudo:auth): conversation failed
Aug 26 13:36:19 123.123.123.123 sudo: pam_unix(sudo:auth): auth could not identify password for [lukas]
3
Answers
Turned out that the problem relates to PAM auth module. In short, I added
auth sufficient pam_permit.so
line to/etc/pam.d/sudo
file. In long this is how I arrived to the solution.Be aware that there can be a million of reasons for PAM to fail, thus this solution might not work for you.
I believe the problem you are having is that you haven’t set the permission escalation user password. In Ansible, when we need
root
permissions, we set thebecome
variable. The method for becomingroot
may vary, so you can set the way using theansible_become_method
.Regarding your problem, I think you need to set up the
ansible_become_user
andansible_become_password
before running yourplaybook
. You can do it in your inventory, or wherever it feels right for your case.Here is the link to the variable list you can configure to modify how Ansible will connect to the hosts.
You can get more information about privilege escalation on the “Understanding Privilege Escalation” of Ansible Docs.
I hope it helps.
Just now I had the same issue and my command was
after removing the space in
I tried executing again and got
And I was like wt fun?
and then I saw that when executing this playbook I already was a
root
userso I’ve changed to a different user and tried executing this command again and thank God it worked.