Question 258962 in Debian Question posted in
The oficial documentation can be found here.

Debian – subprocess (launch ssh process in background) communicate hang if enable stderr

I have next code, it will do next:

  1. subprocess for ssh -f -M to let ssh launch a shared socket in background
  2. As above is in background, so for the second ssh connection we could just reuse the socket /tmp/control-channel to connect ssh server without password.

test.py:

import subprocess
import os
import sys
import stat

ssh_user = "my_user"       # change to your account
ssh_passwd = "my_password" # change to your password

try:
    os.remove("/tmp/control-channel")
except:
    pass

# prepare passwd file
file = open("./passwd","w")
passwd_content = f"#!/bin/shnecho {ssh_passwd}"
file.write(passwd_content)
file.close()
os.chmod("./passwd", stat.S_IRWXU)

# setup shared ssh socket, put it in background
env = {'SSH_ASKPASS': "./passwd", 'DISPLAY':'', 'SSH_ASKPASS_REQUIRE':'force'}
args = ['ssh', '-f', '-o', 'LogLevel=ERROR', '-x', '-o', 'ConnectTimeout=30', '-o', 'ControlPersist=300', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'ServerAliveInterval=15', '-MN', '-S', '/tmp/control-channel', '-p', '22', '-l', ssh_user, 'localhost']
process = subprocess.Popen(args, env=env,
        stdout=subprocess.PIPE,
#        stderr=subprocess.STDOUT,   # uncomment this line to enable stderr will make subprocess hang
        stdin=subprocess.DEVNULL,
        start_new_session=True)
sout, serr = process.communicate()
print(sout)
print(serr)

# use shared socket
args2 = ['ssh', '-o', 'LogLevel=ERROR', '-o', 'ControlPath=/tmp/control-channel', '-p', '22', '-l', ssh_user, 'localhost', 'uname -a']
process2 = subprocess.Popen(args2,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        stdin=subprocess.DEVNULL)
content, _ = process2.communicate()
print(content)

execution:

$ python3 test.py
b''
None
b'Linux shmachine 4.19.0-21-amd64 #1 SMP Debian 4.19.249-2 (2022-06-30) x86_64 GNU/Linuxn'

So far so good, just if I uncomment stderr=subprocess.STDOUT in the first subprocess, it will hang:

$ python3 test.py
^CTraceback (most recent call last):
  File "test.py", line 29, in <module>
    sout, serr = process.communicate()
  File "/usr/lib/python3.7/subprocess.py", line 926, in communicate
    stdout = self.stdout.read()
KeyboardInterrupt

I wonder what’s the problem here?

My environment:

$ python3 --version
Python 3.7.3
$ ssh -V
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1n  15 Mar 2022
$ cat /etc/issue
Debian GNU/Linux 10 n l

UPDATE: I saw this post which similar to my issue, but no answer.

UPDATE2: Change communicate to wait makes it work, but the pipe size which wait use surely less than memory size which communicate use, so I still wonder why I can’t make it work with communicate.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I remove the stdout pipe set, just leave the stderr there to have a minimal check, and finally use strace to confirm it's the bug of old ssh which fixed in Y2020 in ssh 8.4. So python did correct behavior...

    1. I see it stuck as next:
    $ strace python3 test.py
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
read(3, 0x1744f00, 8192)                = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=26779, si_uid=1001, si_status=0, si_utime=1, si_stime=0} ---
read(3,

    $ sudo lsof -a -c python -d 3
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
python3 26778 nxa13855    3r  FIFO   0,12      0t0 9186593 pipe

    $ ls -l "/proc/26778/fd/3"
lr-x------ 1 nxa13855 atg 64 Oct  5 20:22 /proc/26778/fd/3 -> 'pipe:[9186593]'
$ lsof | grep 9186593
python3   26778                   nxa13855    3r     FIFO               0,12      0t0    9186593 pipe
ssh       26783                   nxa13855    2w     FIFO               0,12      0t0    9186593 pipe
    1. This means the ssh -f did not close stderr when put the ssh to background, so communicate can't read a EOF of stderr then hangs as next:
    File "/usr/lib/python3.7/subprocess.py", line 929, in communicate
stderr = self.stderr.read()

    And I confirm it with openssh commit


  2. 0

    The answer really just hides in the python documention subprocess.Popen, and within the documention of Popen.communicate, reads the following:

    Note: The data read is buffered in memory, so do not use this method if the data size is large or unlimited.

    Therefore, a (potentially) working solution (since I don’t really have a working SSH to test on), would be to add a timeout to the communicate call.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search