Question 203860 in Docker Question posted in
The official documentation can be found here.

Docker – Storing std::cout to Python variable

I have a Python function that calls a C++ binary that uses cout << "Stuff" << endl; to output things to the screen. How can I store the output to a variable?

from openEMS import openEMS

def handler(event, context):
    FDTD = openEMS(NrTS=0, EndCriteria=0, TimeStepMethod=0)

    # I wish to redirect the output of this to a variable
    FDTD.Run("/tmp/engine", verbose=0, cleanup=True, numThreads=4)

I am currently able to see the output on the screen, and I do not mind writing to & reading from a file if it makes things easier. I would rather not have to modify the C++ code. Something like python3 script.py > /tmp/output.log 2>&1 and reading from the file would work, but it’s not very convenient as I’m using a Dockerized AWS Lambda function, and calling Python scripts multiple times seems inconvenient (but I’m open to whatever works):

ENTRYPOINT ["/usr/bin/python3", "-m", "awslambdaric"]
CMD ["lambda_handler.handler"]

Edit: @Tim Robers – Can’t post code blocks in comments, but if I understood correctly, I get ValueError: I/O operation on closed file.:

# Run simulation
f = open("/tmp/engine.log", "w")
stdout = sys.stdout.fileno()
sys.stdout.close()
os.dup2(f.fileno(), stdout)
FDTD.Run("/tmp/engine", verbose=0, cleanup=True, numThreads=4)

# Get unused primitives
with open("/tmp/engine.log", "r") as file:
    for line in file:
        print(line)

I guess we need to close the file somehow?

Tags:

2

Answers


  1. 0

    This is very easy for pure Python code, because you can just do sys.stdout = open('mylog.txt'). However, that doesn’t change the C runtime library file handles, so C code will still go to the terminal.

    It can be done, but it requires a little bit of trickery.

    The idea is that you get the "file handle" for your original stdout (which happens to be 2, but that’s beside the point). You then use os.dup2 make a duplicate of the log file handle, and duplicate it AS the original stdout handle number. Now, file handle #2 points to your file.

    So, this actually works. It redirects, runs a subshell, then puts things back where they were.

    import os
import io
import sys

def redirect_stdout(to_fd, stdout):
    """Redirect stdout to the given file descriptor."""
    sys.stdout.close()
    os.dup2(to_fd, stdout)
    sys.stdout = io.TextIOWrapper(os.fdopen(stdout, 'wb'))

f = open('mylog.txt','w')
stdout = sys.stdout.fileno()
saved = os.dup(stdout)

redirect_stdout( f.fileno(), stdout )
os.system('ls -l')
redirect_stdout( saved, stdout )

print("I'm back")

    Here’s how this works. We save a copy of the original stdout’s file handle. By convention, that’s file handle 2 (stdin is 0, sterr is 1). We then use dup to create a copy of this file. This simply creates a new file handle number that points to the original terminal file.

    We then close the Python sys.stdout, which also closes file handle 2, making it available. We then duplicate our NEW file’s handle and force it to be numbered 2. That file is now STDOUT to C code, or cout to C++ code. We then reopen sys.stdout to use that file handle (which is what I omitted before).

    To put things back to normal, we redo that process, but we duplicate the extra copy we made early on, and force THAT to be handle #2.

    This was derived from here:
    https://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/

    Login or Signup to reply.
  2. 0

    Redirecting the standard output (stdout) in Python can be a straightforward task for pure Python code using sys.stdout = open('mylog.txt'). However, this method doesn’t affect the C runtime library file handles, meaning that any C code executed will still output to the terminal.

    To achieve a complete redirection, a slightly more complex approach is required. The idea is to obtain the file handle for the original stdout (which is typically file handle 2), and then use os.dup2 to create a duplicate of the log file handle, replacing the original stdout handle number. This effectively redirects file handle #2 to the log file.

    The provided code demonstrates this approach. It defines a function called redirect_stdout that takes a file descriptor (to_fd) and the original stdout handle (stdout) as arguments. Inside the function, the current stdout is closed, os.dup2 is used to duplicate the log file handle as the original stdout handle number, and then sys.stdout is updated to use this new file handle.

    In the example code, an output file 'mylog.txt' is opened for writing, and the file descriptor for the current stdout is obtained using sys.stdout.fileno(). A copy of the original stdout file handle is saved. Then, redirect_stdout is called to redirect the stdout to the log file, followed by executing the command 'ls -l' using os.system. Afterward, the stdout is redirected back to its original state by calling redirect_stdout again, this time using the saved file handle.

    The explanation provided in the code highlights the steps involved in this process. Initially, a copy of the original stdout file handle is created using dup. This allows us to have a separate file handle pointing to the original terminal file. By closing sys.stdout and the associated file handle, we make it available for redirection. The duplicate file handle is then forced to be numbered as 2, becoming the stdout for C code or cout for C++ code. Finally, sys.stdout is reopened to use the newly redirected file handle.

    To revert the redirection and return to normal behavior, the process is repeated, but this time the extra copy made earlier is duplicated, and it is forced to be handle number 2, effectively restoring the original stdout.

    You can find more details and a source for this approach in the following link: https://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search