Ubuntu – Why does Python unittest auto-discovery not work when running in a subprocess?

I’d like to be able to run Python’s unittest module programmatically via a subprocess (e.g. subprocess.Popen(), subprocess.run(), asyncio.create_subprocess_exec()) and have it auto-discover tests.

I do not want to run the tests by importing the unittest module into my script, because I would like the same code to be able to run any arbitrary command from the command line, and I’d like to avoid handling running tests differently than other commands.

Example Code

Here is a GitHub repository with code that illustrates the issue I’m seeing: https://github.com/sscovil/python-subprocess

For completeness, I’ll include it here as well.

.
├── src
│   ├── __init__.py
│   └── example
│       ├── __init__.py
│       └── runner.py
└── test
    ├── __init__.py
    └── example
        ├── __init__.py
        └── runner_test.py

src/example/runner.py

import asyncio
import os
import shutil
import subprocess
import unittest
from subprocess import CompletedProcess, PIPE
from typing import Final, List

UNIT_TEST_CMD: Final[str] = "python -m unittest discover test '*_test.py' --locals -b -c -f"


def _parse_cmd(cmd: str) -> List[str]:
    """Helper function that splits a command string into a list of arguments with a full path to the executable."""
    args: List[str] = cmd.split(" ")
    args[0] = shutil.which(args[0])
    return args


async def async_exec(cmd: str, *args, **kwargs) -> int:
    """Runs a command using asyncio.create_subprocess_exec() and logs the output."""
    cmd_args: List[str] = _parse_cmd(cmd)
    process = await asyncio.create_subprocess_exec(*cmd_args, stdout=PIPE, stderr=PIPE, *args, **kwargs)
    stdout, stderr = await process.communicate()
    if stdout:
        print(stdout.decode().strip())
    else:
        print(stderr.decode().strip())
    return process.returncode


def popen(cmd: str, *args, **kwargs) -> int:
    """Runs a command using subprocess.call() and logs the output."""
    cmd_args: List[str] = _parse_cmd(cmd)
    with subprocess.Popen(cmd_args, stdout=PIPE, stderr=PIPE, text=True, *args, **kwargs) as process:
        stdout, stderr = process.communicate()
        if stdout:
            print(stdout.strip())
        else:
            print(stderr.strip())
        return process.returncode


def run(cmd: str, *args, **kwargs) -> int:
    """Runs a command using subprocess.run() and logs the output."""
    cmd_args: List[str] = _parse_cmd(cmd)
    process: CompletedProcess = subprocess.run(cmd_args, stdout=PIPE, stderr=PIPE, check=True, *args, **kwargs)
    if process.stdout:
        print(process.stdout.decode().strip())
    else:
        print(process.stderr.decode().strip())
    return process.returncode


def unittest_discover() -> unittest.TestResult:
    """Runs all tests in the given directory that match the given pattern, and returns a TestResult object."""
    start_dir = os.path.join(os.getcwd(), "test")
    pattern = "*_test.py"
    tests = unittest.TextTestRunner(buffer=True, failfast=True, tb_locals=True, verbosity=2)
    results = tests.run(unittest.defaultTestLoader.discover(start_dir=start_dir, pattern=pattern))
    return results


def main():
    """Runs the example."""
    print("nRunning tests using asyncio.create_subprocess_exec...n")
    asyncio.run(async_exec(UNIT_TEST_CMD))

    print("nRunning tests using subprocess.Popen...n")
    popen(UNIT_TEST_CMD)

    print("nRunning tests using subprocess.run...n")
    run(UNIT_TEST_CMD)

    print("nRunning tests using unittest.defaultTestLoader...n")
    unittest_discover()


if __name__ == "__main__":
    main()

test/example/runner_test.py

import unittest

from src.example.runner import async_exec, popen, run, unittest_discover


class AsyncTestRunner(unittest.IsolatedAsyncioTestCase):
    async def test_async_call(self):
        self.assertEqual(await async_exec("echo Hello"), 0)


class TestRunners(unittest.TestCase):
    def test_popen(self):
        self.assertEqual(popen("echo Hello"), 0)

    def test_run(self):
        self.assertEqual(run("echo Hello"), 0)

    def test_unittest_discover(self):
        results = unittest_discover()
        self.assertEqual(results.testsRun, 4)  # There are 4 test cases in this file


if __name__ == "__main__":
    unittest.main()

Expected Behavior

When running tests from the command line, Python’s unittest module auto-discovers tests in the test directory:

python -m unittest discover test '*_test.py' --locals -bcf
....
----------------------------------------------------------------------
Ran 4 tests in 0.855s

OK

Actual Behavior

…but it fails to auto-discover tests when that same command is run using Python’s subprocess module:

$ python -m src.example.runner

Running tests using asyncio.create_subprocess_exec...

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

Running tests using subprocess.Popen...

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

Running tests using subprocess.run...

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

Running tests using unittest.defaultTestLoader...

test_async_call (example.runner_test.AsyncTestRunner.test_async_call) ... ok
test_popen (example.runner_test.TestRunners.test_popen) ... ok
test_run (example.runner_test.TestRunners.test_run) ... ok
test_unittest_discover (example.runner_test.TestRunners.test_unittest_discover) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.864s

OK

Note that the unittest.defaultTestLoader test runner works as expected, because it is explicitly using the unittest module to run the other tests. However, when running tests using asyncio.create_subprocess_exec, subprocess.Popen, or subprocess.run, as if using the CLI from the command line, the tests are not auto-discovered.

Different Python Versions

If you have Docker installed, you can run the tests in a container using any version of Python you like. For example:

Python 3.11 on Alpine Linux

docker run -it --rm -v $(pwd):$(pwd) -w $(pwd) --name test python:3.11-alpine python3 -m src.example.runner

Python 3.10 on Ubuntu Linux

docker run -it --rm -v $(pwd):$(pwd) -w $(pwd) --name test python:3.10 python3 -m src.example.runner

In every version I tried, from 3.8 to 3.11, I saw the same results.

Question

Why does Python unittest auto-discovery not work when running in a subprocess?