Ben Chuanlong Du's Blog

And let it direct your passion with reason.

Hands on the Python module subprocess

General Tips

  1. The method subprocess.run is preferred over older high-level APIs (subprocess.call, subprocess.check_call and subprocess.check_output). The method subprocess.Popen (which powers the high-level APIs) can be used if you need advanced controls. When running a shell command using subprocess.run,

    1. Avoid using system shell (i.e., avoid using shell=True) for 2 reasons. First, avoid shell injection attack. Second, there is no need for you to manually escape special characters in the command.

    2. When passing a command to subprocess.run as a list (instead of a string), you should NOT do manual escaping. For example, if the command poker has an option --cards which requires a single space-separated string value, the cards have to be quoted (to escape spaces) when passed as a shell command.

       subprocess.run("poker --five-cards 'As 8d 9c Kh 3c'", shell=True, check=True)

      And no extra quoting should be used when the command is passed as a list. That is, you should use

       subprocess.run(["poker", "--five-cards", "As 8d 9c Kh 3c"], check=True)

      instead of

       subprocess.run(["poker", "--five-cards", "'As 8d 9c Kh 3c'"], check=True)

      .

    3. When passing a command to subprocess.run as a list (instead of a string), all elements of the command list must be one of the types str, bytes or os.PathLike. Let's say that you have a command name sum which takes space separated integers and sum them together, e.g.,

      sum 1 2 3

      To pass the above command as a list to subprocess.run, you have use

       subprocess.run(["sum", "1", "2", "3"], check=True)

      instead of

       subprocess.run(["sum", 1, 2, 3], check=True)

      .

    4. capture_output=True is a convenient way to capture both stdout and sterr of the shell command. However, unless your code needs to parse and handle the stderr of the shell command, it is NOT a good idea to capture stderr of the shell command as the stderr should be printed to users.

  2. When subprocess.run(...) fails to run, it throws subprocess.CalledProcessError. Sometimes, you might want to know the detailed exceptions/errors which caused the command to fail. In that case, you can capture the stderr and stdout of the child process and then parse CalledProcessError.stderr and CalledProcessError.stdout manually. For more discussions, please refer to How to catch exception output from Python subprocess.check_output()? .

Capture the Standard Ouput and Error

In Python 3.7+, the output (stdout and stderr) of commands can be captured by specifying the option capture_output=True. This option is equivalent to the options stdout=PIPE, stderr=PIPE in older versions of Python.

Capture stdout by specifying stdout=sp.PIPE.

In [23]:
process = sp.run(["pwd"], stdout=sp.PIPE, stderr=sp.PIPE)
print(process.stdout)

b'/app/archives/blog/misc/content\n'

Capture both the standard ouput and error (separately).

In [26]:
process = sp.run(["pwd", "-l"], stdout=sp.PIPE, stderr=sp.PIPE)
print(process.stdout)
print(process.stderr)

b''
b"pwd: invalid option -- 'l'\nTry 'pwd --help' for more information.\n"

Capture both the standard output and error in one place (process.stdout).

In [27]:
process = sp.run(["pwd", "-l"], stdout=sp.PIPE, stderr=sp.STDOUT)
print(process.stdout)

b"pwd: invalid option -- 'l'\nTry 'pwd --help' for more information.\n"

Supress the Output of subprocess.run

To suppress the output of subprocess.run, you can redirect the output to /dev/null.

In [13]:
import os
import subprocess as sp
import sys

Without redicting the standard output to /dev/null (i.e., supressing the standard output), the command outputs results. (Note that there is bug in ipykernel which supress the output. This comamnd outputs results in a regular Python shell.)

In [8]:
sp.run(["ls", "-l"])
Out[8]:
CompletedProcess(args=['ls', '-l'], returncode=0)

After redirecting the standard output to /dev/null (i.e., supressing the standard output), the command does not output any result.

In [1]:
with open(os.devnull, "w") as devnull:
    sp.run(["ls", "-l"], stdout=devnull)

The code below supress both the stdout and stderr by redirecting both of them to /dev/null.

In [9]:
with open(os.devnull, "w") as devnull:
    sp.run(["ls", "-l"], stdout=devnull, stderr=devnull)

Below is an equivalent approach, which merges stderr to stdout first and then redirect stdout to /dev/null.

In [11]:
with open(os.devnull, "w") as devnull:
    sp.run(["ls", "-l"], stdout=devnull, stderr=sp.STDOUT)

Comparison of Differenct Devices

  1. sys.stdout is the standard output stream. subprocess.STDOUT refers to the standard out stream of subprocess. It is either subprocess.PIPE or None.

  2. subprocess.DEVNULL refers to os.devnull.

Possible Exceptions

FileNotFoundError

If the command is not found, subprocess.run throws the exception FileNotFoundError (even check=False).

subprocess.CalledProcessError

  1. If the command fails to run and check=True, subprocess.run throws the exception subprocess.CalledProcessError.

  2. The error message of CalledProcessError is usually Command '***' returned non-zero exit status 1, which is not helpful. You can get more information about the child process via members stdout and stderr.

In [10]:
import subprocess

command = ["ls", "/tmp2"]
try:
    subprocess.check_output(command, stderr=subprocess.STDOUT).decode()
except subprocess.CalledProcessError as e:
    print(type(e))
    print("Error message:", e)
    print("Standard output of the child process:", e.stdout)
    print("Standard error of the child process:", e.stderr)

<class 'subprocess.CalledProcessError'>
Error message: Command '['ls', '/tmp2']' returned non-zero exit status 1.
Standard output of the child process: b'ls: /tmp2: No such file or directory\n'
Standard error of the child process: None

Issues in JupyterLab Notebooks

Running sp.run('ls -a') in a JupyterLab notebook prints nothing while running it in a regular Python shell prints results. This is likely a bug in ipykernel.

The "Inappropriate ioctl for device" Issue

A shell command running using subprocess request input might throw the error message "Inappropriate ioctl for device" if the command requires input from a terminal. The post 使用popen和专用TTY Python运行交互式Bash talks about a way to fix the issue. The Python package pexpect providws even a simpler way to fix the problem. For more details, please refer to Hands on the Python Library pexpect .

References

In [ ]:

Comments