General Tips¶
The method
subprocess.run
is preferred over older high-level APIs (subprocess.call
,subprocess.check_call
andsubprocess.check_output
). The methodsubprocess.Popen
(which powers the high-level APIs) can be used if you need advanced controls. When running a shell command usingsubprocess.run
,Avoid using system shell (i.e., avoid using
shell=True
) for 2 reasons. First, avoid shell injection attack. Second, there is no need for you to manually escape special characters in the command.When passing a command to
subprocess.run
as a list (instead of a string), you should NOT do manual escaping. For example, if the commandpoker
has an option--cards
which requires a single space-separated string value, the cards have to be quoted (to escape spaces) when passed as a shell command.subprocess.run("poker --five-cards 'As 8d 9c Kh 3c'", shell=True, check=True)
And no extra quoting should be used when the command is passed as a list. That is, you should use
subprocess.run(["poker", "--five-cards", "As 8d 9c Kh 3c"], check=True)
instead of
subprocess.run(["poker", "--five-cards", "'As 8d 9c Kh 3c'"], check=True)
.
When passing a command to
subprocess.run
as a list (instead of a string), all elements of the command list must be one of the typesstr
,bytes
oros.PathLike
. Let's say that you have a command namesum
which takes space separated integers and sum them together, e.g.,sum 1 2 3
To pass the above command as a list to
subprocess.run
, you have usesubprocess.run(["sum", "1", "2", "3"], check=True)
instead of
subprocess.run(["sum", 1, 2, 3], check=True)
.
capture_output=True
is a convenient way to capture both stdout and sterr of the shell command. However, unless your code needs to parse and handle the stderr of the shell command, it is NOT a good idea to capture stderr of the shell command as the stderr should be printed to users.
When
subprocess.run(...)
fails to run, it throwssubprocess.CalledProcessError
. Sometimes, you might want to know the detailed exceptions/errors which caused the command to fail. In that case, you can capture the stderr and stdout of the child process and then parseCalledProcessError.stderr
andCalledProcessError.stdout
manually. For more discussions, please refer to How to catch exception output from Python subprocess.check_output()? .
Capture the Standard Ouput and Error¶
In Python 3.7+,
the output (stdout and stderr) of commands can be captured
by specifying the option capture_output=True
.
This option is equivalent to the options stdout=PIPE, stderr=PIPE
in older versions of Python.
Capture stdout by specifying stdout=sp.PIPE
.
process = sp.run(["pwd"], stdout=sp.PIPE, stderr=sp.PIPE)
print(process.stdout)
Capture both the standard ouput and error (separately).
process = sp.run(["pwd", "-l"], stdout=sp.PIPE, stderr=sp.PIPE)
print(process.stdout)
print(process.stderr)
Capture both the standard output and error in one place (process.stdout
).
process = sp.run(["pwd", "-l"], stdout=sp.PIPE, stderr=sp.STDOUT)
print(process.stdout)
Supress the Output of subprocess.run
¶
To suppress the output of subprocess.run
,
you can redirect the output to /dev/null
.
import os
import subprocess as sp
import sys
Without redicting the standard output to /dev/null
(i.e., supressing the standard output),
the command outputs results.
(Note that there is bug in ipykernel which supress the output.
This comamnd outputs results in a regular Python shell.)
sp.run(["ls", "-l"])
After redirecting the standard output to /dev/null
(i.e., supressing the standard output),
the command does not output any result.
with open(os.devnull, "w") as devnull:
sp.run(["ls", "-l"], stdout=devnull)
The code below supress both the stdout and stderr
by redirecting both of them to /dev/null
.
with open(os.devnull, "w") as devnull:
sp.run(["ls", "-l"], stdout=devnull, stderr=devnull)
Below is an equivalent approach,
which merges stderr to stdout first
and then redirect stdout to /dev/null
.
with open(os.devnull, "w") as devnull:
sp.run(["ls", "-l"], stdout=devnull, stderr=sp.STDOUT)
Comparison of Differenct Devices¶
sys.stdout
is the standard output stream.subprocess.STDOUT
refers to the standard out stream of subprocess. It is eithersubprocess.PIPE
orNone
.subprocess.DEVNULL
refers toos.devnull
.
Possible Exceptions¶
FileNotFoundError¶
If the command is not found,
subprocess.run
throws the exception FileNotFoundError
(even check=False
).
subprocess.CalledProcessError¶
If the command fails to run and
check=True
,subprocess.run
throws the exceptionsubprocess.CalledProcessError
.The error message of
CalledProcessError
is usuallyCommand '***' returned non-zero exit status 1
, which is not helpful. You can get more information about the child process via membersstdout
andstderr
.
import subprocess
command = ["ls", "/tmp2"]
try:
subprocess.check_output(command, stderr=subprocess.STDOUT).decode()
except subprocess.CalledProcessError as e:
print(type(e))
print("Error message:", e)
print("Standard output of the child process:", e.stdout)
print("Standard error of the child process:", e.stderr)
Issues in JupyterLab Notebooks¶
Running sp.run('ls -a')
in a JupyterLab notebook prints nothing
while running it in a regular Python shell prints results.
This is likely a bug in ipykernel.
The "Inappropriate ioctl for device" Issue¶
A shell command running using subprocess request input might throw the error message "Inappropriate ioctl for device" if the command requires input from a terminal. The post 使用popen和专用TTY Python运行交互式Bash talks about a way to fix the issue. The Python package pexpect providws even a simpler way to fix the problem. For more details, please refer to Hands on the Python Library pexpect .