Python system call subprocess

Python 3 Subprocess Examples

All examples use Python 3.5 or later (unless noted) and assume you’re running Linux or a unix-based OS.

All examples can be found on this Jupyter notebook

call() example

With suprocess.call() you pass an array of commands and parameters.

subprocess.call() returns the return code of the called process.

import subprocess subprocess.call(["ls", "-lha"]) # >>> 0 (the return code) 

subprocess.call() does not raise an exception if the underlying process errors!

import subprocess # no Python Exception is thrown! subprocess.call(["./bash-script-with-bad-syntax"]) # >>> 127 

call() example with shell=True

If shell=True , the command string is interpreted as a raw shell command.

Using shell=True may expose you to code injection if you use user input to build the command string.

subprocess.call("ls -lha", shell=True) # returns 0 (the return code) 

call() example, capture stdout and stderr

If you are on Python 3.5+, use subprocess.run() instead as it’s safer.

import subprocess import sys # create two files to hold the output and errors, respectively with open('out.txt','w+') as fout: with open('err.txt','w+') as ferr: out=subprocess.call(["ls",'-lha'],stdout=fout,stderr=ferr) # reset file to read from it fout.seek(0) # save output (if any) in variable output=fout.read()) # reset file to read from it ferr.seek(0) # save errors (if any) in variable errors = ferr.read() output # total 20K # drwxrwxr-x 3 felipe felipe 4,0K Nov 4 15:28 . # drwxrwxr-x 39 felipe felipe 4,0K Nov 3 18:31 .. # drwxrwxr-x 2 felipe felipe 4,0K Nov 3 19:32 .ipynb_checkpoints # -rw-rw-r-- 1 felipe felipe 5,5K Nov 4 15:28 main.ipynb errors # '' empty string 

call() example, force exception if process causes error

import subprocess # unlike subprocess.call, this throws a CalledProcessError # if the underlying process errors out subprocess.check_call(["./bash-script-with-bad-syntax"]) 

Run command and capture output

Using universal_newlines=True converts the output to a string instead of a byte array.

import subprocess # errors in the created process are raised here too output = subprocess.check_output(["ls","-lha"],universal_newlines=True) output # total 20K # drwxrwxr-x 3 felipe felipe 4,0K Nov 4 15:28 . # drwxrwxr-x 39 felipe felipe 4,0K Nov 3 18:31 .. # drwxrwxr-x 2 felipe felipe 4,0K Nov 3 19:32 .ipynb_checkpoints # -rw-rw-r-- 1 felipe felipe 5,5K Nov 4 15:28 main.ipynb 

import subprocess # run() returns a CompletedProcess object if it was successful # errors in the created process are raised here too process = subprocess.run(['ls','-lha'], check=True, stdout=subprocess.PIPE, universal_newlines=True) output = process.stdout output # total 20K # drwxrwxr-x 3 felipe felipe 4,0K Nov 4 15:28 . # drwxrwxr-x 39 felipe felipe 4,0K Nov 3 18:31 .. # drwxrwxr-x 2 felipe felipe 4,0K Nov 3 19:32 .ipynb_checkpoints # -rw-rw-r-- 1 felipe felipe 5,5K Nov 4 15:28 main.ipynb 

Run raw string as shell command line

Don’t do this if your string uses user input, as they may inject arbitrary code!

This is similar to the example above, with shell=True

import subprocess # errors in the created process are raised here too output = subprocess.check_output("ls -lha", shell=True, universal_newlines=True) output # total 20K # drwxrwxr-x 3 felipe felipe 4,0K Nov 4 15:28 . # drwxrwxr-x 39 felipe felipe 4,0K Nov 3 18:31 .. # drwxrwxr-x 2 felipe felipe 4,0K Nov 3 19:32 .ipynb_checkpoints # -rw-rw-r-- 1 felipe felipe 5,5K Nov 4 15:28 main.ipynb 

import subprocess # run() returns a CompletedProcess object if it was successful # errors in the created process are raised here too process = subprocess.run('ls -lha', shell=True, check=True, stdout=subprocess.PIPE, universal_newlines=True) output = process.stdout output # total 20K # drwxrwxr-x 3 felipe felipe 4,0K Nov 4 15:28 . # drwxrwxr-x 39 felipe felipe 4,0K Nov 3 18:31 .. # drwxrwxr-x 2 felipe felipe 4,0K Nov 3 19:32 .ipynb_checkpoints # -rw-rw-r-- 1 felipe felipe 5,5K Nov 4 15:28 main.ipynb 

run() example: run command and get return code

run() behaves mostly the same way as call() and you should use it instead of call() for version 3.5 onwards.

subprocess.run() does not raise an exception if the underlying process errors!

import subprocess cp = subprocess.run(["ls","-lha"]) cp # CompletedProcess(args=['ls', '-lha'], returncode=0) 

run() example: run command, force exception if process errors

Use check=True to force the Python method to throw an exception if the underlying process encounters errors:

import subprocess subprocess.run(["ls","foo bar"], check=True) # ------------------------------------------------------------------- # CalledProcessError Traceback (most recent call last) # ----> 1 subprocess.run(["ls","foo bar"], check=True) # /usr/lib/python3.6/subprocess.py in run(input, timeout, check, *popenargs, **kwargs) # 416 if check and retcode: # 417 raise CalledProcessError(retcode, process.args, # --> 418 output=stdout, stderr=stderr) # 419 return CompletedProcess(process.args, retcode, stdout, stderr) # 420 # CalledProcessError: Command '['ls', 'foo bar']' returned non-zero exit status 2. 

run() example: using shell=True

As in the call() example, shell=True , the command string is interpreted as a raw shell command.

Again, Using shell=True may expose you to code injection if you use user input to build the command string.

import subprocess cp = subprocess.run(["ls -lha"],shell=True) cp # CompletedProcess(args=['ls -lha'], returncode=0) 

run() example: store output and error message in string

If the underlying process returns a nonzero exit code, you will not get an exception; the error message can be accessed via the stderr attribute in the CompletedProcess object.

case 1: process return 0 exit code

import subprocess cp = subprocess.run(["ls","-lha"], universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) cp.stdout # total 20K # drwxrwxr-x 3 felipe felipe 4,0K Nov 4 15:28 . # drwxrwxr-x 39 felipe felipe 4,0K Nov 3 18:31 .. # drwxrwxr-x 2 felipe felipe 4,0K Nov 3 19:32 .ipynb_checkpoints # -rw-rw-r-- 1 felipe felipe 5,5K Nov 4 15:28 main.ipynb cp.stderr # '' (empty string) cp.returncode # 0 

import subprocess cp = subprocess.run(["ls","foo bar"], universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) cp.stdout # '' (empty string) cp.stderr # ls: cannot access 'foo bar': No such file or directory cp.returncode # 2 

import subprocess try: cp = subprocess.run(["xxxx","foo bar"], universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) except FileNotFoundError as e: print(e) # [Errno 2] No such file or directory: 'xxxx' 

Popen example: run command and get return code

subprocess.Popen() is used for more complex examples where you need. See Popen() vs call() vs run()

This causes the python program to block until the subprocess returns.

from subprocess import Popen p = Popen(["ls","-lha"]) p.wait() # 0 

Popen example: Store output and error messages in string

import subprocess from subprocess import Popen p = Popen(["ls","-lha"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) output, errors = p.communicate() output # total 20K # drwxrwxr-x 3 felipe felipe 4,0K Nov 4 15:28 . # drwxrwxr-x 39 felipe felipe 4,0K Nov 3 18:31 .. # drwxrwxr-x 2 felipe felipe 4,0K Nov 3 19:32 .ipynb_checkpoints # -rw-rw-r-- 1 felipe felipe 5,5K Nov 4 15:28 main.ipynb errors # '' (empty string) 

Popen example: Redirect output to file

import subprocess from subprocess import Popen path_to_output_file = '/tmp/myoutput.txt' myoutput = open(path_to_output_file,'w+') p = Popen(["ls","-lha"], stdout=myoutput, stderr=subprocess.PIPE, universal_newlines=True) output, errors = p.communicate() output # there's nothing here because we didn't set stdout=subprocess.PIPE errors # '' empty string # stdout has been written to this file with open(path_to_output_file,"r") as f: print(f.read()) # total 20K # drwxrwxr-x 3 felipe felipe 4,0K Nov 4 17:00 . # drwxrwxr-x 39 felipe felipe 4,0K Nov 3 18:31 .. # drwxrwxr-x 2 felipe felipe 4,0K Nov 3 19:32 .ipynb_checkpoints # -rw-rw-r-- 1 felipe felipe 7,7K Nov 4 17:00 main.ipynb 

Popen example: Redirect output and errors to same file

import subprocess from subprocess import Popen path_to_output_file = '/tmp/myoutput.txt' myoutput = open(path_to_output_file,'w+') # file 'foo bar' doesn't exist p = Popen(["ls","foo bar"], stdout=myoutput, stderr=myoutput, universal_newlines=True) output, errors = p.communicate() output # there's nothing here because we didn't send stdout to subprocess.PIPE errors # there's nothing here either # stdout and stderr have been written to this file with open(path_to_output_file,"r") as f: print(f.read()) # ls: cannot access 'foo bar': No such file or directory 

Popen example: Run command in background

By default, calls to Popen() spawn a subprocess in the background and don’t wait for it to terminate (unless you use wait() on the Popen object).

Pipe commands together

from subprocess import Popen,PIPE # this is equivalent to ls -lha | grep "foo bar" p1 = Popen(["ls","-lha"], stdout=PIPE) p2 = Popen(["grep", "foo bar"], stdin=p1.stdout, stdout=PIPE) p1.stdout.close() output = p2.communicate()[0] 

Wait for command to terminate, asynchronously

Method asyncio.create_subprocess_exec() works much the same way as Popen() but calling wait() and communicate() on the returned objects does not block the processor, so the Python interpreter can be used in other things while the external subprocess doesn’t return.

import asyncio proc = await asyncio.create_subprocess_exec( 'ls','-lha', stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE) # if proc takes very long to complete, the CPUs are free to use cycles for # other processes stdout, stderr = await proc.communicate() proc.returncode # 0 # must call decode because stdout is a bytes object stdout.decode() # total 24K # drwxrwxr-x 3 felipe felipe 4,0K Nov 4 17:52 . # drwxrwxr-x 39 felipe felipe 4,0K Nov 3 18:31 .. # drwxrwxr-x 2 felipe felipe 4,0K Nov 3 19:32 .ipynb_checkpoints # -rw-rw-r-- 1 felipe felipe 11K Nov 4 17:52 main.ipynb stderr.decode() # '' empty string 

call() vs run()

As of Python version 3.5, run() should be used instead of call() .

• run() returns a CompletedProcess object instead of the process return code.
  • A CompletedProcess object has attributes like args, returncode, etc. subprocess.CompletedProcess

  Popen vs run() and call()

  call() and run() are convenience functions and should be used for simpler cases.

  Popen() is much more powerful and handles all cases, not just simple ones.

  Источник