Python multiprocessing child process

How to print() from a Child Process in Python

You can print() to stdout from child processes by setting the “flush” argument to True, or by using the ‘fork‘ start method.

In this tutorial you will discover how to print() from child processes in Python.

Problem with print() in Child Processes

Printing to standard out (stdout) with the built-in print() function may not work property from child processes.

For example, you may print output messages for the user or debug messages from a child process and they may never appear, or may only appear when the child process is terminated.

Do you have this problem?
Let me know in the comments below.

This is a very common situation and the cause is well understood and easy to workaround.

Next, let’s look at exactly why this problem occurs.

Run your loops using all CPUs, download my FREE book to learn how.

Why print() Doesn’t Work From Child Processes

When you call print() from a child process created using the ‘spawn‘ start method, the message will not appear.

This is because the messages are block buffered by default and the buffer is not flushed by default after every message. This is unlike the main process that is interactive and will flush messages after each line, e.g. line buffered.

When interactive, the stdout stream is line-buffered. Otherwise, it is block-buffered like regular text files.

— sys — System-specific parameters and functions

Instead, the buffered messages are only flushed occasionally, such as when the child process terminates and the buffer is garbage collected.

Next, let’s look at how we can make the print() function work as expected from child processes.

Confused by the multiprocessing module API?
Download my FREE PDF cheat sheet

How To Print From Child Processes

There are two techniques you can use to make print() messages appear immediately when called from a child process.

Flush Standard Output

You can flush stdout automatically with each call to print().

This can be achieved by setting the ‘flush‘ argument to True.

An alternate approach is to call the flush() function on the sys.stdout object directly.

Use Fork Start Method

There are three main techniques for starting a child process in Python.

On Windows and macOS, the ‘spawn‘ technique is used by default.

Alternatively, on another platform you may choose to use the ‘spawn‘ start method.

The problem with the print() function only occurs when using the ‘spawn‘ start method.

You can change the start method to ‘fork‘ which will cause print() to work as expected.

Note, the ‘fork‘ start method is not supported on Windows at the time of writing.

You can learn more about process start methods in the tutorial:

Now that we know how to fix the print() function from child processes, let’s look at some worked examples.

Free Python Multiprocessing Course

Download my multiprocessing API cheat sheet and as a bonus you will get FREE access to my 7-day email course.

Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.

Example of Broken Print From Child Process

Before we explore how to make the print() function behave as expected from child processes, let’s demonstrate the problem of printing from child processes.

In this example, we will start a child process to execute a custom function that prints a message, then blocks for a second. The main process will start the child process, then wait for it to terminate before printing a message itself.

Instead of the message from the child process appearing immediately after the child process is started, as we might expect, instead, the message from the child process does not appear until the child process is terminated.

First, let’s define a function to be executed in a child process.

This function will report a message then sleep for one second before exiting.

The task() function below implements this.

Источник

Multiprocessing Return Value From Process

You can return a variable from a child process using a multiprocessing.Value or a multiprocessing.Queue.

In this tutorial you will discover how to return a value from a process in Python.

Need to Return Value From Process

A process is a running instance of a computer program.

Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create new child processes in our program in order to execute code concurrently.

Python provides the ability to create and manage new processes via the multiprocessing.Process class.

In multiprocessing programming we typically need to return a value from a process.

This is challenging as there are no direct methods for returning a value from a process to another calling process.

How can we return a value from a process?

Run your loops using all CPUs, download my FREE book to learn how.

How to Return Value From a Process

There are no direct methods to return a value from a process.

Instead, it can be achieved using indirect methods.

There are a number of indirect methods to choose from. The three most convenient and widely used methods to return a value from a process are as follows:

Other approaches might include:

• Use a multiprocessing.Manager, e.g. processes interact with the same object via proxies.
• Use multiprocessing.sharedctypes, e.g. same methods that underlie multiprocessing.Value.
• Use multiprocessing.shared_memory.

Do you know of any other methods?
Let me know in the comments below.

Next, let’s take a closer look at some of these methods.

Return Variable From Process with Value

We can return a variable from a process using a multiprocessing.Value object.

These classes explicitly define data attributes designed to be shared between processes in a process-safe manner.

A process-safe manner means that only one process can read or access the variable at a time. Shared variables mean that changes made in one process are always propagated and made available to other processes.

An instance of the multiprocessing.Value can be defined in the constructor of a custom class as a shared instance variable.

The constructor of the multiprocessing.Value class requires that we specify the data type and an initial value.

The data type can be specified using ctype “type” or a typecode.

You can learn more about ctypes here:

Typecodes are familiar and easy to use, for example ‘i’ for a signed integer or ‘f’ for a single floating-point value.

You can see a handy table of type codes here:

For example, we can define a Value shared memory variable that holds a signed integer and is initialized to the value zero.

Источник