What is the multiprocessing.dummy Module
The multiprocessing.dummy module module provides a wrapper for the multiprocessing module, except implemented using thread-based concurrency.
It provides a drop-in replacement for multiprocessing, allowing a program that uses the multiprocessing API to switch to threads with a single change to import statements.
In this tutorial you will discover the multiprocessing.dummy module in Python.
What is multiprocessing.dummy?
The multiprocessing.dummy module provides the multiprocessing module API except implemented using threads.
The API documentation reports that dummy is a wrapper around the threading module that implements the multiprocessing module API.
multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module.
— multiprocessing — Process-based parallelism
Next, let’s consider why the module exists.
Run your loops using all CPUs, download my FREE book to learn how.
Why does multiprocessing.dummy exist?
The multiprocessing.dummy module is not explicitly mentioned in the PEP 371 that describes the introduction of the multiprocessing module.
Nevertheless, the multiprocessing.dummy does appear in Python 2.6 which saw the introduction of the multiprocessing module and the dummy module is described in the API documentation in this version.
multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module
— multiprocessing — Process-based parallelism
It is stated in the PEP that the multiprocessing module seeks to provide the same API as the threading module, except implemented using processes instead of threads.
The processing package mimics the standard library threading module functionality to provide a process-based approach to threaded programming allowing end-users to dispatch multiple tasks that effectively side-step the global interpreter lock.
— PEP 371 – Addition of the multiprocessing package to the standard library
The multiprocessing module introduces additional capabilities that do not exist in the threading API, such as:
The best guess is that the multiprocessing.dummy module is provided to be a drop-in replacement for the multiprocessing.
This would allow a program to be implemented using the multiprocessing module API with process-based concurrency and switch to thread-based concurrency with a single line change, e.g. switching the import of multiprocessing to multiprocessing.dummy.
Next, let’s explore what classes are provided in the dummy module.
Confused by the ThreadPool class API?
Download my FREE PDF cheat sheet
What Classes does multiprocessing.dummy Provide?
The multiprocessing.dummy module provides many classes and module functions that match the multiprocessing module.
For example, we can report the contents of the module directly via the __all__ list.
Running the example report the contents of the module, as follows:
We can see 3 module methods and the rest are multiprocessing class names and mostly concurrency primitives.
Reviewing the source code for the dummy module, we can see that most of the API involves importing classes from the threading module, such as queues and concurrency primitives such as locks and semaphores.
There a few classes defined in the module, they are:
Next, let’s take a closer look at the multiprocessing.dummy.Pool class.
Free Python ThreadPool Course
Download my ThreadPool API cheat sheet and as a bonus you will get FREE access to my 7-day email course.
Discover how to use the ThreadPool including how to configure the number of worker threads and how to execute tasks asynchronously
multiprocessing.dummy.Pool vs multiprocessing.pool.ThreadPool
Perhaps the most significant class in multiprocessing.dummy is Pool.
This is because it returns a ThreadPool class instance that is a wrapper for Pool, except that it uses threads instead of processes for concurrency.
In particular, the Pool function provided by multiprocessing.dummy returns an instance of ThreadPool, which is a subclass of Pool that supports all the same method calls but uses a pool of worker threads rather than worker processes.
— multiprocessing — Process-based parallelism
It is the only class that is part of multiprocessing.dummy (technically multiprocessing.pool) that warrants mention in the multiprocessing API documentation.
Note A ThreadPool shares the same interface as Pool, which is designed around a pool of processes and predates the introduction of the concurrent.futures module. As such, it inherits some operations that don’t make sense for a pool backed by threads, and it has its own type for representing the status of asynchronous jobs, AsyncResult, that is not understood by any other libraries.
— multiprocessing — Process-based parallelism
ThreadPool is defined in the multiprocessing.pool module, although probably should sit in the multiprocessing.dummy module.
Importantly, it imports the Process class from the dummy module for worker threads, which is an alias for the DummyProcess class that extends threading.Thread.
We can see this in the source code for the dummy module where Pool is defined.