Распараллелить цикл на python

Параллельный цикл for в Python

1. Используйте модуль multiprocessing для распараллеливания цикла for в Python
2. Используйте модуль joblib для распараллеливания цикла for в Python
3. Используйте модуль asyncio для распараллеливания цикла for в Python

Распараллеливание цикла означает параллельное распределение всех процессов с использованием нескольких ядер. Когда у нас много заданий, каждое вычисление не дожидается завершения предыдущего в параллельной обработке. Вместо этого для завершения используется другой процессор.

В этой статье мы распараллелим цикл for в Python.

Используйте модуль multiprocessing для распараллеливания цикла for в Python

Чтобы распараллелить цикл, мы можем использовать пакет multiprocessing в Python, поскольку он поддерживает создание дочернего процесса по запросу другого текущего процесса.

Модуль multiprocessing можно использовать вместо цикла for для выполнения операций над каждым элементом итерации. Можно использовать объект multiprocessing.pool() , поскольку использование нескольких потоков в Python не даст лучших результатов из-за глобальной блокировки интерпретатора.

import multiprocessing  def sumall(value):  return sum(range(1, value + 1))  pool_obj = multiprocessing.Pool()  answer = pool_obj.map(sumall,range(0,5)) print(answer) 

Используйте модуль joblib для распараллеливания цикла for в Python

Модуль joblib использует многопроцессорность для запуска нескольких ядер ЦП для выполнения распараллеливания цикла for . Он предоставляет легкий конвейер, который запоминает шаблон для простых и понятных параллельных вычислений.

Чтобы выполнить параллельную обработку, мы должны установить количество заданий, а количество заданий ограничено количеством ядер в ЦП или количеством доступных или простаивающих в данный момент.

Функция delayed() позволяет нам указать Python, чтобы через некоторое время был вызван конкретный упомянутый метод.

Функция Parallel() создает параллельный экземпляр с указанными ядрами (в данном случае 2).

Источник

Parallel Nested For-Loops in Python

You can convert nested for-loops to execute concurrently or in parallel in Python using thread pools or process pools, depending on the types of tasks that are being executed.

In this tutorial, you will discover how to change a nested for-loop to be concurrent or parallel in Python with a suite of worked examples.

This tutorial was triggered by questions and discussions with Robert L. Thanks again. If you have questions or want to chat through a technical issue in Python concurrency, message me any time.

Nested For-Loop in Python

A nested for-loop is a loop within a loop.

For example, we may need to loop over a number of tasks, and each task has subtasks.

Each task requires effort, e.g. I/O (read or write data) or CPU compute (calculate something), and each subtask also requires some effort.

Importantly, the number and nature of subtasks for each task are a function of the task and may not be known beforehand. The tasks must be computed in order to determine and then issue the subtasks.

Often the tasks are independent of one another, and each subtask is also independent of one another.

Importantly, subtasks are dependent upon tasks. As such, we cannot pre-define a set of function calls prior and issue them all in batch. Instead, we need to navigate the tree or hierarchy of tasks and subtasks.

This raises the question, can we perform the tasks and subtasks concurrently or in parallel?

If so, the concurrent execution of tasks and subtasks can offer a dramatic speed-up.

How can we execute a nested for-loop in parallel in Python?

Run your loops using all CPUs, download my FREE book to learn how.

How to Execute a Parallel Nested For-Loop

A nested for-loop can be converted to run in parallel.

More specifically, we can make it concurrent if the tasks are independent and if the subtasks are independent.

I/O bound tasks like reading and writing from files and sockets can be executed at the same time concurrently using threads. CPU-bound tasks like parsing a document in memory or calculating something can be performed in parallel using process-based concurrency.

You can learn more about the difference between when to use threads vs processes in the tutorial:

Therefore, if we have I/O bound tasks or subtasks, we can use a thread pool to make the loops concurrent via the concurrent.futures.ThreadPoolExecutor class or the multiprocessing.pool.ThreadPool class.

Concurrent for-loops (not nested) are straightforward, for example:

More work is required for concurrent nested for-loops.

If we have CPU-bound tasks or subtasks, we can use a process pool to make loops parallel via the concurrent.futures.ProcessPoolExecutor class or the multiprocessing.Pool class.

Parallel for-loops (not bested) are straightforward, for example:

More work is required for parallel nested for-loops.

There are two main approaches we can use to make a nested for-loop concurrent.

1. Create a pool of workers at each level in the hierarchy.
2. Share a pool of workers across the hierarchy.

Let’s take a closer look at each approach.

Approach 1: One Pool of Workers Per Level

Each level in a nested for-loop can have its own pool of workers.

That is, each task runs, does its work, creates a pool of workers, and issues the subtasks to the pool. If there is another level of subsubtasks, each of these would create its own pool of workers and issue its own tasks.

This is suited to nested for-loops that have a large number of tasks to execute at a given level.

The downside is the redundancy of having many pools of workers competing with each other. This is not a problem with thread pools, as we may have many thousands of concurrent threads, but process pools are typically limited to one worker per CPU core.

As such, some tuning of the number of workers per pool may be required.

Another downside of this approach is when using process pools, child processes are typically daemonic and are unable to create their own child processes. This means if tasks executing in a child process tries to create their own pool of workers it will fail with an error.

As such, this approach may only be viable when working with thread pools, and even then, perhaps only in a nested loop with tasks and subtasks with many subtasks per task.

Approach 2: Shared Pool of Workers Across Levels

Another approach is to create one pool of workers and issue all tasks, subtasks, and subsubtasks to this pool.

When using thread pools in one process, the pool can be shared with tasks and subtasks as a shared global variable, allowing tasks to be issued directly.

When using process pools, things are more tricky. A centralized pool of workers can be created in a server process using a multiprocessing.Manager and the proxy objects for using the centralized server can be shared among all tasks and subtasks.

An alternate design might be to use a shared queue. All tasks and subtasks may be placed onto the queue and a single consumer of tasks can retrieve items from the queue and issue them to the pool of workers.

This is functionally the same, although it separates the concern of issuing tasks from how they are executed, potentially allowing the consumer to decide to use a thread pool or process pool based on the types of tasks issued to the queue.

Now that we have considered some designs on how to convert a nested for-loop to run concurrently, let’s look at some worked examples.

Confused by the multiprocessing module API?
Download my FREE PDF cheat sheet

Example of a Nested For-Loop in Python (slow version)

Firstly let’s develop a nested for-loop that does not run concurrently.

In this example, we will design a loop with 3 levels.

That is, tasks that generate subtasks, that themselves generate subsubtasks.

Each task will simulate effort with a sleep of one second and report a message.

The complete example of a nested for-loop is listed below.

Источник

How to parallelize for loops in Python and Work with Shared Dictionaries

Below is the Sequential way of making 500 Post requests to the API.

On google colab, this piece of code took around 100 seconds to complete.

Multiprocessing

We will be making 500 requests to the above-mentioned API. We will be using the concurrent package. Below is the general format to use multiprocessing for a for loop

concurrent.futures.ProcessPoolExecutor allows you to set the maximum number of proccesses to be used by setting a value for the max_workers parameter

The function parameter of executor.submit() should not have any brackets since we do not want to invoke the function.

submit() returns a future object. To get the actual returned value of the function, we will use the result method of the future object. The returned values can be in any order.

Below is how we would use multiprocessing to make the 500 requests

In google colab, this took around 50 seconds. If your piece of code executes in less than a second, there is probably an error somewhere. To check the error during multiprocessing, simply print the result of one of the future values.

Multithreading

The format for multithreading is pretty similar to multiprocessing. The only difference is instead of concurrent.furtures.ProcessPollExecutor(), we will use concurrent.futures.ThreadPoolExecutor()

Above is the piece of code which makes 500 requests using Multithreading. In google colab it took around 15 seconds. To check the error during multithreading, simply print the result of one of the future values.

concurrent.futures.ThreadPoolExecutor allows you to set the maximum number of threads to be used by setting a value for the max_workers parameter

Sharing Dictionary using Manager

We are going to use a dictionary to store the return values of the function. The key will be the request number and the value will be the response status. Since we are making 500 requests, there will be 500 key-value pairs in our dictionary.

The parameter d is the dictionary that will have to be shared.

To share a dictionary we will use Manager().

Since the function is not returning anything, we do not need to store the future objects returned by the submit function.

The dictionary created using Manager() is thread-safe and has a lock. This might make your program slow if there are a bunch of inserts.

‘Sharing’ Dictionary by combining Dictionaries at the end

We will need to update our function to return a key-value pair instead of directly storing it inside our shared dictionary

We will use a dictionary’s update function to store all the returned values. Below is the piece of code which invokes the function multiple times

As you can see, this time we are storing the future objects since we will need them to store the returned values. We will essentially iterate over the list of future objects and update our dictionary

Comparing Performance of MultiProcessing, MultiThreading

We won’t be comparing with sequential code since sequential code is pretty slow and will take a long time to make a large number of requests.

We will be making 5 , 10 , 50, 100, 500, 1000, 2500, 5000 and 10000 requests.

To make it easier, we will create two functions. One of the functions would use multi-processing while the other would use multi-threading.

We will use the above times to plot a line graph to compare the performance

As you can see from the above graph, Multithreading performs better but this is because we are currently executing I/O functions(making requests to an API). When executing CPU-heavy functions (a lot of calculations), multiprocessing will be faster.

You can refer to the below article to see comparisons b/w Multiprocessing and Multithreading when dealing with CPU-heavy functions. The article also compares the performance with different values for max_workers

Источник