Error in atexit run exitfuncs python

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exiting Python REPL prompt with user without home directory throws error in atexit._run_exitfuncs #64090

Exiting Python REPL prompt with user without home directory throws error in atexit._run_exitfuncs #64090

3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

assignee = None closed_at = Date 2018-08-06.09:42:26.458> created_at = Date 2013-12-05.04:22:10.425> labels = ['3.7', '3.8', 'type-bug', 'library'] title = 'Exiting Python REPL prompt with user without home directory throws error in atexit._run_exitfuncs' updated_at = Date 2018-08-06.09:42:26.454> user = 'https://github.com/vajrasky'

activity = Date 2018-08-06.09:42:26.454> actor = 'methane' assignee = 'none' closed = True closed_date = Date 2018-08-06.09:42:26.458> closer = 'methane' components = ['Library (Lib)'] creation = Date 2013-12-05.04:22:10.425> creator = 'vajrasky' dependencies = [] files = [] hgrepos = [] issue_num = 19891 keywords = ['patch'] message_count = 14.0 messages = ['205273', '205295', '205298', '205300', '205304', '205371', '205372', '205373', '218490', '305161', '322435', '323184', '323186', '323187'] nosy_count = 8.0 nosy_names = ['pitrou', 'r.david.murray', 'methane', 'ejo', 'vajrasky', 'Anthony Sottile', 'dj3ei', 'miss-islington'] pr_nums = ['8483', '8684', '8685'] priority = 'low' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue19891' versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

The text was updated successfully, but these errors were encountered:

Источник

How to fix python multiprocessing atexit error «error in atexit._run_exitfuncs»?

The «Error in atexit._run_exitfuncs» error message appears when using the Python’s multiprocessing module. This error occurs when the multiprocessing module encounters a problem when trying to clean up after the processes it started. The problem arises due to the use of the «atexit» module in combination with the «multiprocessing» module. The «atexit» module is used to run functions when the Python process is about to exit, but it doesn’t work well with the «multiprocessing» module, causing the error message to appear.

Method 1: Use if name == ‘main‘

When using Python Multiprocessing, you may encounter the atexit error «Error in atexit._run_exitfuncs». This error occurs when the atexit functions are not cleaned up properly. To fix this error, you can use the «if name == ‘main‘» statement in your code.

Here is an example code that demonstrates how to fix the atexit error using «if name == ‘main‘»:

import multiprocessing def worker(): print('Worker') if __name__ == '__main__': p = multiprocessing.Process(target=worker) p.start() p.join()

In this example, we define a worker function that prints «Worker». We then create a new Process object and start it. Finally, we join the process to wait for it to complete.

By using «if name == ‘main‘», we ensure that the worker function is only executed in the main process. This prevents the atexit error from occurring.

Here is another example that demonstrates how to use «if name == ‘main‘» with a Pool of processes:

import multiprocessing def worker(x): return x * x if __name__ == '__main__': with multiprocessing.Pool(processes=4) as pool: result = pool.map(worker, range(10)) print(result)

In this example, we define a worker function that takes a number and returns its square. We then create a Pool of 4 processes and use the map method to apply the worker function to a range of numbers from 0 to 9.

Again, by using «if name == ‘main‘», we ensure that the worker function is only executed in the main process. This prevents the atexit error from occurring.

In summary, to fix the Python Multiprocessing atexit error «Error in atexit._run_exitfuncs», you can use «if name == ‘main‘» in your code to ensure that the atexit functions are cleaned up properly.

Method 2: Use multiprocessing.set_start_method()

If you are facing the «Error in atexit._run_exitfuncs» error while using Python multiprocessing, you can fix it by using the multiprocessing.set_start_method() method. This method allows you to set the start method for the multiprocessing module. The start method determines how the child processes are started.

Here is an example code that shows how to use multiprocessing.set_start_method() to fix the «Error in atexit._run_exitfuncs» error:

import multiprocessing def worker(): print('Worker') if __name__ == '__main__': multiprocessing.set_start_method('spawn') # Set the start method to 'spawn' p = multiprocessing.Process(target=worker) p.start() p.join()

In this example, we set the start method to ‘spawn’ using multiprocessing.set_start_method(‘spawn’) . The ‘spawn’ start method creates a new process by calling the os.fork() method. This method creates a new process by duplicating the calling process. The new process is an exact copy of the calling process, except for the process ID and the parent process ID.

Once the start method is set, we create a new process using the multiprocessing.Process() method and start it using the p.start() method. We then wait for the process to finish using the p.join() method.

By using multiprocessing.set_start_method() and setting the start method to ‘spawn’, we can avoid the «Error in atexit._run_exitfuncs» error and run our multiprocessing code without any issues.

Note: The multiprocessing.set_start_method() method should be called at the beginning of your code, before any other multiprocessing code is executed.

I hope this helps you to fix the «Error in atexit._run_exitfuncs» error while using Python multiprocessing.

Method 3: Use a Manager instead of a Pool

To fix the «Error in atexit._run_exitfuncs» error in Python Multiprocessing, you can use a Manager instead of a Pool. Here are the steps to do it:

import multiprocessing manager = multiprocessing.Manager()

def worker(shared_list): # do some work on the shared list shared_list.append(1)

processes = [] for i in range(5): p = multiprocessing.Process(target=worker, args=(shared_list,)) processes.append(p)

for p in processes: p.start()

This code example shows how to use a Manager instead of a Pool to fix the «Error in atexit._run_exitfuncs» error in Python Multiprocessing. By using a Manager, you can create shared objects that can be accessed by multiple processes without causing the atexit error.

Method 4: Manually Terminate the Processes

To fix the Python Multiprocessing atexit Error «Error in atexit._run_exitfuncs» using the «Manually Terminate the Processes» method, follow these steps:

import multiprocessing import time

def worker(): print("Worker started") time.sleep(5) print("Worker finished")

p = multiprocessing.Process(target=worker)

Источник

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incompatibility with concurrent.futures #62

incompatibility with concurrent.futures #62

Comments

using tblib.pickling_support.install() causes an exception when using concurrent.futures.ProcessPoolExecutor . Here is a minimal example:

from concurrent.futures import ProcessPoolExecutor import tblib.pickling_support tblib.pickling_support.install() def foo(x): if x > 2: raise RuntimeError('bar') if __name__ == "__main__": with ProcessPoolExecutor(max_workers=2) as executor: try: list( executor.map(foo, range(50)) ) except Exception as ex: print('exception: ', ex) print(' --- done --- ') 

I get an exception when the atexit functions run, this is the output:

exception: bar --- done --- Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/home/nick/lego-root/lib/python3.7/concurrent/futures/process.py", line 102, in _python_exit thread_wakeup.wakeup() File "/home/nick/lego-root/lib/python3.7/concurrent/futures/process.py", line 90, in wakeup self._writer.send_bytes(b"") File "/home/nick/lego-root/lib/python3.7/multiprocessing/connection.py", line 183, in send_bytes self._check_closed() File "/home/nick/lego-root/lib/python3.7/multiprocessing/connection.py", line 136, in _check_closed raise OSError("handle is closed") OSError: handle is closed Process finished with exit code 0 

commenting out tblib.pickling_support.install() solves the problem, I then get the expected output with no exceptions:

exception: bar --- done --- Process finished with exit code 0 

The text was updated successfully, but these errors were encountered:

Hey thanks for the follow up.

So it reproduces on Python 3.8.2 (Ubuntu 20.04). I see you mentioned running on Fedora, I wonder if that could influence it. Below is a Docker example that reproduces on Ubuntu. I’ll try Fedora also.

See this docker file Dockerfile.txt (you’ll have to rename from ‘Dockerfile.txt’ to ‘Dockerfile’, github won’t me let upload otherwise)

to build it: docker build -t map_bug .

to run it, without installing pickling support, to show the expected output: docker run -it map_bug python3 test.py

to run it, with picking support, to show the error: docker run -it map_bug python3 test.py bug

install pickling support exception: bar --- done --- Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/usr/lib/python3.8/concurrent/futures/process.py", line 102, in _python_exit thread_wakeup.wakeup() File "/usr/lib/python3.8/concurrent/futures/process.py", line 90, in wakeup self._writer.send_bytes(b"") File "/usr/lib/python3.8/multiprocessing/connection.py", line 183, in send_bytes self._check_closed() File "/usr/lib/python3.8/multiprocessing/connection.py", line 136, in _check_closed raise OSError("handle is closed") OSError: handle is closed 

Well I’ve looked at it, it does reproduces and I have no clue what that _python_exit code is for.

Either way, this seems like a benign problem. Contrast that to what happens if you register signal handlers (it deadlocks — https://bugs.python.org/issue29759).

I did a little bit of digging after noticing that this issue disappears on Python 3.9. So I had a look at what was different between concurrent.futures.process between 3.7 and 3.9. Turns out that backporting the _ThreadWakeup class from Python 3.9 was enough to fix the issue for me:

import multiprocessing as mp import concurrent.futures.process as pe from concurrent.futures import ProcessPoolExecutor import tblib.pickling_support class _ThreadWakeup: def __init__(self): self._closed = False self._reader, self._writer = mp.Pipe(duplex=False) def close(self): if not self._closed: self._closed = True self._writer.close() self._reader.close() def wakeup(self): if not self._closed: self._writer.send_bytes(b"") def clear(self): if not self._closed: while self._reader.poll(): self._reader.recv_bytes() pe._ThreadWakeup = _ThreadWakeup tblib.pickling_support.install() def foo(x): if x > 2: raise RuntimeError("bar") if __name__ == "__main__": with ProcessPoolExecutor(max_workers=2) as executor: try: list(executor.map(foo, range(50))) except Exception as ex: print("exception: ", ex) print(" --- done --- ")

Here’s the issue with the pull request that fixed the issue: python/cpython#83285. Hope this helps anyone facing the same issue.

Источник