Python oserror cannot allocate memory

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python subprocess.Popen “OSError: [Errno 12] Cannot allocate memory” #55

Python subprocess.Popen “OSError: [Errno 12] Cannot allocate memory” #55

Comments

This problems happens when running multiple concurrent calls to the process of calculating embeddings in TokenLine

def TokenLine(line, lang='en', lower_case=True, romanize=False): assert lower_case, 'lower case is needed by all the models' roman = lang if romanize else 'none' tok = check_output( REM_NON_PRINT_CHAR + '|' + NORM_PUNC + lang + '|' + DESCAPE + '|' + MOSES_TOKENIZER + lang + ('| python3.6 -m jieba -d ' if lang == 'zh' else '') + ('|' + MECAB + '/bin/mecab -O wakati -b 50000 ' if lang == 'ja' else '') + '|' + ROMAN_LC + roman, input=line, encoding='UTF-8', shell=True) return tok.strip()

Here is the detailed strack trace:

 File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory INFO:default:called LaserEmbeddingHandler. ERROR:default:[Errno 12] Cannot allocate memory Traceback (most recent call last): File "/tornado_api/handlers/embeddingLaserHandler.py", line 195, in post embeddings = vector_embedding.embedding_line(model=model_laser,lang=lang,bpe_codes=QUALITY_MODEL_PATH + "/93langs.fcodes",input_text=text) File "/tornado_api/deeplearning/vector_embedding.py", line 50, in embedding_line embeddings.append(t.result()[0].tolist() ) File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/tornado_api/deeplearning/similarity_search.py", line 99, in pipeline lower_case=lower_case) File "/tornado_api/deeplearning/lib/text_processing.py", line 62, in TokenLine shell=True) File "/usr/lib/python3.6/subprocess.py", line 336, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 403, in run with Popen(*popenargs, **kwargs) as process: File "/usr/lib/python3.6/subprocess.py", line 709, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory INFO:default:called LaserEmbeddingHandler. ERROR:default:[Errno 12] Cannot allocate memory Traceback (most recent call last): File "/tornado_api/handlers/embeddingLaserHandler.py", line 195, in post embeddings = vector_embedding.embedding_line(model=model_laser,lang=lang,bpe_codes=QUALITY_MODEL_PATH + "/93langs.fcodes",input_text=text) File "/tornado_api/deeplearning/vector_embedding.py", line 50, in embedding_line embeddings.append(t.result()[0].tolist() ) File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/tornado_api/deeplearning/similarity_search.py", line 99, in pipeline lower_case=lower_case) File "/tornado_api/deeplearning/lib/text_processing.py", line 62, in TokenLine shell=True) File "/usr/lib/python3.6/subprocess.py", line 336, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 403, in run with Popen(*popenargs, **kwargs) as process: File "/usr/lib/python3.6/subprocess.py", line 709, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory 

The text was updated successfully, but these errors were encountered:

Источник

[Solved] Oserror: [Errno 12] Cannot Allocate Memory

We all know that while doing any operations on file CPU uses resources from our system. It may be memory or some I/O devices. Moreover, depending on the load of the operations, it sometimes shares heavy or graphical operations to GPUs. Now, it may be possible that the system can always fulfill the memory or I/O requirement every time. In that case, the CPU may stop its execution and wait for the fulfillment of the resources, or sometimes it may terminate the program. In this article, we will keep some light on one such issue named OsError: [errno 12] cannot allocate memory.

Oserror: [errno 12] cannot allocate memory is raised by the system when CPU won’t get enough memory resources to process pipelined operations and execute them. It may be possible that the system doesn’t have enough memory for storing intermediaries while the program is in the process. Or, it may run out of available memory due to its usage by other pipelined operations.

What is Oserror: [errno 12] cannot allocate memory?

Oserror: [errno 12] cannot allocate memory is the memory unavailability error when we run our python program, and the system does not fulfill the memory requirement. This unavailability of resources leads to the termination of our python program with the raised error Oserror: [errno 12] cannot allocate memory.

Why do I get “Oserror: [errno 12] cannot allocate memory” Error?

There is no other reason than mentioned above for the error. Any operation we perform on our computer requires some memory resources and some input/output devices. These memory resources are used to store the intermediate files or variables created while executing our program. CPU fetches these files and variables from the main memory for further execution. Now, when the system cannot save those files in the main memory due to unavailable space, it raises the given error.

[Note: The availability of Secondary memory cannot fulfill the memory requirement by default as it is much slower than main memory. This increases the throughput of the program and can be durably expensive.]

Solution to Oserror: [errno 12] cannot allocate memory

The solution to the given error is that we need to increase the system’s memory space so that it can store the intermediate files there. Now, increasing RAM memory may not be practically possible for everyone. In that case, we need to check for some other alternative.

Now, it may not be necessary to store intermediate files. We necessarily need to extend our RAM as it is not a feasible option. What we can do we create a virtual memory in our system. This virtual memory is the part of hard disk memory allocated to work as the main memory( RAM). The use of this virtual memory is that the CPU can store temporary files between the operations. The virtual memory is also called swap memory. We need to assign 64 GB of swap memory to the system so that the CPU can use it whenever there is a requirement for that. It can be a feasible option and requires no extra cost for that. The only obstacle in this process is that it is slower whenever the CPU uses this swap memory because fetching data from the main memory is faster than a hard disk.

Python subprocess.Popen “OSError: [Errno 12] Cannot allocate memory”

However, you may also get the error while using subprocess.Popen. While using the subprocess.Popen() we somehow call the fork instance, which means that we are creating a child process or subprocess, and it is going to consume the same amount of memory that is already getting consumed by the python. Now, there can be hundreds of MB of memory space required for that. Subsequently, there may be a memory shortage, and we get the error. Now, to solve the issue, there may be two options: to increase the memory space, or the second option is to write efficient code controlling the script memory.

However, to do that, we have the alternative of using vfork and posix_spawn. But maybe we don’t want to rewrite the code for the subprocess.Popen in terms of posix_spawn/vfork. In that condition, one can use a subprocess.Popen but at the beginning of the script so that the memory that will be consumed is minimum due to fewer resources, and we got the shell script. Now, we can use the shell script for our parallel processing with the lighter tasks such as free/ps/sleep, whatever else is in a loop.

Conclusion

So, today in this article, we learned about Oserror: [errno 12] cannot allocate memory Error. We have seen what the possible reasons for the given error are. Then we have discussed the possible solutions for that. After following the above explanation, you will be able to solve the error.

I hope this article has helped you. Thank you.

Источник