Limit RAM usage to python program
I’m trying to limit the RAM usage from a Python program to half so it doesn’t totally freezes when all the RAM is used, for this I’m using the following code which is not working and my laptop is still freezing:
import sys import resource def memory_limit(): rsrc = resource.RLIMIT_DATA soft, hard = resource.getrlimit(rsrc) soft /= 2 resource.setrlimit(rsrc, (soft, hard)) if __name__ == '__main__': memory_limit() # Limitates maximun memory usage to half try: main() except MemoryError: sys.stderr.write('MAXIMUM MEMORY EXCEEDED') sys.exit(-1) I’m using other functions which I call from the main function. What am I doing wrong? Thanks in advance. PD: I already searched about this and found the code I’ve put but it’s still not working.
Perhaps you want ulimit or prlimit outside the Python script. Or set up a memory limited cgroup and run the script there. I’m not sure trying to self-limit is the best idea — what happens if the code that tries to check or enforce the limit needs to allocate memory in the process?
why is all your RAM being used? Are you loading a lot of data into memory? Have you tried using generators?
@roymustang86 it’s for a university’s homework, we must generate a tree using a DFS straetgy which gets into an infinite loop because of the task statement.
3 Answers 3
I’ve done some research and found a function to get the memory from Linux systems here: Determine free RAM in Python and I modified it a bit to set the memory hard limit to half of the free memory available.
import resource import sys def memory_limit(): """Limit max memory usage to half.""" soft, hard = resource.getrlimit(resource.RLIMIT_AS) # Convert KiB to bytes, and divide in two to half resource.setrlimit(resource.RLIMIT_AS, (get_memory() * 1024 / 2, hard)) def get_memory(): with open('/proc/meminfo', 'r') as mem: free_memory = 0 for i in mem: sline = i.split() if str(sline[0]) in ('MemFree:', 'Buffers:', 'Cached:'): free_memory += int(sline[1]) return free_memory # KiB if __name__ == '__main__': memory_limit_half() try: main() except MemoryError: sys.stderr.write('\n\nERROR: Memory Exception\n') sys.exit(1) I modify the answer of @Ulises CT. Because I think to change too much original function is not so good, so I turn it to a decorator. I hope it helps.
import resource import platform import sys def memory_limit(percentage: float): """ 只在linux操作系统起作用 """ if platform.system() != "Linux": print('Only works on linux!') return soft, hard = resource.getrlimit(resource.RLIMIT_AS) resource.setrlimit(resource.RLIMIT_AS, (get_memory() * 1024 * percentage, hard)) def get_memory(): with open('/proc/meminfo', 'r') as mem: free_memory = 0 for i in mem: sline = i.split() if str(sline[0]) in ('MemFree:', 'Buffers:', 'Cached:'): free_memory += int(sline[1]) return free_memory def memory(percentage=0.8): def decorator(function): def wrapper(*args, **kwargs): memory_limit(percentage) try: return function(*args, **kwargs) except MemoryError: mem = get_memory() / 1024 /1024 print('Remain: %.2f GB' % mem) sys.stderr.write('\n\nERROR: Memory Exception\n') sys.exit(1) return wrapper return decorator @memory(percentage=0.8) def main(): print('My memory is limited to 80%.') How to set memory limit for thread or process in python?
I’m writing program in Python which would be able to run untrusted python code in some kind of sandbox. So, I need a way to limit the amount of memory that untrusted code can allocate. At now I can limit the maximum length of range(), list, dictionary and the others by overriding default python data structures in my sandboxed environment. Any ideas?
You probably just about need to do this at the OS level, not inside of Python (e.g., in Windows you might want to use a job object).
You really can’t do this without some OS-specific code, as existing OSes (Windows, Mac, Linux, etc) just vary too much. You can take a cue from subprocess , though: import sys and then use code like mswindows = (sys.platform == «win32») , and define functions inside if mswindows: . else: . .
2 Answers 2
Under Unix, you could use resource.setrlimit(resource.RLIMIT_AS, . ) to restrict «the maximum area (in bytes) of address space which may be taken by the process.»
import sys import resource soft, hard = 10**7, 10**7 # soft, hard = 10**8, 10**8 # uncommenting this allows program to finish resource.setrlimit(resource.RLIMIT_AS,(soft, hard)) memory_hog = <> try: for x in range(10000): print(x) memory_hog[str(x)]='The sky is so blue' except MemoryError as err: sys.exit('memory exceeded') # memory exceeded I don’t know how it is done under Windows. Hopefully someone else can supply that part of the solution.
I don’t know how it is done under Windows. Hopefully someone else can supply that part of the solution.
Here’s some example code to set the limit on Windows using ctypes .
import ctypes PROCESS_SET_QUOTA = 0x100 PROCESS_TERMINATE = 0x1 JobObjectExtendedLimitInformation = 9 JOB_OBJECT_LIMIT_PROCESS_MEMORY = 0x100 class IO_COUNTERS(ctypes.Structure): _fields_ = [('ReadOperationCount', ctypes.c_uint64), ('WriteOperationCount', ctypes.c_uint64), ('OtherOperationCount', ctypes.c_uint64), ('ReadTransferCount', ctypes.c_uint64), ('WriteTransferCount', ctypes.c_uint64), ('OtherTransferCount', ctypes.c_uint64)] class JOBOBJECT_BASIC_LIMIT_INFORMATION(ctypes.Structure): _fields_ = [('PerProcessUserTimeLimit', ctypes.c_int64), ('PerJobUserTimeLimit', ctypes.c_int64), ('LimitFlags', ctypes.c_uint32), ('MinimumWorkingSetSize', ctypes.c_void_p), ('MaximumWorkingSetSize', ctypes.c_void_p), ('ActiveProcessLimit', ctypes.c_uint32), ('Affinity', ctypes.c_void_p), ('PriorityClass', ctypes.c_uint32), ('SchedulingClass', ctypes.c_uint32)] class JOBOBJECT_EXTENDED_LIMIT_INFORMATION(ctypes.Structure): _fields_ = [('BasicLimitInformation', JOBOBJECT_BASIC_LIMIT_INFORMATION), ('IoInfo', IO_COUNTERS), ('ProcessMemoryLimit', ctypes.c_void_p), ('JobMemoryLimit', ctypes.c_void_p), ('PeakProcessMemoryUsed', ctypes.c_void_p), ('PeakJobMemoryUsed', ctypes.c_void_p)] # Set memory limit for process with specfied 'pid', to specified 'size' in bytes def set_limit(pid, size): job_info = JOBOBJECT_EXTENDED_LIMIT_INFORMATION() out_size = ctypes.c_uint32() job = ctypes.windll.kernel32.CreateJobObjectA(None, None) assert job != 0 success = ctypes.windll.kernel32.QueryInformationJobObject(job, JobObjectExtendedLimitInformation, ctypes.POINTER(JOBOBJECT_EXTENDED_LIMIT_INFORMATION)(job_info), ctypes.sizeof(JOBOBJECT_EXTENDED_LIMIT_INFORMATION), ctypes.POINTER(ctypes.c_uint32)(out_size)) assert success job_info.BasicLimitInformation.LimitFlags |= JOB_OBJECT_LIMIT_PROCESS_MEMORY job_info.ProcessMemoryLimit = size success = ctypes.windll.kernel32.SetInformationJobObject(job, JobObjectExtendedLimitInformation, ctypes.POINTER(JOBOBJECT_EXTENDED_LIMIT_INFORMATION)(job_info), ctypes.sizeof(JOBOBJECT_EXTENDED_LIMIT_INFORMATION)) assert success process = ctypes.windll.kernel32.OpenProcess(PROCESS_SET_QUOTA | PROCESS_TERMINATE, False, pid) assert process != 0 success = ctypes.windll.kernel32.AssignProcessToJobObject(job, process) assert success success = ctypes.windll.kernel32.CloseHandle(job) assert success success = ctypes.windll.kernel32.CloseHandle(process) assert success if __name__ == '__main__': import os five_mb = 5 * 1024 * 1024 def can_we_allocate_five_mb(): try: s = 'x' * five_mb return True except MemoryError: return False print can_we_allocate_five_mb() set_limit(os.getpid(), five_mb) print can_we_allocate_five_mb() . although there’s probably no need to create a separate job objects for each process — you should be able to associate all restricted processes with a single job.
Upper memory limit?
Is there a limit to memory for python? I’ve been using a python script to calculate the average values from a file which is a minimum of 150mb big. Depending on the size of the file I sometimes encounter a MemoryError . Can more memory be assigned to the python so I don’t encounter the error? EDIT: Code now below NOTE: The file sizes can vary greatly (up to 20GB) the minimum size of the a file is 150mb
file_A1_B1 = open("A1_B1_100000.txt", "r") file_A2_B2 = open("A2_B2_100000.txt", "r") file_A1_B2 = open("A1_B2_100000.txt", "r") file_A2_B1 = open("A2_B1_100000.txt", "r") file_write = open ("average_generations.txt", "w") mutation_average = open("mutation_average", "w") files = [file_A2_B2,file_A2_B2,file_A1_B2,file_A2_B1] for u in files: line = u.readlines() list_of_lines = [] for i in line: values = i.split('\t') list_of_lines.append(values) count = 0 for j in list_of_lines: count +=1 for k in range(0,count): list_of_lines[k].remove('\n') length = len(list_of_lines[0]) print_counter = 4 for o in range(0,length): total = 0 for p in range(0,count): number = float(list_of_lines[p][o]) total = total + number average = total/count print average if print_counter == 4: file_write.write(str(average)+'\n') print_counter = 0 print_counter +=1 file_write.write('\n') What is your script trying to do? It looks to me like you want to calculate the average value of every fourth column in each of the input files. Is that right?
I have noticed significant performance differences in regard to Memory when running the same Python application on Windows (XP) and OS X/Linux. The performance on the Windows side tends to be the worst.
5 Answers 5
(This is my third answer because I misunderstood what your code was doing in my original, and then made a small but crucial mistake in my second—hopefully three’s a charm.
Edits: Since this seems to be a popular answer, I’ve made a few modifications to improve its implementation over the years—most not too major. This is so if folks use it as template, it will provide an even better basis.
As others have pointed out, your MemoryError problem is most likely because you’re attempting to read the entire contents of huge files into memory and then, on top of that, effectively doubling the amount of memory needed by creating a list of lists of the string values from each line.
Python’s memory limits are determined by how much physical ram and virtual memory disk space your computer and operating system have available. Even if you don’t use it all up and your program «works», using it may be impractical because it takes too long.
Anyway, the most obvious way to avoid that is to process each file a single line at a time, which means you have to do the processing incrementally.
To accomplish this, a list of running totals for each of the fields is kept. When that is finished, the average value of each field can be calculated by dividing the corresponding total value by the count of total lines read. Once that is done, these averages can be printed out and some written to one of the output files. I’ve also made a conscious effort to use very descriptive variable names to try to make it understandable.
try: from itertools import izip_longest except ImportError: # Python 3 from itertools import zip_longest as izip_longest GROUP_SIZE = 4 input_file_names = ["A1_B1_100000.txt", "A2_B2_100000.txt", "A1_B2_100000.txt", "A2_B1_100000.txt"] file_write = open("average_generations.txt", 'w') mutation_average = open("mutation_average", 'w') # left in, but nothing written for file_name in input_file_names: with open(file_name, 'r') as input_file: print('processing file: <>'.format(file_name)) totals = [] for count, fields in enumerate((line.split('\t') for line in input_file), 1): totals = [sum(values) for values in izip_longest(totals, map(float, fields), fillvalue=0)] averages = [total/count for total in totals] for print_counter, average in enumerate(averages): print(' '.format(average)) if print_counter % GROUP_SIZE == 0: file_write.write(str(average)+'\n') file_write.write('\n') file_write.close() mutation_average.close()