Python temp file in memory

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Create temporary files and temporary dirs in memory-based filesystems on Linux.

License

mbello/memory-tempfile

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Sign In Required

Please sign in to use Codespaces.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching Xcode

If nothing happens, download Xcode and try again.

Launching Visual Studio Code

Your codespace will open once ready.

There was a problem preparing your codespace, please try again.

Latest commit

Git stats

Files

Failed to load latest commit information.

README.md

Often there is a need to temporarily save a file to ‘disk’ for the consumption of external tools. Or maybe you can pipe some input info to an external tool but has no way of forcing such external tool to pipe its output straight to your software: it wants to write a file to disk. Disk operations are slow and if repeated too often can shorten the lifespan of underlying media.

On Linux, most distributions offer a /tmp directory BUT it is usually on physical media. However, modern distributions often offer at least two places where one can safely create temporary files in RAM: /dev/run/, /run/shm and /dev/shm

/dev/run/ is ideal for your temporary files. It is writable and readable only by your user. /dev/shm is usually world-readable and world-writable (just like /tmp), it is often used for IPC (inter process communication) and can serve well as a temporary RAM-based tempdir.

This module is very simple and tries not to reinvent the wheel. It will check /tmp to see if it in a ramdisk or not. And it will also check if you have other options where to place your temporary files/dirs on a memory-based file system like tmpfs or ramfs.

Once you know the suitable path on a memory-based file system where you can have your files, you are well served by python’s builtin modules and external packages like pathlib or pyfilesystem2 to move on to do your things.

This module searches for paths hosted on filesystems of type belonging to MEM_BASED_FS=[‘tmpfs’, ‘ramfs’] Paths in SUITABLE_PATHS=[‘/tmp’, ‘/run/user/’, ‘/run/shm’, ‘/dev/shm’] are searched and the first path found that exists and is stored on a filesystem whose type belongs to MEM_BASED_FS will be used as the tempdir. If no suitable path is found, then if fallback = True, we will fallback to default tempdir (as determined by tempfile stdlib). If fallback is a path, then we will default to it. If fallback is false, a RunTimeError exception is raised.

The MemoryTempfile constructor has arguments that let you change how the algorithm works. You can change the order of paths (with ‘preferred_paths’), add new paths to the search (with ‘preferred_paths’ and/or with ‘additional_paths’) and you can exclude certain paths from SUITABLE_PATHS (with removed_paths). All paths containing the string will have it replaced by the user id. You can change the filesystem types you accept (with filesystem_types) and specify whether or not to fallback to a vanilla tempdir as a last resort.

Then, all methods available from tempfile stdlib are available through MemoryTempfile.

Here is the list of accepted parameters:

• preferred_paths: list = None
• remove_paths: list or bool = None
• additional_paths: list = None
• filesystem_types: list = None
• fallback: str or bool = None

The path list that will be searched from first to last item will be constructed using the algorithm:

paths = preferred_paths + (SUITABLE_PATHS - remove_paths) + additional_paths 

If remove_paths is boolean ‘true’, SUITABLE_PATHS will be eliminated, this is a way for you to take complete control of the path list that will be used without relying on this package’s hardcoded constants.

The only other hardcoded constant MEM_BASED_FS=[‘tmpfs’, ‘ramfs’] will not be used at all if you pass your own ‘filesystem_types’ argument. By the way, if you wish to add other file system types, you must match what Linux uses in /proc/self/mountinfo (at the 9th column).

• Python 3
• Works only on Linux
• Compatible with chroot and/or namespaces, needs access to /proc/self/mountinfo

from memory_tempfile import MemoryTempfile tempfile = MemoryTempfile() with tempfile.TemporaryFile() as tf: # as usual. 

# We now do not want to use /dev/shm or /run/shm and no ramfs paths # If /run/user/ is available, we prefer it to /tmp # And we want to try /var/run as a last resort # If all fails, fallback to platform's tmp dir from memory_tempfile import MemoryTempfile import memory_tempfile # By the way, all paths with string will have it replaced with the user id tempfile = MemoryTempfile(preferred_paths=['/run/user/'], remove_paths=['/dev/shm', '/run/shm'], additional_paths=['/var/run'], filesystem_types=['tmpfs'], fallback=True) if tempfile.found_mem_tempdir(): print('We could use any of the following paths: <>'.format(tempfile.get_usable_mem_tempdir_paths())) print('And we are using now: <>'.format(tempfile.gettempdir())) with tempfile.NamedTemporaryFile() as ntf: # use it as usual. pass 

About

Create temporary files and temporary dirs in memory-based filesystems on Linux.

Источник

python tempfile in memory

Tempfile is a Python module used in a situation, where we need to read multiple files, change or access the data in the file, and gives output files based on the result of processed data. Each of the output files produced during the program execution was no longer needed after the program was done.

How do I create a Tempfile in Python?

1. TemporaryFile() This function creates a temporary file in the temp directory and returns a file object, similar to built-in open() function. .
2. NamedTemporaryFile() This function is similar to TemporaryFile() function. .
3. TemporaryDirectory() This function creates a temporary directory. .
4. mkstemp() .
5. mkdtemp() .
6. gettempdir()

How do I open a Tempfile in Python?

1. # Import os module. import os. # Define the name of the temporary file. filename = ‘temp.txt’ .
2. # Import tempfile module. import tempfile. # Declare object to open temporary file for writing. tmp = tempfile.TemporaryFile(‘w+t’) .
3. # Import tempfile module. import tempfile. # Import os module. import os.

How do I get rid of NamedTemporaryFile?

You just want a file object (pointing to an empty file) which has a filename associated to it and hence you cannot use a StringIO object: from tempfile import NamedTemporaryFile f = NamedTemporaryFile() # use f .. Once f is garbage collected, or closed explicitly, the file will automatically be removed from disk.

Solve

How do you write a matrix without NumPy in Python?How do you solve a linear equation in python without NumPy?How do you find eigenvalues in python wit.

Latest

It’s their latest long-term supported release and includes user feedback capabilities, widgets, the Discover software manager, System Settings improve.

Postgresql

Second Method – Install PostgreSQL on CentOS 7 using the PostgreSQL repositoryAccess Your Server. . Download PostgreSQL Using Wget. . Install Posg.

Latest news, practical advice, detailed reviews and guides. We have everything about the Linux operating system

Источник

memory-tempfile 2.2.3

Helper functions to identify and use paths on the OS (Linux-only for now) where RAM-based tempfiles can be created.

Навигация

Ссылки проекта

Статистика

Метаданные

Лицензия: MIT License (MIT)

Метки tempfile in RAM, tempfile in memory, file in memory, tempfs, ramdisk, /dev/shm, /run/user

Требует: Python >=3.6,

Сопровождающие

Классификаторы

Описание проекта

Overview

Often there is a need to temporarily save a file to ‘disk’ for the consumption of external tools. Or maybe you can pipe some input info to an external tool but has no way of forcing such external tool to pipe its output straight to your software: it wants to write a file to disk. Disk operations are slow and if repeated too often can shorten the lifespan of underlying media.

On Linux, most distributions offer a /tmp directory BUT it is usually on physical media. However, modern distributions often offer at east two places where one can safely create temporary files in RAM: /dev/run/, /run/shm and /dev/shm

/dev/run/ is ideal for your temporary files. It is writable and readable only by your user. /dev/shm is usually world-readable and world-writable (just like /tmp), it is often used for IPC (inter process communication) and can serve well as a temporary RAM-based tempdir.

This module is very simple and tries not to reinvent the wheel. It will check /tmp to see if it in a ramdisk or not. And it will also check if you have other options where to place your temporary files/dirs on a memory-based file system like tmpfs or ramfs.

Once you know the suitable path on a memory-based file system where you can have your files, you are well served by python’s builtin modules and external packages like pathlib or pyfilesystem2 to move on to do your things.

This module searches for paths hosted on filesystems of type belonging to MEM_BASED_FS=[‘tmpfs’, ‘ramfs’] Paths in SUITABLE_PATHS=[‘/tmp’, ‘/run/user/’, ‘/run/shm’, ‘/dev/shm’] are searched and the first path found that exists and is stored on a filesystem whose type belongs to MEM_BASED_FS will be used as the tempdir. If no suitable path is found, then if fallback = True, we will fallback to default tempdir (as determined by tempfile stdlib). If fallback is a path, then we will default to it. If fallback is false, a RunTimeError exception is raised.

The MemoryTempfile constructor has arguments that let you change how the algorithm works. You can change the order of paths (with ‘preferred_paths’), add new paths to the search (with ‘preferred_paths’ and/or with ‘additional_paths’) and you can exclude certain paths from SUITABLE_PATHS (with removed_paths). All paths containing the string will have it replaced by the user id. You can change the filesystem types you accept (with filesystem_types) and specify whether or not to fallback to a vanilla tempdir as a last resort.

Then, all methods available from tempfile stdlib are available through MemoryTempfile.

The constructor:

Here is the list of accepted parameters:

• preferred_paths: list = None
• remove_paths: list or bool = None
• additional_paths: list = None
• filesystem_types: list = None
• fallback: str or bool = None

The path list that will be searched from first to last item will be constructed using the algorith:

paths = preferred_paths + (SUITABLE_PATHS - remove_paths) + additional_paths 

If remove_paths is boolean ‘true’, SUITABLE_PATHS will be eliminated, this is a way for you to take complete control of the path list that will be used without relying on this package’s hardcoded constants.

The only other hardcoded constant MEM_BASED_FS=[‘tmpfs’, ‘ramfs’] will not be used at all if you pass your own ‘filesystem_types’ argument. By the way, if you wish to add other file system types, you must match what Linux uses in /proc/self/mountinfo (at the 9th column).

Requirements

• Python 3
• Works only on Linux
• Compatible with chroot and/or namespaces, needs access to /proc/self/mountinfo

Usage

Example 1:

from memory_tempfile import MemoryTempfile tempfile = MemoryTempfile() with tempfile.TemporaryFile() as tf: # as usual. 

Example 2:

# We now do not want to use /dev/shm or /run/shm and no ramfs paths # If /run/user/ is available, we prefer it to /tmp # And we want to try /var/run as a last resort # If all fails, fallback to platform's tmp dir from memory_tempfile import MemoryTempfile import memory_tempfile # By the way, all paths with string will have it replaced with the user id tempfile = MemoryTempfile(preferred_paths=['/run/user/'], remove_paths=['/dev/shm', '/run/shm'], additional_paths=['/var/run'], filesystem_types=['tmpfs'], fallback=True) if tempfile.found_mem_tempdir(): print('We could use any of the followig paths: <>'.format(tempfile.get_usable_mem_tempdir_paths())) print('And we are using now: <>'.format(tempfile.gettempdir())) with tempfile.NamedTemporaryFile() as ntf: # use it as usual. pass 

Источник