Write thread-safe to file in python
How can I write data to a file in Python thread-safe? I want to safe some variables to a file for every request and every hour I want to do some grouping and write it to mysql. In Java I now put it in an Array which is cached and this is written to a file when the array is full. How can I do this in Python? There are many concurrent requests so it has to be thread-safe. EDIT: We ended up using the logging module which works fine.
Why not just write the rows to MySQL as you go and let its concurrency handle that, then aggregate that and do the needful?
It’s possible with locking and queuing and all sorts, but it’s easier to just throw it at the server, aggregated it there, or in a cronjob or similar, aggregate it in the server and insert the aggregates, or pull the data back, aggregate and insert from the client side.
@JonClements Writing to mysql is what I want to prevent. This takes to much resources since it gets around 300 requests per second.
4 Answers 4
We used the logging module:
import logging logpath = "/tmp/log.log" logger = logging.getLogger('log') logger.setLevel(logging.INFO) ch = logging.FileHandler(logpath) ch.setFormatter(logging.Formatter('%(message)s')) logger.addHandler(ch) def application(env, start_response): logger.info("%s %s".format("hello","world!")) start_response('200 OK', [('Content-Type', 'text/html')]) return ["Hello!"] +1 for abusing logging to thread-safely write lines to a file, I do this sometimes. There is some required boilerplate and understanding of how logging works, but not reinventing the wheel is nice.
this is a good use of the logging module. logging is thread-safe and has a FileHandler built in. (docs.python.org/3/library/logging.html#thread-safety)
Look at the Queue class, it is thread safe.
from Queue import Queue writeQueue = Queue() writeQueue.put(repr(some_object)) Then to dump it to a file,
outFile = open(path,'w') while writeQueue.qsize(): outFile.write(writeQueue.get()) outFile.flush() outFile.close() Queue will accept any python object, so if you’re trying to do something other than print to a file, just store the objects from the worker threads via Queue.put .
If you need to split the commits across multiple invocations of the script, you’ll need a way to cache partially built commits to disk. To avoid multiple copies trying to write to the file at the same time, use the lockfile module, available via pip. I usually use json to encode data for these purposes, it supports serializing strings, unicode, lists, numbers, and dicts, and is safer than pickle.
with lockfile.LockFile('/path/to/file.sql'): fin=open('/path/to/file') data=json.loads(fin.read()) data.append(newdata) fin.close() fout=open('/path/to/file','w') fout.write(json.dumps(data)) fout.close() Note that depending on OS features, the time taken to lock and unlock the file as well as rewrite it for every request may be more than you expect. If possible, try to just append to the file, as that will be faster. Also, you may want to use a client/server model, where each ‘request’ launches a worker script which connects to a server process and forwards the data on via a network socket. This sidesteps the need for lockfiles; depending on how much data you’re talking, it may be able to hold it all in memory in the server process, or the server may need to serialize it to disk and pass it to the database that way.
from Queue import Queue q=Queue() def flushQueue(): with open(path,'w') as f: while q.qsize(): f.write(q.get()) def application(env, start_response): q.put("Hello World!") if q.qsize() > 999: flushQueue() start_response('200 OK', [('Content-Type', 'text/html')]) return ["Hello!"] How to test file locking in Python
So I want to write some files that might be locked/blocked for write/delete by other processes and like to test that upfront. As I understand: os.access(path, os.W_OK) only looks for the permissions and will return true although a file cannot currently be written. So I have this little function:
def write_test(path): try: fobj = open(path, 'a') fobj.close() return True except IOError: return False It actually works pretty well, when I try it with a file that I manually open with a Program. But as a wannabe-good-developer I want to put it in a test to automatically see if it works as expected. Thing is: If I just open(path, ‘a’) the file I can still open() it again no problem! Even from another Python instance. Although Explorer will actually tell me that the file is currently open in Python! I looked up other posts here & there about locking. Most are suggesting to install a package. You migth understand that I don’t wanna do that to test a handful lines of code. So I dug up the packages to see the actual spot where the locking is eventually done. fcntl? I don’t have that. win32con? Don’t have it either. Now in filelock there is this:
self.fd = os.open(self.lockfile, os.O_CREAT|os.O_EXCL|os.O_RDWR) When I do that on a file it moans that the file exists!! Ehhm . yea! That’s the idea! But even when I do it on a non-existing path. I can still open(path, ‘a’) it! Even from another python instance. I’m beginning to think that I fail to understand something very basic here. Am I looking for the wrong thing? Can someone point me into the right direction? Thanks!
Two python scripts writing with mutually exclusive txt file and wait for each other — without threading
I have two scripts running in python on the same machine as endless loops, they are monitoring some values constantly. They communicate via file.txt B adds messages to a file.txt so they pile up. When A checks file.txt it reads them in and empties the file.txt A
while true: Open file.txt if B is not using it or when B is done f = open('file.txt', 'r+') message=f.read() #read all the messages that have accumulated f.truncate() #delete the messages f.close() ---use message do other things while true: ---Gather information for the message Open file.txt if A is not using it or when A is done f = open('file.txt’,’a’) f.write(‘message’) f.close() The pickle is that I want each of the scripts to open file.txt only when the other one is not using it, or wait for it to stop using. So while B is writing and A wants to read then B will halt for short time and then continue. There is no need to sync the two. Actually I want them to be in-dependant, not threaded or run in parallel. For example B can go to cycles while A only goes one or vice versa. Or if A is delayed for some reason the messages would just pile up in file.txt since B continues running. I have read about putting lock on the file. Would it be possible to do in each of the scripts:
A if "file locked by B": wait for unlock lock file open file read from file empty file close file unlock file