How to upload file with python requests?
I’m performing a simple task of uploading a file using Python requests library. I searched Stack Overflow and no one seemed to have the same problem, namely, that the file is not received by the server:
import requests url='http://nesssi.cacr.caltech.edu/cgi-bin/getmulticonedb_release2.cgi/post' files= values= r=requests.post(url,files=files,data=values) I’m filling the value of ‘upload_file’ keyword with my filename, because if I leave it blank, it says
Error - You must select a file to upload! File file.txt of size bytes is uploaded successfully! Query service results: There were 0 lines. 9 Answers 9
If upload_file is meant to be the file, use:
files = values = r = requests.post(url, files=files, data=values) and requests will send a multi-part form POST body with the upload_file field set to the contents of the file.txt file.
The filename will be included in the mime header for the specific field:
>>> import requests >>> open('file.txt', 'wb') # create an empty demo file >>> files = >>> print(requests.Request('POST', 'http://example.com', files=files).prepare().body.decode('ascii')) --c226ce13d09842658ffbd31e0563c6bd Content-Disposition: form-data; name="upload_file"; filename="file.txt" --c226ce13d09842658ffbd31e0563c6bd-- Note the filename=»file.txt» parameter.
You can use a tuple for the files mapping value, with between 2 and 4 elements, if you need more control. The first element is the filename, followed by the contents, and an optional content-type header value and an optional mapping of additional headers:
This sets an alternative filename and content type, leaving out the optional headers.
If you are meaning the whole POST body to be taken from a file (with no other fields specified), then don’t use the files parameter, just post the file directly as data . You then may want to set a Content-Type header too, as none will be set otherwise. See Python requests — POST data from a file.
@William: you can use a sequence of 2-value tuples too, which lets you re-use field names: files = [(‘attachment’, open(‘attachment1.txt’, ‘rb’)), (‘attachment’, open(‘attachment2.txt’, ‘rb’))] . Each tuple is a pair of key and value.
Also you can also use files= <'file':('nameoffile',open('namoffile','rb'),'Content-Type':'text/html','other header'),'file2':('nameoffile2',open('nameoffile2','rb'),'Content-Type':'application/xml','other header')>but If files=<> is used then headers= must not be used! -> @martijn-pieters: bacause the multipart/form-data Content-Type must include the boundary value used to deliniate the parts in the post body. Not setting the Content-Type header ensures that requests sets it to the correct value.
@zaki: don’t conflate the mime type of individual parts with the content type of the POST request. Your syntax in also incorrect, be careful when mixing tuples and dictionaries, and if you are only setting the mime type, just use a string in the 3rd position of a 3-element tuple: files= <'file':('nameoffile', open('nameoffile', 'rb'), text/html'), 'file2': ('nameoffile2',open('nameoffile2', 'rb'), 'application/xml')>.
(2018) the new python requests library has simplified this process, we can use the ‘files’ variable to signal that we want to upload a multipart-encoded file
url = 'http://httpbin.org/post' files = r = requests.post(url, files=files) r.text hello, its been awhile since I’ve used this library. nice question. could you give me and the others a hand by typing lsof | grep «filename» and share your results with us? thanks 🙂
With the use of lsof , is seems that the file remains open, or at least, this is how I interpret the following results. Before, running the open there is no record in lsof table about the filename . Then after the open is executed, multiple records appear with read access. After executing the requests.post , the records are still there indicating that the file did not close.
(2021) If you also need parameters when upload a file, you can add params like this: r = requests.post(url,files=files,params=<"key":value>)
Client Upload
If you want to upload a single file with Python requests library, then requests lib supports streaming uploads, which allow you to send large files or streams without reading into memory.
with open('massive-body', 'rb') as f: requests.post('http://some.url/streamed', data=f) Server Side
Then store the file on the server.py side such that save the stream into file without loading into the memory. Following is an example with using Flask file uploads.
@app.route("/upload", methods=['POST']) def upload_file(): from werkzeug.datastructures import FileStorage FileStorage(request.stream).save(os.path.join(app.config['UPLOAD_FOLDER'], filename)) return 'OK', 200 Or use werkzeug Form Data Parsing as mentioned in a fix for the issue of «large file uploads eating up memory» in order to avoid using memory inefficiently on large files upload (s.t. 22 GiB file in ~60 seconds. Memory usage is constant at about 13 MiB.).
@app.route("/upload", methods=['POST']) def upload_file(): def custom_stream_factory(total_content_length, filename, content_type, content_length=None): import tempfile tmpfile = tempfile.NamedTemporaryFile('wb+', prefix='flaskapp', suffix='.nc') app.logger.info("start receiving file . filename => " + str(tmpfile.name)) return tmpfile import werkzeug, flask stream, form, files = werkzeug.formparser.parse_form_data(flask.request.environ, stream_factory=custom_stream_factory) for fil in files.values(): app.logger.info(" ".join(["saved form name", fil.name, "submitted as", fil.filename, "to temporary file", fil.stream.name])) # Do whatever with stored file at `fil.stream.name` return 'OK', 200 How to Upload Files with Python’s requests Library
Python is supported by many libraries which simplify data transfer over HTTP. The requests library is one of the most popular Python packages as it’s heavily used in web scraping. It’s also popular for interacting with servers! The library makes it easy to upload data in a popular format like JSON, but also makes it easy to upload files as well.
In this tutorial, we will take a look at how to upload files using Python’s requests library. The article will start by covering the requests library and the post() function signature. Next, we will cover how to upload a single file using the requests package. Last but not least, we upload multiple files in one request.
Uploading a Single File with Python’s Requests Library
This tutorial covers how to send the files, we’re not concerned about how they’re created. To follow along, create three files called my_file.txt , my_file_2.txt and my_file_3.txt .
The first thing we need to do is install our the request library in our workspace. While not necessary, it’s recommended that you install libraries in a virtual environment:
Activate the virtual environment so that we would no longer impact the global Python installation:
Now let’s install the requests library with pip :
Create a new file called single_uploader.py which will store our code. In that file, let’s begin by importing the requests library:
Now we’re set up to upload a file! When uploading a file, we need to open the file and stream the content. After all, we can’t upload a file we don’t have access to. We’ll do this with the open() function.
The open() function accepts two parameters: the path of the file and the mode. The path of the file can be an absolute path or a relative path to where the script is being run. If you’re uploading a file in the same directory, you can just use the file’s name.
The second argument, mode, will take the «read binary» value which is represented by rb . This argument tells the computer that we want to open the file in the read mode, and we wish to consume the data of the file in a binary format:
test_file = open("my_file.txt", "rb") Note: it’s important to read the file in binary mode. The requests library typically determines the Content-Length header, which is a value in bytes. If the file is not read in bytes mode, the library may get an incorrect value for Content-Length , which would cause errors during file submission.
For this tutorial, we’ll make requests to the free httpbin service. This API allows developers to test their HTTP requests. Let’s create a variable that stores the URL we’ll post our files to:
test_url = "http://httpbin.org/post" We now have everything to make the request. We’ll use the post() method of the requests library to upload the file. We need two arguments to make this work: the URL of the server and files property. We’ll also save the response in a variable, write the following code:
test_response = requests.post(test_url, files = "form_field_name": test_file>) The files property takes a dictionary. The key is the name of the form field that accepts the file. The value is the bytes of the opened file you want to upload.
Normally to check if your post() method was successful we check the HTTP status code of the response. We can use the ok property of the response object, test_url . If it’s true, we’ll print out the response from the HTTP server, in this case, it will echo the request:
Free eBook: Git Essentials
Check out our hands-on, practical guide to learning Git, with best-practices, industry-accepted standards, and included cheat sheet. Stop Googling Git commands and actually learn it!
if test_response.ok: print("Upload completed successfully!") print(test_response.text) else: print("Something went wrong!") Let’s try it out! In the terminal, execute your script with the python command:
Your output would be similar to this:
Upload completed successfully! < "args": <>, "data": "", "files": < "form_field_name": "This is my file\nI like my file\n" >, "form": <>, "headers": < "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "189", "Content-Type": "multipart/form-data; boundary=53bb41eb09d784cedc62d521121269f8", "Host": "httpbin.org", "User-Agent": "python-requests/2.25.0", "X-Amzn-Trace-Id": "Root=1-5fc3c190-5dea2c7633a02bcf5e654c2b" >, "json": null, "origin": "102.5.105.200", "url": "http://httpbin.org/post" > As a sanity check, you can verify the form_field_name value matches what’s in your file.
Uploading Multiple Files with Python’s requests Library
Uploading multiple files using requests is quite similar to a single file, with the major difference being our use of lists. Create a new file called multi_uploader.py and the following setup code:
import requests test_url = "http://httpbin.org/post" Now create a variable called test_files that’s a dictionary with multiple names and files:
test_files = < "test_file_1": open("my_file.txt", "rb"), "test_file_2": open("my_file_2.txt", "rb"), "test_file_3": open("my_file_3.txt", "rb") > Like before, the keys are the names of the form fields and the values are the files in bytes.
We can also create our files variables as a list of tuples. Each tuple contains the name of the form field accepting the file, followed by the file’s contents in bytes:
test_files = [("test_file_1", open("my_file.txt", "rb")), ("test_file_2", open("my_file_2.txt", "rb")), ("test_file_3", open("my_file_3.txt", "rb"))] Either works so choose whichever one you prefer!
Once the list of files is ready, you can send the request and check its response like before:
test_response = requests.post(test_url, files = test_files) if test_response.ok: print("Upload completed successfully!") print(test_response.text) else: print("Something went wrong!") Execute this script with the python command:
Upload completed successfully! < "args": <>, "data": "", "files": < "test_file_1": "This is my file\nI like my file\n", "test_file_2": "All your base are belong to us\n", "test_file_3": "It's-a me, Mario!\n" >, "form": <>, "headers": < "Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Content-Length": "470", "Content-Type": "multipart/form-data; boundary=4111c551fb8c61fd14af07bd5df5bb76", "Host": "httpbin.org", "User-Agent": "python-requests/2.25.0", "X-Amzn-Trace-Id": "Root=1-5fc3c744-30404a8b186cf91c7d239034" >, "json": null, "origin": "102.5.105.200", "url": "http://httpbin.org/post" > Good job! You can upload single and multiple files with requests !
Conclusion
In this article, we learned how to upload files in Python using the requests library. Where it’s a single file or multiple files, only a few tweaks are needed with the post() method. We also verified our response to ensure that our uploads were successful.