Python requests send form data

How to send a POST with Python Requests?

A POST request is a particular type of HTTP method used when we send data to services on the web. We use them on web sites that use forms — when we login, when we send messages or post an image. Applications also use POST requests to interact with other services to send data with JSON being a common data format for exchange.

The Requests library is one of the most popular HTTP client libraries for Python. It currently has over 45k stars on Github, with downloads on PyPI of 115M a month! It makes sending POST requests much simpler programmatically than having to send data via a headless browser. With a headless browser we’d need to write scripts to navigate to a site, move between fields, populate each of them with the desired data before finally submitting the data. By sending a POST request, we skip straight to the final submission step. Requests also is a much, much smaller library than a browser resulting in better performance and memory usage.

In this article we’ll cover how to construct a POST request using Requests and how it can make the process much simpler for us.

Building a JSON POST Request with Requests

As an example, lets start by building a JSON POST request the hard way. Don’t worry Requests will simplify this for us later! We’re using the httpbin.org service, which returns a JSON response detailing the content that was sent.

1. Set the Request Method to POST

Requests has a really simple HTTP verb based design, meaning that to get a response we call it’s .post() method supplying our URI as an argument. It also provides similar methods for GET, PUT and PATCH requests.

import requests r = requests.post("https://httpbin.org/post") 

2. Set the POST data

To actually send some data, we supply a data argument. By default data is sent using as a HTML form as we would get by submitting any form on the web. Requests sets the content type to ‘application/x-www-form-urlencoded’, so it isn’t necessary to set any headers.

import requests import json r = requests.post("https://httpbin.org/post", data="key": "value">, ) 

Inspecting the response from the httpbin.org service, we can see what was sent as form data under the «form» key.

>>> r.text '\n "args": <>, \n "data": "", \n "files": <>, \n "form": \n "key": "value"\n >, \n "headers": \n "Accept": "*/*", \n "Accept-Encoding": "gzip, deflate", \n "Content-Length": "9", \n "Content-Type": "application/x-www-form-urlencoded", \n "Host": "httpbin.org", \n "User-Agent": "python-requests/2.25.1", \n "X-Amzn-Trace-Id": "Root=1-60df1a04-0384d3ce7d9ac00b5855064b"\n >, \n "json": null, \n "origin": "**.***.**.***", \n "url": "https://httpbin.org/post"\n>\n' 

If we want to send JSON, we can supply a JSON formatted string. In our example, we’re using the json module to convert a dictionary to JSON formatted string ready to send.

import requests import json r = requests.post("https://httpbin.org/post", data=json.dumps("key": "value">), ) 

3. Set the POST Headers

Our JSON example isn’t going to work as it is now. With many services, we’ll likely get a 400 (Bad Request) HTTP status code. To prevent this we also need to inform the service we’re calling that our data is JSON so it can be handled correctly. To do so, we set the ‘Content-Type’ to ‘application/json’ in the request headers:

import requests import json r = requests.post( "https://httpbin.org/post", data=json.dumps("key": "value">), headers="Content-Type": "application/json">, ) 

4. POST JSON Data

If you think our JSON examples so far look a bit complex — you’re totally right. Requests makes it very easy to reduce all this down to a much simpler call. We can just supply a ‘json’ argument with our data. Requests will correctly set our headers and encode the JSON formatted string for us automatically.

import requests r = requests.post('https://httpbin.org/post', json='key':'value'>) 

Reading JSON Responses

By inspecting the response from our service, we can see the data returned is a JSON formatted string too. This is only text, so we’d need to parse it ourselves to use it within our Python script.

>>> r.text '\n "args": <>, \n "data": "\\"key\\": \\"value\\">", \n "files": <>, \n "form": <>, \n "headers": \n "Accept": "*/*", \n "Accept-Encoding": "gzip, deflate", \n "Content-Length": "16", \n "Content-Type": "application/json", \n "Host": "httpbin.org", \n "User-Agent": "python-requests/2.25.1", \n "X-Amzn-Trace-Id": "Root=1-60df0aaa-3105fc35436042571335fa22"\n >, \n "json": \n "key": "value"\n >, \n "origin": "**.***.**.***", \n "url": "https://httpbin.org/post"\n>\n' 

Alternatively, if we want to access the response as a JSON object we can also use Requests built in JSON decoder by calling .json() on the response object.

>>> r.json() 'args': <>, 'data': '', 'files': <>, 'form': <>, 'headers': 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '16', 'Content-Type': 'application/json', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.25.1', 'X-Amzn-Trace-Id': 'Root=1-60deee08-4e9c76b6457826d5001b76fa'>, 'json': 'key': 'value'>, 'origin': '**.***.**.***', 'url': 'http://httpbin.org/post'> 

Making POST requests within a Session

As a final example, lets login to Hacker News using Requests and submit some form data to login in combination with a requests.Session() object. Any cookies we receive will be stored in the object, and will be reused in later .get() calls to the session.

For our example, we’ll search the response text of the news page to confirm if we’re shown the logout link indicating we’re being served a logged in page.

import requests session = requests.Session() r = session.post("https://news.ycombinator.com/login", data=< "acct": "username", "pw": "***not-telling***" >) r = session.get("https://news.ycombinator.com/news") logout_pos = r.text.find("logout") print(r.text[logout_pos-20:logout_pos+20]) # 

Great! Our link is in the response indicating we’re logged in. We can now continue to interact with the site using the session object via Requests.

Conclusion

You’ve seen just how simple Requests makes it to post data to sites or services and how we can reduce much of the common code in our applications by using it. This simple design has certainly contributed to it’s success in the Python community, making it a firm favorite with developers. Before you reach for a full blown headless browser to send your data, you should definitely consider using Requests.

You will often need proxies for your web scraping projects, don’t hesitate to checkout this article on how to use proxies with Python Requests.

Ian is a freelance developer with a passion for simple solutions. He has written code to power surveys, studio pipelines and holds a PhD in distributed computing.

Источник

Send Form Data With Requests in Python

The requests library is a great tool for making requests to the web using Python. This article will discuss using this module to send form data to a site. In particular, we will be using the requests.post(url, data=None, json=None) function.

For demonstration purposes, we will send POST requests to https://httpbin.org – an open-source site that allows one to inspect HTTPS requests before using them in production.

Example 1

Let’s start by posting a string to httpbin.org and see the result.

Output (truncated):

< "args": <>, "data": "value1", "files": <>, "form": <>, … "json": null, "origin": "105.161.11.xxx", "url": "https://httpbin.org/post" >

The server response in the output shows the data value we passed into requests.post() is posted under “data”. Form data should be posted on the “form” property. It is empty at the moment. Let’s post some form data in the following example.

Example 2

To pass form-encoded data, just like you will do on HTML forms, you need to pass dictionary-formatted items to the data argument on requests.posts().

The dictionary will be automatically passed as form data when the POST request is made. Here is an example.

Output (truncated):

< "args": <>, "data": "", "files": <>, "form": < "comments": "this is a comment", "custemail": "[email protected]", "custname": "Karl", "custtel": "01883", "delivery": "12:00", "size": "large", "topping": "cheese" >, … "origin": "105.161.11.xxx", "url": "https://httpbin.org/post" >

As shown in the output, the dictionary data was passed as a form when the request was made. That is why, in this example, the “form” property of the response has the data we passed to requests.post() function. Let’s go into another interesting example.

Example 3

The requests.post() also allows the user to pass multiple values to a single dictionary key (form property). This can be done in two ways:

• Using a list of tuples. Here is an example with two values for the “items” property.

data = [ ( «customer_name» , «Smith» ) , ( «delivery» , «12:00» ) , ( «items» , «item1» ) , ( «items» , «item2» ) ]

• Using a dictionary with all possible values of a given key in a list. The example in the first method can be represented as:

Источник