Saved searches
Use saved searches to filter your results more quickly
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.
Pythonic HTML Parsing for Humans™
License
psf/requests-html
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Name already in use
A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Sign In Required
Please sign in to use Codespaces.
Launching GitHub Desktop
If nothing happens, download GitHub Desktop and try again.
Launching GitHub Desktop
If nothing happens, download GitHub Desktop and try again.
Launching Xcode
If nothing happens, download Xcode and try again.
Launching Visual Studio Code
Your codespace will open once ready.
There was a problem preparing your codespace, please try again.
Latest commit
Git stats
Files
Failed to load latest commit information.
README.rst
Requests-HTML: HTML Parsing for Humans™
This library intends to make parsing HTML (e.g. scraping the web) as simple and intuitive as possible.
When using this library you automatically get:
- Full JavaScript support! (Using Chromium, thanks to pyppeteer)
- CSS Selectors (a.k.a jQuery-style, thanks to PyQuery).
- XPath Selectors, for the faint of heart.
- Mocked user-agent (like a real web browser).
- Automatic following of redirects.
- Connection–pooling and cookie persistence.
- The Requests experience you know and love, with magical parsing abilities.
- Async Support
Make a GET request to ‘python.org’, using Requests:
>>> from requests_html import HTMLSession >>> session = HTMLSession() >>> r = session.get('https://python.org/')
Try async and get some sites at the same time:
>>> from requests_html import AsyncHTMLSession >>> asession = AsyncHTMLSession() >>> async def get_pythonorg(): . r = await asession.get('https://python.org/') . return r . >>> async def get_reddit(): . r = await asession.get('https://reddit.com/') . return r . >>> async def get_google(): . r = await asession.get('https://google.com/') . return r . >>> results = asession.run(get_pythonorg, get_reddit, get_google) >>> results # check the requests all returned a 200 (success) code [, , ] >>> # Each item in the results list is a response object and can be interacted with as such >>> for result in results: . print(result.html.url) . https://www.python.org/ https://www.google.com/ https://www.reddit.com/
Note that the order of the objects in the results list represents the order they were returned in, not the order that the coroutines are passed to the run method, which is shown in the example by the order being different.
Grab a list of all links on the page, as–is (anchors excluded):
Grab a list of all links on the page, in absolute form (anchors excluded):
Select an element with a CSS Selector:
>>> about = r.html.find('#about', first=True)
Grab an element’s text contents:
>>> print(about.text) About Applications Quotes Getting Started Help Python Brochure
Introspect an Element’s attributes:
Render out an Element’s HTML:
Select Elements within Elements:
Search for links within an element:
Search for text on the page:
>>> r.html.search('Python is a <> language')[0] programming
More complex CSS Selector example (copied from Chrome dev tools):
>>> r = session.get('https://github.com/') >>> sel = 'body > div.application-main > div.jumbotron.jumbotron-codelines > div > div > div.col-md-7.text-center.text-md-left > p' >>> print(r.html.find(sel, first=True).text) GitHub is a development platform inspired by the way you work. From open source to business, you can host and review code, manage projects, and build software alongside millions of other developers.
Let’s grab some text that’s rendered by JavaScript. Until 2020, the Python 2.7 countdown clock (https://pythonclock.org) will serve as a good test page:
>>> r = session.get('https://pythonclock.org')
Let’s try and see the dynamically rendered code (The countdown clock). To do that quickly at first, we’ll search between the last text we see before it (‘Python 2.7 will retire in. ‘) and the first text we see after it (‘Enable Guido Mode’).
>>> r.html.search('Python 2.7 will retire in. <>Enable Guido Mode')[0] '\n