Finding Python package dependencies
To find a Python package’s dependencies, you can parse a package’s PyPi JSON endpoint. These endpoints can be found at:
You can parse the JSON at this URL with the requests package, as follows:
package_json = requests.get('https://pypi.org/pypi/seaborn/0.11.2/json', verify = False).json() package_json['info']['requires_dist'] If you run this for the Seaborn package, you’ll get: [‘numpy (>=1.15)’, ‘scipy (>=1.0)’, ‘pandas (>=0.23)’, ‘matplotlib (>=2.2)’]
But there’s an issue here. Some of Seaborn’s dependencies have dependencies themselves. They aren’t listed here. Luckily, there is a solution. Read on!
Find package dependencies with Pipdeptree
If you want to find all (including nested) dependencies: use pipdeptree, which is a command-line tool with many options for listing and visualizing dependencies.
pip install pipdeptree pipdeptree
Runnin the pipdeptree command in your terminal will print all of your packages’ dependencies. In the example below, I took a snapshot from my pipdeptree output and show what Seaborn’s dependencies look like.
seaborn==0.11.2 - matplotlib [required: >=2.2, installed: 3.5.1] - cycler [required: >=0.10, installed: 0.11.0] - fonttools [required: >=4.22.0, installed: 4.31.2] - kiwisolver [required: >=1.0.1, installed: 1.4.1] - numpy [required: >=1.17, installed: 1.22.3] - packaging [required: >=20.0, installed: 21.3] - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.7] - pillow [required: >=6.2.0, installed: 9.0.1] - pyparsing [required: >=2.2.1, installed: 3.0.7] - python-dateutil [required: >=2.7, installed: 2.8.2] - six [required: >=1.5, installed: 1.16.0] - numpy [required: >=1.15, installed: 1.22.3] - pandas [required: >=0.23, installed: 1.4.1] - numpy [required: >=1.21.0, installed: 1.22.3] - python-dateutil [required: >=2.8.1, installed: 2.8.2] - six [required: >=1.5, installed: 1.16.0] - pytz [required: >=2020.1, installed: 2022.1] - scipy [required: >=1.0, installed: 1.8.0] - numpy [required: >=1.17.3,If you want to get pipdeptree’s output in a text file, use:
pipdeptree >> dependencies.txtCongratulations, you can now find the dependencies of Python packages.
Dependency Resolution#
pip is capable of determining and installing the dependencies of packages. The process of determining which version of a dependency to install is known as dependency resolution. This behaviour can be disabled by passing --no-deps to pip install .
How it works#
When a user does a pip install (e.g. pip install tea ), pip needs to work out the package’s dependencies (e.g. spoon , hot-water , tea-leaves etc.) and what the versions of each of those dependencies it should install. At the start of a pip install run, pip does not have all the dependency information of the requested packages. It needs to work out the dependencies of the requested packages, the dependencies of those dependencies, and so on. Over the course of the dependency resolution process, pip will need to download distribution files of the packages which are used to get the dependencies of a package.
Backtracking#
During dependency resolution, pip needs to make assumptions about the package versions it needs to install and, later, check these assumptions were not incorrect. When pip finds that an assumption it made earlier is incorrect, it has to backtrack, which means also discarding some of the work that has already been done, and going back to choose another path. This can look like pip downloading multiple versions of the same package, since pip explicitly presents each download to the user. The backtracking of choices made during is not unexpected behaviour or a bug. It is part of how dependency resolution for Python packages works.
Example The user requests pip install tea . The package tea declares a dependency on hot-water , spoon , cup , amongst others. pip starts by picking the most recent version of tea and get the list of dependencies of that version of tea . It will then repeat the process for those packages, picking the most recent version of spoon and then cup . Now, pip notices that the version of cup it has chosen is not compatible with the version of spoon it has chosen. Thus, pip will “go back” (backtrack) and try to use another version of cup . If it is successful, it will continue onto the next package (like sugar ). Otherwise, it will continue to backtrack on cup until it finds a version of cup that is compatible with all the other packages. This can look like:
$ pip install tea Collecting tea Downloading tea-1.9.8-py2.py3-none-any.whl (346 kB) |████████████████████████████████| 346 kB 10.4 MB/s Collecting spoon==2.27.0 Downloading spoon-2.27.0-py2.py3-none-any.whl (312 kB) |████████████████████████████████| 312 kB 19.2 MB/s Collecting cup>=1.6.0 Downloading cup-3.22.0-py2.py3-none-any.whl (397 kB) |████████████████████████████████| 397 kB 28.2 MB/s INFO: pip is looking at multiple versions of this package to determine which version is compatible with other requirements. This could take a while. Downloading cup-3.21.0-py2.py3-none-any.whl (395 kB) |████████████████████████████████| 395 kB 27.0 MB/s Downloading cup-3.20.0-py2.py3-none-any.whl (394 kB) |████████████████████████████████| 394 kB 24.4 MB/s Downloading cup-3.19.1-py2.py3-none-any.whl (394 kB) |████████████████████████████████| 394 kB 21.3 MB/s Downloading cup-3.19.0-py2.py3-none-any.whl (394 kB) |████████████████████████████████| 394 kB 26.2 MB/s Downloading cup-3.18.0-py2.py3-none-any.whl (393 kB) |████████████████████████████████| 393 kB 22.1 MB/s Downloading cup-3.17.0-py2.py3-none-any.whl (382 kB) |████████████████████████████████| 382 kB 23.8 MB/s Downloading cup-3.16.0-py2.py3-none-any.whl (376 kB) |████████████████████████████████| 376 kB 27.5 MB/s Downloading cup-3.15.1-py2.py3-none-any.whl (385 kB) |████████████████████████████████| 385 kB 30.4 MB/s INFO: pip is looking at multiple versions of this package to determine which version is compatible with other requirements. This could take a while. Downloading cup-3.15.0-py2.py3-none-any.whl (378 kB) |████████████████████████████████| 378 kB 21.4 MB/s Downloading cup-3.14.0-py2.py3-none-any.whl (372 kB) |████████████████████████████████| 372 kB 21.1 MB/sThese multiple Downloading cup-
lines show that pip is backtracking choices it is making during dependency resolution. If pip starts backtracking during dependency resolution, it does not know how many choices it will reconsider, and how much computation would be needed. For the user, this means it can take a long time to complete when pip starts backtracking. In the case where a package has a lot of versions, arriving at a good candidate can take a lot of time. The amount of time depends on the package size, the number of versions pip must try, and various other factors. Backtracking reduces the risk that installing a new package will accidentally break an existing installed package, and so reduces the risk that your environment gets messed up. To do this, pip has to do more work, to find out which version of a package is a good candidate to install.
Possible ways to reduce backtracking#
There is no one-size-fits-all answer to situations where pip is backtracking excessively during dependency resolution. There are ways to reduce the degree to which pip might backtrack though. Nearly all of these approaches require some amount of trial and error.
Allow pip to complete its backtracking#
In most cases, pip will complete the backtracking process successfully. This could take a very long time to complete, so this may not be your preferred option. However, it is a possible that pip will not be able to find a set of compatible versions. For this, pip will try every possible combination that it needs to and determine that there is no compatible set. If you’d prefer not to wait, you can interrupt pip (Ctrl+c) and try the strategies listed below.
Reduce the number of versions pip is trying to use#
It is usually a good idea to add constraints the package(s) that pip is backtracking on (e.g. in the above example - cup ). You could try something like:
$ python -m pip install tea "cup >= 3.13"