Python installed packages location

How does python find packages?

I just ran into a situation where I compiled and installed Python 2.7.9 from source on Ubuntu, but Python could not find the packages I had previously installed. This naturally raises the question — how does Python know where to find packages when you call import ? This post applies specifically to Python 2.7.9, but I’m guessing Python 3x works very similarly.

In this post I first describe how Python finds packages, and then I’ll finish with the discovery I made regarding the default Python that ships with Ubuntu and how it differs from vanilla Python in how it finds packages.

sys.path

Python imports work by searching the directories listed in sys.path .

Using my default Ubuntu 14.04 Python:

> import sys > print '\n'.join(sys.path) /usr/lib/python2.7 /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages

So Python will find any packages that have been installed to those locations.

How sys.path gets populated

As the docs explain, sys.path is populated using the current working directory, followed by directories listed in your PYTHONPATH environment variable, followed by installation-dependent default paths, which are controlled by the site module.

You can read more about sys.path in the Python docs.

Assuming your PYTHONPATH environment variable is not set, sys.path will consist of the current working directory plus any manipulations made to it by the site module.

The site module is automatically imported when you start Python, you can read more about how it manipulates your sys.path in the Python docs.

You can manipulate sys.path

You can manipulate sys.path during a Python session and this will change how Python finds modules. For example:

import sys, os # This won't work - there is no hi module import hi Traceback (most recent call last): File "", line 1, in module> ImportError: No module named hi # Create a hi module in your home directory. home_dir = os.path.expanduser("~") my_module_file = os.path.join(home_dir, "hi.py") with open(my_module_file, 'w') as f: f.write('print "hi"\n') f.write('a=10\n') # Add the home directory to sys.path sys.path.append(home_dir) # Now this works, and prints hi! import hi print hi.a 

The module __file__ attribute

When you import a module, you usually can check the __file__ attribute of the module to see where the module is in your filesystem:

> import numpy > numpy.__file__ '/usr/local/lib/python2.7/dist-packages/numpy/__init__.pyc' 

The file attribute is not present for C modules that are statically linked into the interpreter; for extension modules loaded dynamically from a shared library, it is the pathname of the shared library file.

So, for example this doesn’t work:

> import sys > sys.__file__ Traceback (most recent call last): File "", line 1, in module> AttributeError: 'module' object has no attribute '__file__' 

It makes sense that the sys module is statically linked to the interpreter — it is essentially part of the interpreter!

The imp module

Python exposes the entire import system through the imp module. That’s pretty cool that all of this stuff is exposed for us to abuse, if we wanted to.

imp.find_module can be used to find a module:

> import imp > imp.find_module('numpy') (None, '/usr/local/lib/python2.7/dist-packages/numpy', ('', '', 5)) 

You can also import and arbitrary Python source as a module using imp.load_source . This is the same example before, except imports our module using imp instead of by manipulating sys.path :

import sys, os, imp # Create a hi module in your home directory. home_dir = os.path.expanduser("~") my_module_file = os.path.join(home_dir, "hi.py") with open(my_module_file, 'w') as f: f.write('print "hi"\n') f.write('a=10\n') # Load the hi module using imp hi = imp.load_source('hi', my_module_file) # Now this works, and prints hi! import hi print hi.a # a is 10! print type(hi) # it's a module! 

Passing ‘hi’ to imp.load_source simply sets the __name__ attribute of the module.

Ubuntu Python

Now back to the issue of missing packages after installing a new version of Python compiled from source. By comparing the sys.path from both the Ubuntu Python, which resides at /usr/bin/python , and the newly installed Python, which resides at /usr/local/bin/python , I could sort things out:

Ubuntu Python ( /usr/bin/python ):

>>> import sys >>> print '\n'.join(sys.path) /usr/lib/python2.7 /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages

Python compiled from source ( /usr/local/bin/python )

>>> import sys >>> print '\n'.join(sys.path) /usr/local/lib/python27.zip /usr/local/lib/python2.7 /usr/local/lib/python2.7/plat-linux2 /usr/local/lib/python2.7/lib-tk /usr/local/lib/python2.7/lib-old /usr/local/lib/python2.7/lib-dynload /usr/local/lib/python2.7/site-packages

Turns out what mattered for me was dist-packages vs. site-packages . Using Ubuntu’s Python, my packages were installed to /usr/local/lib/python2.7/dist-packages , whereas the new Python I installed expects packages to be installed to /usr/local/lib/python2.7/site-packages . I just had to manipulate the PYTHONPATH environment variable to point to dist-packages in order to gain access to the previously installed packaged with the newly installed version of Python.

How did Ubuntu manipulate the sys.path ?

So how does the Ubuntu distribution of Python know to use /usr/local/lib/python2.7/dist-packages in sys.path ? It’s hardcoded into their site module! First, find where the site module code lives:

> import site > site.__file__ '/usr/lib/python2.7/site.pyc' 

Here is an excerpt from Ubuntu Python’s site.py , which I peeked by opening /usr/lib/python2.7/site.py in a text editor. First, a comment at the top:

For Debian and derivatives, this sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python /dist-packages, Debian addons install into /usr//python /dist-packages. /usr/lib/python /site-packages is not used.

OK so there you have it. They explain how the Debian distribution of Python is different.

And now, for the code that implementes this change:

def getsitepackages(): """Returns a list containing all global site-packages directories (and possibly site-python). For each directory present in the global ``PREFIXES``, this function will find its `site-packages` subdirectory depending on the system environment, and will return a list of full paths. """ sitepackages = [] seen = set() for prefix in PREFIXES: if not prefix or prefix in seen: continue seen.add(prefix) if sys.platform in ('os2emx', 'riscos'): sitepackages.append(os.path.join(prefix, "Lib", "site-packages")) elif os.sep == '/': sitepackages.append(os.path.join(prefix, "local/lib", "python" + sys.version[:3], "dist-packages")) sitepackages.append(os.path.join(prefix, "lib", "python" + sys.version[:3], "dist-packages")) else: sitepackages.append(prefix) sitepackages.append(os.path.join(prefix, "lib", "site-packages")) if sys.platform == "darwin": # for framework builds *only* we add the standard Apple # locations. from sysconfig import get_config_var framework = get_config_var("PYTHONFRAMEWORK") if framework: sitepackages.append( os.path.join("/Library", framework, sys.version[:3], "site-packages")) return sitepackages 

It’s all there, if you are crazy enough to dig this deep.

© Lee Mendelowitz – Built with Pure Theme for Pelican

Источник

Where Does pip Install Packages

To see where pip installs packages on your system, run the following command:

And replace with the actual name of the package.

Example: NumPy Location

For example, let’s see where NumPy is installed:

Name: numpy Version: 1.22.2 Summary: NumPy is the fundamental package for array computing with Python. Home-page: https://www.numpy.org Author: Travis E. Oliphant et al. Author-email: None License: BSD Location: /usr/local/lib/python3.8/site-packages Requires: Required-by: torchvision, perfplot, opencv-python, matplotx, DALL-E, benchit

Here you can see that the location field says the package is installed at /usr/local/lib/python3.8/site-packages.

The location obviously depends on your system and Python version.

How to View All pip Package Locations

To list all the installed package locations, run the following command:

This spits out a huge list of different packages and their locations:

alabaster 0.7.8 /usr/lib/python3/dist-packages apparmor 2.13.3 /usr/lib/python3/dist-packages appdirs 1.4.3 /usr/lib/python3/dist-packages apturl 0.5.2 /usr/lib/python3/dist-packages .

Now you understand how to check the pip package locations using the command line/terminal.

Next, let’s take a look at how you can find this information using a Python script.

How to View pip Package Location in Python Script?

In addition to using the command line to figure out the location of packages installed via pip, you can run a Python script to get the information.

There are two ways to do this:

The site Module

To find the general location of pip packages in a Python script:

1. Import the site package.
2. Call the getsitepackages() function of the module.
3. See a list of global package locations.

Here is how it looks in code:

>>> import site >>> site.getsitepackages() ['/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.8/dist-packages']

And to get a user-specific package location as a string, call the getusersitepackages() function instead.

The help() Function

Of course, you can always use the help() function to get all kinds of information about any Python package or object.

This also shows you the location where the package is installed.

For example, lets’ see where the pandas package is installed:

>>> import pandas >>> help(pandas)

Running this piece of code opens up the package-specific manual.

If you scroll all the way down to the end of this output, you can see the FILE section where it shows you the path of the package.

By the way, in case you happen to be unfamiliar with the help() function in Python, I highly recommend you read this article. Using help() can help you a lot and save valuable coding time!

About the Author

Hi, I’m Artturi Jalli!

I’m a Tech enthusiast from Finland.

I make Coding & Tech easy and fun with well-thought how-to guides and reviews.

I’ve already helped 5M+ visitors reach their goals!

ChatGPT Review (and How to Use It)—A Full Guide (2023)

ChatGPT is the newest Artificial Intelligence language model developed by OpenAI. Essentially, ChatGPT is an AI-based chatbot that can answer any question. It understands complex topics, like.

10 Best AI Art Generators of 2023 (Reviewed & Ranked)

Choosing the right type of AI art generator is crucial to produce unique, original, and professional artwork. With the latest advancements in AI art generation, you can.

How to Make an App — A Complete 10-Step Guide (in 2023)

Are you looking to create the next best-seller app? Or are you curious about how to create a successful mobile app? This is a step-by-step guide on.

9 Best Graphic Design Courses + Certification (in 2023)

Do you want to become a versatile and skilled graphic designer? This is a comprehensive article on the best graphic design certification courses. These courses prepare you.

8 Best Python Courses with Certifications (in 2023)

Are you looking to become a professional Python developer? Or are you interested in programming but don’t know where to start? Python is a beginner-friendly and versatile.

8 Best Swift & iOS App Development Courses [in 2023]

Are you looking to become an iOS developer? Do you want to create apps with an outstanding design? Do you want to learn to code? IOS App.

Источник