Python package path linux

How To Set Python Module Search Path To Find Modules

When you want to import a python module library in your python source code, you need first to make the python module library importable by adding the module package path in the PYTHONPATH system environment variable. You can also add the python module package path to the python module search path at runtime in python source code. This example will show you how to do it.

1. Add Python Module Package Path In System Environment Variable PYTHONPATH.

Suppose your python module is saved in folder /tmp. We will add /tmp folder in the PYTHONPATH environment variable value.

Open a terminal and go to the user home directory use cd ~ command.

$ ls -al . -rw-r--r--@ 1 zhaosong staff 1176 Apr 30 09:15 .bash_profile .

2. Display Python Library Search Path In Python Source Code.

Run into python interactive console in a terminal.

$ python3 Python 3.6.5 |Anaconda, Inc.| (default, Apr 26 2018, 08:42:37) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>>

>>> import sys >>> for path in sys.path: . print(path) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa

3. Append Directory To Python Library Search Path.

1. Python sys.path.append function can append directory to the end of python library search directory.

>>> sys.path.append('/abc') >>> >>> >>> for line in sys.path: . print(line) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa /abc

4. Append Exist Module Library Directory To Python Library Search Directory.

1. Python module’s __file__ attribute returns the module file saved directory. You can append that directory to the python library search path as below.

>>> import sys, os # os.__file__ will return the os module directory. >>> sys.path.append(os.__file__) >>> >>> >>> for line in sys.path: . print(line) . /tmp /Users/zhaosong/anaconda3/lib/python36.zip /Users/zhaosong/anaconda3/lib/python3.6 /Users/zhaosong/anaconda3/lib/python3.6/lib-dynload /Users/zhaosong/.local/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages /Users/zhaosong/anaconda3/lib/python3.6/site-packages/aeosa /abc /Users/zhaosong/anaconda3/lib/python3.6/os.py

5. Question & Answer.

5.1 How can I add another python source directory in the Python search path for a large python project.

1. My team just handle an old python project from another team, the python project is so large, there are a lot of python source files in the python project. Our development environment is Linux and no IDE, only in the command line. And when I run the python script with the command python abc.py it prompts the error ImportError: no module named com.test_module. And there is a lot of such kind of errors in other python scripts. All the old python project files are saved in a folder like /codebase/old_python_project. And there are a lot of subfolders in the project base folder. How can I make python search all the modules in the project folder and it’s subfolders to fix the import error? Thanks.
2. You can add your existing python project folder in the PYTHONPATH system environment variable to fix your issue. You can also use the function sys.path.append(‘/codebase/old_python_project’) in your python script source code and then can import the python modules. If you want to add all the project subfolders in the python module search path, you can reverse loop your project folder and when you reach it’s subfolder then you can call the sys.path.append() function to add the subfolder to the python module search path, you can try it.

import os, sys def reverse_add_python_module_search_path(module_dir): files_array = [] files_array = os.listdir(module_dir) for file in files_array: if os.path.isdir(file): sys.path.append(file) reverse_add_python_module_search_path(file)

Источник

How does python find packages?

I just ran into a situation where I compiled and installed Python 2.7.9 from source on Ubuntu, but Python could not find the packages I had previously installed. This naturally raises the question — how does Python know where to find packages when you call import ? This post applies specifically to Python 2.7.9, but I’m guessing Python 3x works very similarly.

In this post I first describe how Python finds packages, and then I’ll finish with the discovery I made regarding the default Python that ships with Ubuntu and how it differs from vanilla Python in how it finds packages.

sys.path

Python imports work by searching the directories listed in sys.path .

Using my default Ubuntu 14.04 Python:

> import sys > print '\n'.join(sys.path) /usr/lib/python2.7 /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages

So Python will find any packages that have been installed to those locations.

How sys.path gets populated

As the docs explain, sys.path is populated using the current working directory, followed by directories listed in your PYTHONPATH environment variable, followed by installation-dependent default paths, which are controlled by the site module.

You can read more about sys.path in the Python docs.

Assuming your PYTHONPATH environment variable is not set, sys.path will consist of the current working directory plus any manipulations made to it by the site module.

The site module is automatically imported when you start Python, you can read more about how it manipulates your sys.path in the Python docs.

You can manipulate sys.path

You can manipulate sys.path during a Python session and this will change how Python finds modules. For example:

import sys, os # This won't work - there is no hi module import hi Traceback (most recent call last): File "", line 1, in module> ImportError: No module named hi # Create a hi module in your home directory. home_dir = os.path.expanduser("~") my_module_file = os.path.join(home_dir, "hi.py") with open(my_module_file, 'w') as f: f.write('print "hi"\n') f.write('a=10\n') # Add the home directory to sys.path sys.path.append(home_dir) # Now this works, and prints hi! import hi print hi.a 

The module __file__ attribute

When you import a module, you usually can check the __file__ attribute of the module to see where the module is in your filesystem:

> import numpy > numpy.__file__ '/usr/local/lib/python2.7/dist-packages/numpy/__init__.pyc' 

The file attribute is not present for C modules that are statically linked into the interpreter; for extension modules loaded dynamically from a shared library, it is the pathname of the shared library file.

So, for example this doesn’t work:

> import sys > sys.__file__ Traceback (most recent call last): File "", line 1, in module> AttributeError: 'module' object has no attribute '__file__' 

It makes sense that the sys module is statically linked to the interpreter — it is essentially part of the interpreter!

The imp module

Python exposes the entire import system through the imp module. That’s pretty cool that all of this stuff is exposed for us to abuse, if we wanted to.

imp.find_module can be used to find a module:

> import imp > imp.find_module('numpy') (None, '/usr/local/lib/python2.7/dist-packages/numpy', ('', '', 5)) 

You can also import and arbitrary Python source as a module using imp.load_source . This is the same example before, except imports our module using imp instead of by manipulating sys.path :

import sys, os, imp # Create a hi module in your home directory. home_dir = os.path.expanduser("~") my_module_file = os.path.join(home_dir, "hi.py") with open(my_module_file, 'w') as f: f.write('print "hi"\n') f.write('a=10\n') # Load the hi module using imp hi = imp.load_source('hi', my_module_file) # Now this works, and prints hi! import hi print hi.a # a is 10! print type(hi) # it's a module! 

Passing ‘hi’ to imp.load_source simply sets the __name__ attribute of the module.

Ubuntu Python

Now back to the issue of missing packages after installing a new version of Python compiled from source. By comparing the sys.path from both the Ubuntu Python, which resides at /usr/bin/python , and the newly installed Python, which resides at /usr/local/bin/python , I could sort things out:

Ubuntu Python ( /usr/bin/python ):

>>> import sys >>> print '\n'.join(sys.path) /usr/lib/python2.7 /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages

Python compiled from source ( /usr/local/bin/python )

>>> import sys >>> print '\n'.join(sys.path) /usr/local/lib/python27.zip /usr/local/lib/python2.7 /usr/local/lib/python2.7/plat-linux2 /usr/local/lib/python2.7/lib-tk /usr/local/lib/python2.7/lib-old /usr/local/lib/python2.7/lib-dynload /usr/local/lib/python2.7/site-packages

Turns out what mattered for me was dist-packages vs. site-packages . Using Ubuntu’s Python, my packages were installed to /usr/local/lib/python2.7/dist-packages , whereas the new Python I installed expects packages to be installed to /usr/local/lib/python2.7/site-packages . I just had to manipulate the PYTHONPATH environment variable to point to dist-packages in order to gain access to the previously installed packaged with the newly installed version of Python.

How did Ubuntu manipulate the sys.path ?

So how does the Ubuntu distribution of Python know to use /usr/local/lib/python2.7/dist-packages in sys.path ? It’s hardcoded into their site module! First, find where the site module code lives:

> import site > site.__file__ '/usr/lib/python2.7/site.pyc' 

Here is an excerpt from Ubuntu Python’s site.py , which I peeked by opening /usr/lib/python2.7/site.py in a text editor. First, a comment at the top:

For Debian and derivatives, this sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python /dist-packages, Debian addons install into /usr//python /dist-packages. /usr/lib/python /site-packages is not used.

OK so there you have it. They explain how the Debian distribution of Python is different.

And now, for the code that implementes this change:

def getsitepackages(): """Returns a list containing all global site-packages directories (and possibly site-python). For each directory present in the global ``PREFIXES``, this function will find its `site-packages` subdirectory depending on the system environment, and will return a list of full paths. """ sitepackages = [] seen = set() for prefix in PREFIXES: if not prefix or prefix in seen: continue seen.add(prefix) if sys.platform in ('os2emx', 'riscos'): sitepackages.append(os.path.join(prefix, "Lib", "site-packages")) elif os.sep == '/': sitepackages.append(os.path.join(prefix, "local/lib", "python" + sys.version[:3], "dist-packages")) sitepackages.append(os.path.join(prefix, "lib", "python" + sys.version[:3], "dist-packages")) else: sitepackages.append(prefix) sitepackages.append(os.path.join(prefix, "lib", "site-packages")) if sys.platform == "darwin": # for framework builds *only* we add the standard Apple # locations. from sysconfig import get_config_var framework = get_config_var("PYTHONFRAMEWORK") if framework: sitepackages.append( os.path.join("/Library", framework, sys.version[:3], "site-packages")) return sitepackages 

It’s all there, if you are crazy enough to dig this deep.

© Lee Mendelowitz – Built with Pure Theme for Pelican

Источник