Python all files in dir recursive

Python all files in dir recursive

How to get the home directory in Python?

This will ensure it works on all platforms. Or we can do:

from os.path import expanduser home = expanduser("~")

What is the difference of os.path.basename() and os.path.dirname() functions?

Both functions use the os.path.split(path) function to split the pathname path into a pair; (head, tail).

1. path=«/foo/bar/item»
2. The os.path.dirname(path) function returns the head of the path.

Note that if we have a slash(‘/’) at the end of the path, we get different result:

1. path=«/foo/bar/item / «
2. The os.path.dirname(path) function returns the head of the path.

>>> os.path.dirname(path) '/foo/bar/item'

os.walk(top, topdown=True, onerror=None, followlinks=False)

The os.walk() generate the file names in a directory tree by walking the tree either top-down or bottom-up.

For each directory in the tree rooted at directory top, it yields a 3-tuple:
(dirpath, dirnames, filenames)

The dirpath is a string for the path to the directory. The dirnames is a list of the names of the subdirectories in dirpath (excluding ‘.’ and ‘..’). The filenames is a list of the names of the non-directory files in dirpath.

Note that the names in the lists contain no path components. To get a full path (which begins with top) to a file or directory in dirpath, do os.path.join(dirpath, name).

As a preview, we get the following output from the tree like this:

import os path = "./TEST" for root,d_names,f_names in os.walk(path): print root, d_names, f_names

./TEST ['D1', 'D2'] ['root.txt'] ./TEST/D1 [] ['d1_b.txt', 'd1_a.txt'] ./TEST/D2 [] ['d2_a.txt']

We can make a full path for each file:

import os path = "./TEST" fname = [] for root,d_names,f_names in os.walk(path): for f in f_names: fname.append(os.path.join(root, f)) print("fname = %s" %fname)

fname = ['./TEST/root.txt', './TEST/D1/d1_b.txt', './TEST/D1/d1_a.txt', './TEST/D2/d2_a.txt']

As another example, we’re going to use the tree below:

import os for dirpath, dirs, files in os.walk("."): print dirpath

import os for dirpath, dirs, files in os.walk("."): print dirs

import os for dirpath, dirs, files in os.walk("."): print files

['f1'] ['f3', 'f2'] ['f4', 'f5'] ['f9'] ['f6'] ['f7', 'f8']

Listing files in directories recursively?

import os for dirpath, dirs, files in os.walk("."): path = dirpath.split('/') print '|', (len(path))*'---', '[',os.path.basename(dirpath),']' for f in files: print '|', len(path)*'---', f

Suppose we are now in TREE directory, then the output should look like this:

| --- [ . ] | --- f1 | --- f0.py | ------ [ A ] | ------ f3 | ------ f2 | --------- [ AA ] | --------- f4 | --------- f5 | ------ [ C ] | ------ f9 | ------ [ B ] | ------ f6 | --------- [ BB ] | --------- f7 | --------- f8

How to traverse a directory tree?

One of the answers may be to use os.walk() to recursively traverse directories.

So, in this section, we want to print all file contents recursively using the os.walk():

import os for dirpath, dirs, files in os.walk("./TREE/"): for filename in files: fname = os.path.join(dirpath,filename) with open(fname) as myfile: print(myfile.read())

The key here is to use os.path.join() when we read the files. Note that the names in the lists contain no path components. To get a full path (which begins with top) to a file or directory in dirpath, do os.path.join(dirpath, filename).

Here is another example. It reads in CMakeLists.txt, and generate a csv file with the targets.

import os home = os.path.expanduser("~") root_dir = os.path.join(home, "TEST/TF/tf") cmake_path = os.path.join(root_dir, "CMakeLists.txt") # make target list def make_target_list(): target_tag="set(TARGET_NAME" target_list = [] for dirpath, dirs, files in os.walk(root_dir): if "CMakeLists.txt" in files: for f in files: cmake_path = os.path.join(dirpath, f) with open(cmake_path,'rb') as lines: for line in lines: if target_tag in line: target = line[line.find(target_tag)+len(target_tag)+1:line.find(")")] target_list.append(target.strip('"')) return target_list # writing csv def write_csv(t): import csv with open('tf.csv', 'wb') as f: w = csv.writer(f, delimiter=' ') w.writerow(t) if __name__ == '__main__': target = make_target_list() print target write_csv(target)

['assignment-client', 'ice-server', 'plugins', 'octree', 'embedded-webserver', 'audio', 'script-engine', 'entities-renderer', 'render-utils', 'model', 'render', 'animation', 'gpu', 'input-plugins', 'networking', 'fbx', 'ui', 'shared', 'avatars', 'display-plugins', 'entities', 'environment', 'audio-client', 'auto-updater', 'physics', 'gpu-test', 'render-utils-test', 'ui-test', 'shaders-test', 'entities-test', 'interface', 'gvr-interface', 'scribe', 'vhacd-util', 'mtc', 'domain-server']

The csv file looks like this:

$ cat tf.csv assignment-client ice-server plugins octree embedded-webserver audio script-engine entities-renderer render-utils model render animation gpu input-plugins networking fbx ui shared avatars display-plugins entities environment audio-client auto-updater physics gpu-test render-utils-test ui-test shaders-test entities-test interface gvr-interface scribe vhacd-util mtc domain-server

This is almost the same as previous ones.

I need to find files which have more than one unit of Google Ads (I am supposed to have only one of the type per page).

So, during the recursive file traversing, I have to include only (*.php) files not *.png, *.txt, etc. Also, I have to count the occurrence of the Ad-unit in a file. If a file has more than one unit, this code prints out two things : full path and the count.

import os for dirpath, dirs, files in os.walk("."): for filename in files: fname = os.path.join(dirpath,filename) if fname.endswith('.php'): with open(fname) as myfile: line = myfile.read() c = line.count('bogo_sq') if c > 1: print fname, c

../AngularJS/AngularJS_More_Directives_HTML_DOM.php 2 ../DevOps/AWS/aws_S3_Simple_Storage_Service.php 2 ../Android/android22Threads.php 2 ../python/pytut.php 2

Recently, I switched to https (SSL) from http. But found out, my CSSs were not loading properly, and it broke my pages. It turned out, I was using http, for example, «http://apis.google.com/js/plusone.js». So, I had to browse all files, and replace it to «//apis.google.com/js/plusone.js».

Here is the the script I used for the task:

import os import sys import fileinput path = "../public_html" # construct full file path recursively file_list = [] for root,d_names,f_names in os.walk(path): for f in f_names: file_list.append(os.path.join(root, f)) print("file_list : %s" %file_list) search = ['http://netdna', 'http://fonts', 'http://maxcdn', 'http://getbootstrap', 'http://php', 'http://tinymce', 'http://wiki.moxiecode', 'http://www.google', 'http://www.apple', 'http://activex.microsoft', 'http://download.macromedia', 'http://cdn.mathjax' 'http://code.jquery', 'http://w.sharethis', 'http://s7.addthis', 'http://ajax.googleapis', 'http://cdn.mathjax', 'http://d3js.org', 'http://apis.google', 'http://pagead2.googlesyndication'] replace = ['//netdna', '//fonts', '//maxcdn', '//getbootstrap', '//php', '//tinymce', '//wiki.moxiecode', '//www.google', '//www.apple', '//activex.microsoft', '//download.macromedia', '//cdn.mathjax' '//code.jquery','//w.sharethis', '//s7.addthis', '//ajax.googleapis','//cdn.mathjax','//d3js.org', '//apis.google', '//pagead2.googlesyndication'] if (len(search) != len(replace)) : sys.exit("Error: search does not match with replace") # Loop through files and replace "search" with "replace" for filename in file_list: # exclude img/video files if filename.lower().endswith(('.png', '.jpg', '.jpeg', '.gif', '.bak', '.mp4', '.ogv', '.mp3', '.ismv', '.ts', '.ismc', '.m3u8', '.mov', '.avi', '.dv', '.wmv', '.ogg', '.mkv', '.webm', '.idx', '.tar', '.gz', '.zip', '.pdf', '.svg', '.bz')): continue print(filename) # Read in the file with open(filename, 'r') as file : filedata = file.read() # Replace the target string for i in range(len(search)): filedata = filedata.replace(search[i], replace[i]) # Write the file out again with open(filename, 'w') as file: file.write(filedata)

I replaced quite a few API calls with something like this: «//apis.google». Just dropped the «http:». I also excluded some files such as media or compressed ones. I did not use in-place replacing since my python version was 2.7

The output (just listing the processed files):

. ../public_html/dj/static/admin/js/jquery.init.js ../public_html/dj/static/admin/js/collapse.js ../public_html/dj/static/admin/js/prepopulate.js .

Источник