Read All Files of Directory and Subdirectories with Recursive Generators in JavaScript
Reading all files from a directory and its subdirectories is a common requirement in many projects. While there are several ways to accomplish this, one approach is to use a recursive generator function to create a sequence of file paths. Let’s start by taking a look at the code:
import fs from 'fs'; import path from 'path'; export function* readAllFiles(dir: string): Generatorstring> const files = fs.readdirSync(dir, withFileTypes: true >); for (const file of files) if (file.isDirectory()) yield* readAllFiles(path.join(dir, file.name)); > else yield path.join(dir, file.name); > > > The function readAllFiles takes a directory path as input and returns a generator object that creates a sequence of file paths. Inside the function, we first use fs.readdirSync to read all files and directories in the specified directory. We pass the withFileTypes option to get fs.Dirent objects instead of filenames. Then we loop through each file or directory with a for..of loop. If the current item is a directory, we recursively call the readAllFiles function with the new directory path using the yield* syntax. This creates a nested sequence of file paths for each subdirectory. If the current item is a file, we use the path.join function to construct the full path of the file and yield it. Now that we have our generator function, let’s see how we can use it to read all the files in a directory and its subdirectories.
for (const file of readAllFiles('/path/to/directory')) console.log(file); > We can simply use a for..of loop to iterate through the generator object and print each file path to the console. Using a recursive generator function to read all the files in a directory and its subdirectories has several benefits over other approaches. Instead of pushing all file paths into an array, the generator yields each path one at a time. This means that only one file path is kept in memory at a time, making it more memory efficient. Moreover, the generator only generates file paths when they are requested by the consuming code. This means that if we only need to process a subset of the files, we don’t waste time and resources generating paths for files we don’t need.
How to read multiple files at once using the FileReader class in JavaScript
Learn how to read multiple JavaScript files at once using the FileReader in JavaScript.
Recently, I had to work on a very simple feature of an application that needs to read multiple files at once and concatenate its content into a single one. Curiously, such a feature could be not so easy to implement if you use inadequate methods.
In this article, I will explain to you how to easily read single or multiple files at once in JavaScript using the FileReader class and the Promises API.
Reading a single file
To understand how we are going to proceed reading multiple files, we will explain as well how the process of reading a single file works. In this case, we will use the Promises API of JavaScript, as it makes the code way easier to read and understand. In the markup, we will have a single file input that allows the user to select a single file from the system. With JavaScript, we’ll attach an event listener that reacts to the change event (when the user selects a file), when there’s a selected file, we’ll use the readFileAsText method that we created using a simple JavaScript promise, so when the FileReaders finishes with the reading of the file, it will return its context as a result when the promise is fulfilled:
This implementation is pretty useful and straightforward. However, as it’s implemented now, it can become painful to handle multiple files especially when you don’t know how many files are uploaded.
Reading multiple files at once
Now, if you think about it, there will be cases when the content of multiple files are needed at once, so if you read every file using the previously described method, the result code will be painful to read as we are working with callbacks. That’s why we are going to take advantage of the Promises API using the Promises.all method. We will use the same readFileAsText method from the first example, but we are going to proceed differently. Keep the same single file input but this time, it has the multiple attribute that will allow the user to select multiple files at once. When the user selects a file or multiple files, the attached event listener (onchange) will be triggered. We will store a promise of the readFileAsText method from every selected file into an array. Once we loop through all the files, we’ll trigger them with Promises.all that will be fulfilled and triggered once after reading every single file:
And that’s it! Quite easy isn’t it? Remember that in this example we used a very basic configuration of the FileReader class of JavaScript. In other cases, you may need the content of files as binary strings, base64, and so on, the important thing here is that you get the idea of how to read multiple files and obtain its content at once.
Node.js: How to Read All Files within a Directory Using JavaScript
To read files asynchronously, you can use either synchronous or promised versions of the function. The synchronous version reads files and stores them in an array with natural sorting. On the other hand, the promised version uses promisify, which is explained in detail in an article. If you plan on sending files after the reading is complete, it’s recommended to use the synchronous versions or promises.
Reading all files in a directory, store them in objects, and send the object
The process can be divided into three sections — reading, storing, and transmitting.
var fs = require('fs'); function readFiles(dirname, onFileContent, onError) < fs.readdir(dirname, function(err, filenames) < if (err) < onError(err); return; >filenames.forEach(function(filename) < fs.readFile(dirname + filename, 'utf-8', function(err, content) < if (err) < onError(err); return; >onFileContent(filename, content); >); >); >); > var data = <>; readFiles('dirname/', function(filename, content) < data[filename] = content; >, function(err) < throw err; >); It’s your responsibility to send them, whether you choose to send them individually or all at once after you’ve finished reading.
To send files after reading completion, it’s recommended to utilize the sync versions of fs functions or promises. Using async callbacks is not a recommended style.
It is advisable to ask your questions one at a time. Moreover, requesting for a complete solution tailored specifically for you when it comes to stripping an extension may not be feasible.
A contemporary Promise rendition has been created, which adopts a Promise.all strategy to fulfill all assurances upon completion of reading all files.
/** * Promise all * @author Loreto Parisi (loretoparisi at gmail dot com) */ function promiseAllP(items, block) < var promises = []; items.forEach(function(item,index) < promises.push( function(item,i) < return new Promise(function(resolve, reject) < return block.apply(this,[item,index,resolve,reject]); >); >(item,index)) >); return Promise.all(promises); > //promiseAll /** * read files * @param dirname string * @return Promise * @author Loreto Parisi (loretoparisi at gmail dot com) * @see http://stackoverflow.com/questions/10049557/reading-all-files-in-a-directory-store-them-in-objects-and-send-the-object */ function readFiles(dirname) < return new Promise((resolve, reject) => < fs.readdir(dirname, function(err, filenames) < if (err) return reject(err); promiseAllP(filenames, (filename,index,resolve,reject) =>< fs.readFile(path.resolve(dirname, filename), 'utf-8', function(err, content) < if (err) return reject(err); return resolve(); >); >) .then(results => < return resolve(results); >) .catch(error => < return reject(error); >); >); >); > How to Use It:
readFiles( EMAIL_ROOT + '/' + folder) .then(files => < console.log( "loaded ", files.length ); files.forEach( (item, index) =>< console.log( "item",index, "size ", item.contents.length); >); >) .catch( error => < console.log( error ); >); If you possess an additional set of directories, it is possible to also loop through them. This is because the internal promise.all will resolve each one of them asynchronously.
var folders=['spam','ham']; folders.forEach( folder => < readFiles( EMAIL_ROOT + '/' + folder) .then(files => < console.log( "loaded ", files.length ); files.forEach( (item, index) =>< console.log( "item",index, "size ", item.contents.length); >); >) .catch( error => < console.log( error ); >); >); How it Works
By utilizing the promiseAll , a function block with the signature function(item,index,resolve,reject) can be executed. This function block takes in the current item in the array ( item ), its position in the array ( index ), as well as the callback functions ( resolve and reject ). An anonymous function call is then used to push each promise into an array, with the current index and item as arguments.
promises.push( function(item,i) < return new Promise(function(resolve, reject) < return block.apply(this,[item,index,resolve,reject]); >); >(item,index)) Then all promises will be resolved:
return Promise.all(promises); To execute all the examples provided, both the fs and path modules must be imported.
const fs = require('fs'); const path = require('path'); Read files asynchronously
function readFiles(dir, processFile) < // read directory fs.readdir(dir, (error, fileNames) => < if (error) throw error; fileNames.forEach(filename => < // get current file name const name = path.parse(filename).name; // get current file extension const ext = path.parse(filename).ext; // get current file path const filepath = path.resolve(dir, filename); // get information about the file fs.stat(filepath, function(error, stat) < if (error) throw error; // check if the current path is a file or a folder const isFile = stat.isFile(); // exclude folders if (isFile) < // callback, do something with the file processFile(filepath, name, ext, stat); >>); >); >); > // use an absolute path to the folder where files are located readFiles('absolute/path/to/directory/', (filepath, name, ext, stat) => < console.log('file path:', filepath); console.log('file name:', name); console.log('file extension:', ext); console.log('file information:', stat); >); Read files synchronously, store in array, natural sorting
/** * @description Read files synchronously from a folder, with natural sorting * @param dir Absolute path to directory * @returns List of object, each object represent a file * structured like so: `< filepath, name, ext, stat >` */ function readFilesSync(dir) < const files = []; fs.readdirSync(dir).forEach(filename =>< const name = path.parse(filename).name; const ext = path.parse(filename).ext; const filepath = path.resolve(dir, filename); const stat = fs.statSync(filepath); const isFile = stat.isFile(); if (isFile) files.push(< filepath, name, ext, stat >); >); files.sort((a, b) => < // natural sort alphanumeric strings // https://stackoverflow.com/a/38641281 return a.name.localeCompare(b.name, undefined, < numeric: true, sensitivity: 'base' >); >); return files; > // return an array list of objects // each object represent a file const files = readFilesSync('absolute/path/to/directory/'); Read files async using promise
This article contains additional details about promisify.
const < promisify >= require('util'); const readdir_promise = promisify(fs.readdir); const stat_promise = promisify(fs.stat); function readFilesAsync(dir) < return readdir_promise(dir, < encoding: 'utf8' >) .then(filenames => < const files = getFiles(dir, filenames); return Promise.all(files); >) .catch(err => console.error(err)); > function getFiles(dir, filenames) < return filenames.map(filename =>< const name = path.parse(filename).name; const ext = path.parse(filename).ext; const filepath = path.resolve(dir, filename); return stat(< name, ext, filepath >); >); > function stat(< name, ext, filepath >) < return stat_promise(filepath) .then(stat =>< const isFile = stat.isFile(); if (isFile) return < name, ext, filepath, stat >; >) .catch(err => console.error(err)); > readFiles('absolute/path/to/directory/') // return an array list of objects // each object is a file // with those properties: < name, ext, filepath, stat >.then(files => console.log(files)) .catch(err => console.log(err)); Kindly note that you can filter out the folders when returning undefined .
readFiles('absolute/path/to/directory/') .then(files => files.filter(file => file !== undefined)) .catch(err => console.log(err)); Node.js fs.promise.readdir() Method, Return Value: It returns a resolved or rejected promise. The promise is resolved with the list of the names of files and folders if a directory
How to list the contents in directories with Node.js [13 of 26]
Navigate the file system and learn how to explore and list folder contents using the Node.js
Duration: 5:47
NodeJS FileSystem: Read Contents of Directory in NodeJS
In this video, we look into how we can list the names of the files and folders in a directory Duration: 8:22
Read all files from a folder with Node.js
When requesting res.send() , it is limited to one per request. Therefore, if you plan on using res.send() , you must gather all the content and submit it in a single request.
When dealing with several asynchronous operations simultaneously, it becomes necessary to monitor their completion. To simplify this process, promises can be utilized with async/await .
If your goal is to combine all the HTML that you generate, one approach is to use the following method.
const fsp = require('fs').promises; const path = require('path'); app.post('/api/world', async (req, res) => < try < let html = ""; const basePath = path.join(__dirname, 'posts'); const files = await fsp.readdir(basePath); for (let f of files) < const fileData = await fsp.readFile(path.join(basePath, f)); html += converter.makeHtml(fileData); >res.send(html); > catch(e) < // log error and send error response console.log(e); res.sendStatus(500); >>); Read content of all files inside a directory in order using node.js, readdir(dirPath, (err, dir)=> · for( ; var i=0 · length; i++)< fileName = dir[i]; ; console.log · "A: "+fileName); fs.
Read and write all files from a directory in nodejs
The following code snippet can be utilized to traverse through a directory and retrieve all the files stored within it.
var fs = require('fs') fs.readdir('/folder/with/files', (err, files) => < if (err) < console.log(err); return; >files.forEach(file => < fs.readFile(file, 'utf8', function (err,result) < if (err) < return console.log(err); >result = //logic; fs.writeFile('/new/file/path',result,'utf8', function (err) < if (err) < return console.log(err); >>); >); >); Ensure that your reasoning is in sync and operates in a sequential manner. Otherwise, opt for promises. This should be sufficient for basic file handling.
Get all directories within directory nodejs, log(«fileName: » + obj.isFile); >); >; fs.readdir(basePath, (err, files) => < var results
Read content of all files inside a directory in order using node.js
Consider using fs.lstatSync instead of the asynchronous function fs.lstat to achieve the same desired outcome.
It appears that the issue is related to scoping, as it keeps occurring repeatedly.
Nodejs read all files in directory Code Example, node get all files in folder. javascript by Adventurous Aardvark on Mar 15 2021 Comment. 11 ; Get List of all files in a directory in Node.js. javascript by