feeding input to variable from txt line by line in python
I have a variable DOMAIN which takes url as input. I want to feed it list of URLs one by one from txt file. My txt file looks like this:
www.yahoo.com www.google.com www.bing.com with open('list.txt') as f: content = f.readlines() content = [x.strip() for x in content] DOMAIN = content But the variable DOMAIN takes all URLs once, not separately. It must process one URL as whole and second on another operation. On a side note, this DOMAIN variable is feed to scrapy for crawling. part of codebase:
from scrapy.selector import HtmlXPathSelector from scrapy.spider import BaseSpider from scrapy.http import Request with open('list.txt') as f: content = f.readlines() # you may also want to remove whitespace characters like `\n` at the end of each line content = [x.strip() for x in content] DOMAIN = content URL = 'http://%s' % DOMAIN class MySpider(BaseSpider): name = DOMAIN allowed_domains = [DOMAIN] start_urls = [ URL ] scrapy.downloadermiddlewares.retry] DEBUG: Retrying executing as scrapy runspider spider.py from scrapy.selector import HtmlXPathSelector from scrapy.spider import BaseSpider from scrapy.http import Request DOMAIN = 'google.com' URL = 'http://%s' % DOMAIN class MySpider(BaseSpider): name = DOMAIN allowed_domains = [DOMAIN] start_urls = [ URL ] def parse(self, response): hxs = HtmlXPathSelector(response) for url in hxs.select('//a/@href').extract(): if not ( url.startswith('http://') or url.startswith('https://') ): url= URL + url print url yield Request(url, callback=self.parse) Read a multi-line string as one line in Python
I am writing a program that analyzes a large directory text file line-by-line. In doing so, I am trying to extract different parts of the file and categorize them as ‘Name’, ‘Address’, etc. However, due to the format of the file, I am running into a problem. Some of the text i have is split into two lines, such as:
How can I make it so that even through line-by-line analysis, Python returns this as a single-line string in the form of ‘123 ABCDEF ST APT 456’ ?
I get the feeling that, since you’re saying «Line by line analysis», you don’t want all newlines removed, but only those, eg, between single-quotes. Is that true?
6 Answers 6
if you want to remove newlines:
It works as it is supposed to do. However, after having removed the newlines some words «collide» with each other. How to fix this problem? How to put a space in between?
If you do » «.join( my_string.splitlines()) you’ll get a space-separated string instead. But you will then get things like trailing spaces; at that point you probably want » «.join(line.strip() for line in mystring,splitlines())
Assuming you are using windows if you do a print of the file to your screen you will see
the \n represent the line breaks.
so there are a number of ways to get rid of the new lines in the file. One easy way is to split the string on the newline characters and then rejoin the items from the list that will be created when you do the split
myList = [item for item in myFile.split('\n')] newString = ' '.join(myList) To replace the newlines with a space:
address = '123 ABCDEF ST\nAPT 456\n' address.replace("\n", " ") import re def mergeline(c, l): if c: return c.rstrip() + " " + l else: return l def getline(fname): qstart = re.compile(r'^\'[^\']*$') qend = re.compile(r'.*\'$') with open(fname) as f: linecache, halfline = ("", False) for line in f: if not halfline: linecache = "" linecache = mergeline(linecache, line) if halfline: halfline = not re.match(qend, line) else: halfline = re.match(qstart, line) if not halfline: yield linecache if halfline: yield linecache for line in getline('input'): print line.rstrip() This hurt my head to follow, but looks pretty efficient. I tested it and it works as long as there’s only one
field per row, but OP mentioned Name and Address so could be multiple fields per row, each potentially split on multiple lines. For example, this wouldn’t be processed correctly ‘name’ ‘address\nmore address’ . But yeah very functional, and unlike most of the other answers, won’t just return one giant single line with all newlines replaced by spaces.Thanks! I don’t think this will fail for the case you are mentioning because qstart will not match the ‘name’ .
You’re right that the qstart regex won’t match name, because it only matches single quote at the beginning of a row followed by any character other than single quote until the end of the line. So my point was that the intention (I’m guessing of course) is probably that it should match the beginning of the address field and pull the rest of the address up to the same line, even when address is not the first field in the row. Ultimately this is probably better handled by the csv library or similar and opening with newlines=»
You may be right, my understanding is that the OP wants to read split quoted strings as a single line without worrying about this problem. I read the question again and can’t say I’m right or wrong 🙂
How to Read a File line by line in Python? (with code)
Python is a robust programming language with built-in support for reading and writing files. In this article, we’ll go over the various approaches to reading a text file line-by-line in Python. But first we have to revise some concepts about reading text files in general.
How to Read a .txt File in Python?
We need text files for many reasons and so, each coder should know how to operate them. We use the open() function to read a text file in Python. This function is used to open a file and returns a file object. It can have two arguments: the file name or path and the mode in which to open the file. Here is the simple syntax for this operation.
Here ‘example.txt’ is the file name that we want to open, and ‘r’ is the mode that signifies that we want to read the file.
The other modes available are as follows:
- Read Mode (‘r’): It simply reads the contents of a file.
- Write Mode (‘w’): If your file already exists, this will overwrite the content of your file. Otherwise creates a new file.
- Append Mode (‘a’): It appends the content in an already existing file instead of overwriting, If a file does not exist It creates a new file.
Now that we have a fundamental understanding of how to open and operate a file, Here is a simple example:
file = open('example.txt', 'r') read_=file.read() print(read_)
Python is a high-level, interpreted programming language that is popular among developers for its simplicity, readability, and flexibility. It was first released in 1991 by Guido van Rossum and has since become one of the most widely used programming languages in the world. Python is often used for web development, scientific computing, data analysis, artificial intelligence, and machine learning. One of the main advantages of Python is its large and active community, which has created a vast number of libraries and frameworks that make it easy to get started and build complex applications quickly.
A demo file called «example.txt» that contains some text about Python was created, and its contents were read using the read() function. As you can see, the output includes the entire content.
How to read a Text File line-by-line in Python?
Sometimes, reading a text file is not enough and we need to separate each line in the content of the file, so we have a great function by Python for this purpose.
We can read the file line by line in Python using a while loop and the readline() function. With the readlin() function called on the file, we read each single line of text, storing it in the line variable. We end the loop if the line variable is empty because that indicates that the file has ended. If not, the line is printed.
file = open('example.txt', 'r') while True: line = file.readline() if not line: break print(line)
Python is a high-level, interpreted programming language that is popular among developers for its simplicity, readability, and flexibility. It was first released in 1991 by Guido van Rossum and has since become one of the most widely used programming languages in the world. Python is often used for web development, scientific computing, data analysis, artificial intelligence, and machine learning. One of the main advantages of Python is its large and active community, which has created a vast number of libraries and frameworks that make it easy to get started and build complex applications quickly.
Please keep in mind that this function keeps track of its current position in the file. That’s why we need a loop to iterate over the file and read the next line of text from the file.
The key distinction between the two approaches is that the read() function reads the entire file in a single pass and disregards any available sentence spaces, whereas the readline() function reads a single line in a single pass, from the first character to the next available sentence space, prints it, and repeats the process.
However, you should also know about the readlines() method. With the help of this, we can quickly read every line in a text file while maintaining the sentence breaks. The readlines() functions reads every line and stores it in the variable lines in this case. Then, display each line after iterating through the lines using a «for» loop.
file = open('example.txt', 'r') lines = file.readlines() for line in lines: print(line)
Python is a high-level, interpreted programming language that is popular among developers for its simplicity, readability, and flexibility. It was first released in 1991 by Guido van Rossum and has since become one of the most widely used programming languages in the world. Python is often used for web development, scientific computing, data analysis, artificial intelligence, and machine learning. One of the main advantages of Python is its large and active community, which has created a vast number of libraries and frameworks that make it easy to get started and build complex applications quickly.
How it works is that the entire file is read into memory as a list of strings, which can be inefficient for large files. So, it is more memory-efficient to use a loop with the readline() method.
How to Read a File into a List?
What is actually happening here is that Python is maintaining a list with every line of the file’s content as an element, and then using the ‘for’ loop we iterate over the list to print individual lines. As you can see in the previous method, readlines() stores all the lines in a variable. Therefore, we can simply print the list variable if we want a list of the information. For Example:
file = open('example.txt', 'r') lines = file.readlines() print(lines)
['Python is a high-level, interpreted programming language that is popular among developers for its simplicity, readability, and flexibility. \n', 'It was first released in 1991 by Guido van Rossum and has since become one of the most widely used programming languages in the world.\n', 'Python is often used for web development, scientific computing, data analysis, artificial intelligence, and machine learning. \n', 'One of the main advantages of Python is its large and active community, which has created a vast number of libraries and frameworks that make it easy to get started and build complex applications quickly.']
There is an alternative method that uses the strip() function, as shown below:
with open('example.txt', 'r') as file: lines = [] for line in file: line = line.strip() lines.append(line) print(lines)
['Python is a high-level, interpreted programming language that is popular among developers for its simplicity, readability, and flexibility.', 'It was first released in 1991 by Guido van Rossum and has since become one of the most widely used programming languages in the world.', 'Python is often used for web development, scientific computing, data analysis, artificial intelligence, and machine learning.', 'One of the main advantages of Python is its large and active community, which has created a vast number of libraries and frameworks that make it easy to get started and build complex applications quickly.']
Using a «with» block, the code reads the contents of a file called «example.txt» in read-only mode (‘r’). The following step involves reading each line from the file, using the «strip()» method to eliminate any leading or trailing white space characters, and adding the resulting line to a list called «lines». The list of lines that were stripped is then printed.
Conclusion
It is very easy to read a line-by-line in Python using readline method that you learned above. If you have an assignment that is related to this problem, we provide assignment help in Python to assist you with that. Happy Learning:)