How to read a file line by line in Python

To read a file line by line in Python is a fundamental task for data processing and log analysis. Python’s built-in functions offer efficient methods to handle this common operation.

In this article, we'll explore several techniques for this task. You'll find practical tips, see real-world applications, and get debugging advice to help you confidently handle any text file.

Using a for loop to read lines

with open('sample.txt', 'r') as file:
for line in file:
print(line.strip())--OUTPUT--Hello
World
Python

The most Pythonic way to read a file is by iterating directly over the file object. This approach is highly memory-efficient because it processes the file one line at a time, rather than loading the entire contents into memory. The file object itself acts as an iterator, yielding each line on demand.

Using the with statement is crucial as it automatically handles closing the file once you're done, even if errors occur. The line.strip() method is called to remove leading and trailing whitespace, which includes the newline character at the end of each line, ensuring your output is clean.

Alternative file reading methods

Beyond the standard for loop, you can also read files using methods like readlines() and readline() or even a concise list comprehension for different situations.

Using the readlines() method

with open('sample.txt', 'r') as file:
lines = file.readlines()

for line in lines:
print(line.strip())--OUTPUT--Hello
World
Python

The readlines() method reads all lines from a file and returns them as a list of strings. Unlike iterating directly over the file, this approach loads the entire file's contents into memory at once.

• This is useful when you need the whole file in a list, for example, to access lines by index.
• The main trade-off is memory usage. It’s best to avoid readlines() for very large files to prevent performance issues.

Using readline() in a while loop

with open('sample.txt', 'r') as file:
line = file.readline()
while line:
print(line.strip())
line = file.readline()--OUTPUT--Hello
World
Python

The readline() method offers granular control by reading just one line at a time. The while line: loop continues as long as readline() successfully fetches a new line. When it reaches the end of the file, it returns an empty string, which evaluates to False and elegantly terminates the loop.

• This approach is memory-efficient, similar to the for loop, as it only processes a single line at a time.
• It's more verbose than iterating directly over the file object but can be useful in specific scenarios where you need to manually control the line-by-line reading process.

Using list comprehension

with open('sample.txt', 'r') as file:
lines = [line.strip() for line in file]

print(lines)--OUTPUT--['Hello', 'World', 'Python']

A list comprehension offers a concise syntax for creating a list. In this case, it iterates through the file, strips each line of whitespace, and collects the results into a new list—all in one line.

• It's a highly readable and Pythonic way to build a list from a file.
• Like the readlines() method, this approach loads the entire file into memory, so it's best for smaller files.

Advanced file reading techniques

As you move past the fundamentals, you'll find more powerful tools for handling errors, reading large files efficiently, and processing multiple files with the fileinput module.

Reading with error handling

try:
with open('sample.txt', 'r') as file:
for line in file:
print(line.strip())
except FileNotFoundError:
print("The file was not found.")
except IOError:
print("Error reading the file.")--OUTPUT--Hello
World
Python

Wrapping your file operations in a try...except block is essential for writing robust code. It lets you handle potential issues gracefully instead of letting your program crash. If an error occurs inside the try block, Python executes the code in the matching except block.

• The FileNotFoundError exception handles cases where the file doesn't exist.
• IOError is a broader exception for other input/output problems, such as not having permission to read the file.

Using generators for memory-efficient reading

def read_large_file(file_path):
with open(file_path, 'r') as file:
for line in file:
yield line.strip()

for line in read_large_file('sample.txt'):
print(line)--OUTPUT--Hello
World
Python

A generator function offers a powerful way to handle large files without consuming much memory. The yield keyword is what makes this work—it turns the read_large_file function into a generator that produces lines one by one, on demand.

• Unlike methods that load the entire file, this approach keeps memory usage low because it processes data lazily.
• Each time the for loop asks for a line, the generator runs just enough to yield the next one and then pauses. This makes it perfect for processing massive datasets that wouldn't fit into memory.

Using the fileinput module

import fileinput

for line in fileinput.input(files=('sample.txt')):
print(f"{fileinput.filename()}, line {fileinput.lineno()}: {line.strip()}")--OUTPUT--sample.txt, line 1: Hello
sample.txt, line 2: World
sample.txt, line 3: Python

The fileinput module is a handy tool for processing lines from multiple input sources, treating them as a single, continuous stream. It's particularly useful for creating scripts that can read from a series of files or standard input without complex logic.

• The fileinput.input() function creates an iterator that loops over the lines of all specified files.
• It also provides helpful metadata, allowing you to track the current file's name with fileinput.filename() and the cumulative line number with fileinput.lineno().

Move faster with Replit

Replit is an AI-powered development platform that lets you skip setup and start coding instantly. It comes with all Python dependencies pre-installed, so you don't have to worry about managing environments. This is where you can go from learning individual techniques, like reading a file line by line, to building complete applications with Agent 4.

Instead of piecing together code, you can describe the app you want to build and let the Agent take it from idea to working product. For example, you could build:

• A log analyzer that reads through server logs to isolate errors and generate a summary report.
• A data cleaning utility that processes raw text files, removing unwanted whitespace from each line to standardize formatting.
• A report consolidator that reads multiple daily sales files and combines them into a single, unified weekly document.

Simply describe your app, and Replit will write the code, test it, and fix issues automatically, all within your browser.

Common errors and challenges

Even with the right methods, you might run into a few common pitfalls, but most are straightforward to diagnose and resolve.

• Handling binary file reading errors: Attempting to read a non-text file, like an image or an executable, in text mode ('r') will cause problems. Python will try to decode bytes that don't represent text, often leading to a UnicodeDecodeError. The fix is to open the file in binary mode by adding a 'b' to the mode string, such as 'rb'.
• Resolving UnicodeDecodeError with proper encoding: This error also occurs when the text file's encoding doesn't match what Python expects—usually UTF-8 by default. To solve this, you must specify the correct encoding when opening the file using the encoding parameter, for example, open('data.csv', encoding='latin-1').
• Avoiding resource leaks with proper file closing: If a file isn't closed after being opened, it can lead to resource leaks that lock the file and consume memory. While you can call close() manually, the foolproof method is to use the with statement. It guarantees the file is closed automatically, even if errors occur.

Handling binary file reading errors

Trying to read a binary file, like an image, in standard text mode ('r') will cause a UnicodeDecodeError. This happens because Python can't decode the file's raw bytes into text characters. The code below shows this error in action.

with open('image.jpg', 'r') as file:
content = file.read()
print(content[:20]) # Will raise UnicodeDecodeError

The 'r' mode tells Python to read a text file, but image.jpg contains binary data, not text. This mismatch triggers the error. The following code demonstrates the proper way to handle such files.

with open('image.jpg', 'rb') as file:
content = file.read()
print(f"File size: {len(content)} bytes")

The solution is to open the file in binary mode using 'rb'. This tells Python to read the file as a sequence of raw bytes instead of trying to decode it as text. This method is essential for handling any non-text file—like images, executables, or audio clips—and completely avoids the UnicodeDecodeError. You'll get the raw data without any decoding issues, allowing you to work with the file's actual contents as intended.

Resolving UnicodeDecodeError with proper encoding

You'll also encounter a UnicodeDecodeError when a text file is saved with an encoding different from Python's default UTF-8. This mismatch confuses the interpreter, as it can't correctly read the characters. The code below shows what happens in this scenario.

with open('international.txt', 'r') as file:
content = file.read()
print(content) # Fails with non-UTF-8 encoded files

The code reads international.txt in text mode ('r'), which defaults to UTF-8. This assumption clashes with the file's actual encoding, triggering the error. The corrected approach is shown below.

with open('international.txt', 'r', encoding='utf-8') as file:
content = file.read()
print(content)

The fix is to explicitly tell Python which encoding to use. By adding the encoding='utf-8' parameter to the open() function, you specify how the file's bytes should be interpreted as text. This prevents the UnicodeDecodeError that occurs when a file's encoding doesn't match Python's default. You'll often encounter this when working with files created on different operating systems or from various data sources, so it's a good practice to be explicit.

Avoiding resource leaks with proper file closing

When you use open() but forget to explicitly close the file, you risk creating a resource leak. Your program continues to hold onto the file, which can lock it from other processes and consume memory. The following code demonstrates this common error.

file = open('data.txt', 'r')
content = file.read()
print(content)
# File never closed, potential resource leak

This code calls open() but never close(), so the file isn't released. This becomes a bigger problem if an error happens between the two operations. See the correct implementation in the code below.

with open('data.txt', 'r') as file:
content = file.read()
print(content)
# File automatically closed when exiting the with block

This code demonstrates the most reliable way to handle files in Python. The with statement creates a self-managing context that ensures the file is automatically closed once you're done with it. This happens whether the code inside the block finishes successfully or raises an error. By using this structure, you completely avoid the risk of resource leaks caused by unclosed files, making your program more stable and efficient. It's the standard you should always follow.

Real-world applications

Now that you've mastered the techniques, you can apply them to real-world tasks like processing CSV files and analyzing text.

Processing a CSV file with csv module

Python's csv module is designed specifically for tabular data, making it easy to parse each row into a more usable format like a list or dictionary.

import csv

with open('employees.csv', 'r') as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
if int(row['salary']) > 50000:
print(f"{row['name']} - ${row['salary']}")

This code snippet leverages Python's csv module to efficiently parse structured data. Instead of treating each line as a simple string, csv.DictReader transforms every row into a dictionary. This makes your code more readable and robust, as you can access data by column names like row['name'] instead of relying on fragile index positions.

• The loop iterates through each employee record.
• It checks if the salary is above 50,000 before printing the employee's details.

Text analysis with word frequency counting

Another practical application is text analysis, where you can read a file to count word frequencies and uncover key themes.

from collections import Counter

def count_word_frequency(filename):
with open(filename, 'r') as file:
text = file.read().lower()
words = text.split()
return Counter(words).most_common(5)

print(count_word_frequency('article.txt'))

This code leverages the Counter object from Python's collections module. It's a specialized dictionary subclass designed for counting hashable objects, making it perfect for this kind of text processing.

• The function first reads the file's content, converts it to lowercase with lower(), and breaks it into a list of words using split().
• This list is then passed to Counter, which automatically tallies every word.
• The most_common(5) method conveniently extracts the five most frequent words and their counts.

Get started with Replit

Now, turn your knowledge into a real tool with Replit Agent. Describe what you want to build, like “a log analyzer that filters for error lines” or “a utility that converts a CSV to JSON.”

Replit Agent writes the code, tests for errors, and deploys the app, handling the entire development lifecycle. Start building with Replit.