How to iterate through a file in Python

You will often need to iterate through a file in Python. This process lets you handle data line by line, a key skill for many common programming tasks.

In this article, you'll explore several iteration techniques, from basic loops to advanced methods. You'll get practical tips, see real-world applications, and learn to debug common errors with files.

Basic iteration with for loop

with open('example.txt', 'r') as file:
for line in file:
print(line.strip())--OUTPUT--Line 1: Hello, world!
Line 2: Python file handling
Line 3: Iterating through files
Line 4: End of file

The most Pythonic way to read a file is by directly iterating over the file object with a for loop. This works because the file object itself acts as an iterator, feeding you its contents one line at a time. This approach is memory-efficient since it doesn't load the entire file at once, making it ideal for handling large datasets.

Each line read from the file includes a trailing newline character. Calling line.strip() is a common practice to remove this whitespace, which prevents extra blank lines from appearing in the output.

Simple file reading techniques

While the for loop is a great default, methods like readline() and readlines() offer alternative ways to process files when you need different control.

Using readline() method in a while loop

file = open('example.txt', 'r')
line = file.readline()
while line:
print(line.strip())
line = file.readline()
file.close()--OUTPUT--Line 1: Hello, world!
Line 2: Python file handling
Line 3: Iterating through files
Line 4: End of file

The readline() method gives you more granular control by reading just one line at a time. You can pair it with a while loop to process a file sequentially. This setup continues as long as readline() returns a non-empty string.

• The loop condition while line: works because readline() returns an empty string ('') at the end of the file, which Python treats as False.
• It's crucial to call readline() again inside the loop to advance to the next line and prevent an infinite loop.
• Unlike using a with statement, this approach requires you to manually call file.close() to release the file from memory.

Reading all lines at once with readlines()

with open('example.txt', 'r') as file:
lines = file.readlines()
for line in lines:
print(line.strip())--OUTPUT--Line 1: Hello, world!
Line 2: Python file handling
Line 3: Iterating through files
Line 4: End of file

The readlines() method takes a different approach by reading the entire file into a list of strings. This can be handy if you need to access all the lines at once for operations that require the full dataset, like sorting.

• The primary drawback is memory usage. Loading a large file this way can consume significant resources, so it's best for smaller files.
• Since readlines() returns a standard Python list, you can easily manipulate it before you even start iterating.

Using list comprehension for file processing

with open('example.txt', 'r') as file:
lines = [line.strip() for line in file]
print(lines)--OUTPUT--['Line 1: Hello, world!', 'Line 2: Python file handling', 'Line 3: Iterating through files', 'Line 4: End of file']

List comprehensions offer a compact and readable way to create lists from an iterable. The expression [line.strip() for line in file] iterates through the file, applies strip() to each line, and builds a new list—all in a single, elegant line of code.

• It’s a more concise alternative to building a list with a standard for loop.
• Like the readlines() method, this approach loads the entire file into memory, so it's best suited for files that aren't excessively large.

Advanced file iteration approaches

Beyond the standard methods, you can use specialized tools like the fileinput module, custom generators, and itertools to handle more complex iteration scenarios.

Leveraging the fileinput module

import fileinput
for line in fileinput.input('example.txt'):
print(f"{fileinput.filename()}, line {fileinput.lineno()}: {line.strip()}")
fileinput.close()--OUTPUT--example.txt, line 1: Line 1: Hello, world!
example.txt, line 2: Line 2: Python file handling
example.txt, line 3: Line 3: Iterating through files
example.txt, line 4: Line 4: End of file

The fileinput module is a powerful tool for iterating over lines from one or more input files. It’s especially useful when you need context about where each line comes from. The fileinput.input() function creates an iterator that you can loop through, making it easy to process multiple files as if they were a single stream.

• The module provides helpful metadata, like the current file's name via fileinput.filename().
• You can also get the cumulative line number across all files using fileinput.lineno().

Just remember to call fileinput.close() when you're done to release the resources.

Creating custom generators for file iteration

def read_file_generator(filename):
with open(filename, 'r') as file:
for line in file:
yield line.strip()

for line in read_file_generator('example.txt'):
print(line)--OUTPUT--Line 1: Hello, world!
Line 2: Python file handling
Line 3: Iterating through files
Line 4: End of file

A custom generator offers a flexible way to handle file iteration. When a function uses the yield keyword, it doesn't just return a value—it becomes a special kind of iterator. It pauses execution and hands back one line at a time, picking up where it left off on the next loop cycle.

• This method is memory-efficient because it never loads the entire file at once.
• It also lets you package your file-processing logic, like calling strip(), into a clean, reusable function.

Memory-efficient iteration with itertools

import itertools
with open('example.txt', 'r') as file:
for line in itertools.islice(file, 2): # Read only first 2 lines
print(line.strip())--OUTPUT--Line 1: Hello, world!
Line 2: Python file handling

The itertools module is your go-to for advanced iterator operations. With itertools.islice(), you can take a slice of any iterable—like a file—without loading it all into memory. It creates a new iterator that yields only the items you need, making it extremely memory-efficient.

• Unlike list slicing, islice() doesn't read the entire file first. It processes just enough to satisfy the slice and then stops.
• In this example, islice(file, 2) instructs Python to read from the file object and stop after yielding two lines.

Move faster with Replit

Replit is an AI-powered development platform that transforms natural language into working applications. You can take the file iteration concepts from this article and use Replit Agent to build complete apps—with databases, APIs, and deployment—directly from a description.

For the file iteration techniques we've explored, such as using for loops or the fileinput module, Replit Agent can turn them into production-ready tools:

• Build a log file analyzer that reads server logs line by line to count errors or track user activity.
• Create a data migration script that processes large CSV files, transforms each row, and loads the results into a database.
• Deploy a configuration parser that reads settings from a text file to dynamically configure an application on startup.

Simply describe your application, and Replit Agent can write the code, run tests, and deploy it for you, all from your browser.

Common errors and challenges

Iterating through files can sometimes lead to errors, but knowing how to handle them will make your code much more robust.

Forgetting to close files when not using with statement

When you use open() to get a file object, you're responsible for calling file.close() when you're done. It's an easy step to forget, but it can lead to problems like resource leaks, where your program holds onto files unnecessarily. This is why using a with statement is strongly recommended. It automatically closes the file for you—even if your code runs into an error—making your script cleaner and more reliable.

Handling FileNotFoundError when opening files

It’s a classic scenario: your script tries to open a file that isn't there, and everything grinds to a halt with a FileNotFoundError. To prevent this crash, you can wrap your file operations in a try...except block. Place the code that might fail, like your open() call, inside the try block. Then, add an except FileNotFoundError: block to catch that specific error. Inside this block, you can handle the problem gracefully, perhaps by printing a message to the user or creating a default file.

Fixing character encoding issues when reading files

If you've ever opened a file and seen a mess of strange symbols instead of text, you've likely run into an encoding issue. This happens when a file is saved with one text encoding but read using another, which can cause a UnicodeDecodeError or simply display garbled characters. The fix is to be explicit. When you call open(), use the encoding argument to specify the correct format, like open('data.csv', 'r', encoding='utf-8'). UTF-8 is a safe bet for most text files today.

Forgetting to close files when not using with statement

It's easy to forget to call file.close() after you're done with a file you've opened manually. While your script might run without an obvious crash, leaving files open can lead to subtle bugs and resource issues. The code below shows what this looks like.

file = open('example.txt', 'r')
for line in file:
print(line.strip())
# Missing file.close() can lead to resource leaks

This code opens the file but never calls file.close(). The file object is left in memory, which can tie up system resources unnecessarily. The following example shows the proper way to manage the file's lifecycle.

file = open('example.txt', 'r')
for line in file:
print(line.strip())
file.close()

By adding file.close() after the loop, you manually release the file object from memory. This simple step is crucial for preventing resource leaks, especially in larger applications where many files are handled. You'll need to remember this whenever you use the basic open() function without wrapping it in a with statement, as the cleanup isn't automatic. This practice ensures your program remains efficient and stable.

Handling FileNotFoundError when opening files

When your code calls open() with a path to a file that doesn't exist, Python raises a FileNotFoundError and stops execution. This is a common issue that can crash your application if it's not handled. The following example demonstrates this scenario.

def read_data_file(filename):
file = open(filename, 'r')
data = file.read()
file.close()
return data

print(read_data_file('nonexistent_file.txt'))

Calling read_data_file() with 'nonexistent_file.txt' causes the open() function to fail because the file can't be found. This immediately halts the program. The following example shows how to manage this potential crash.

def read_data_file(filename):
try:
with open(filename, 'r') as file:
return file.read()
except FileNotFoundError:
return f"Error: File '{filename}' not found"

print(read_data_file('nonexistent_file.txt'))

By wrapping the file operation in a try...except block, you can gracefully handle a FileNotFoundError. The code attempts to open the file inside the try block. If the file doesn't exist, Python jumps to the except block instead of crashing. This lets you return a helpful error message or perform another action. It's a robust way to handle missing files, especially when dealing with user-provided file paths or configuration that might not be present.

Fixing character encoding issues when reading files

Reading a file with the wrong text encoding can lead to garbled output or a UnicodeDecodeError. This mismatch prevents Python from correctly interpreting the file's contents. The code below demonstrates what happens when you try to read a file without specifying the correct encoding.

with open('utf8_file.txt', 'r') as file:
content = file.read()
print(content)

The open() function defaults to the system's text encoding. When that differs from the file's actual format, Python misreads the data, leading to errors. See how to provide the correct encoding in the code below.

with open('utf8_file.txt', 'r', encoding='utf-8') as file:
content = file.read()
print(content)

By adding the encoding='utf-8' argument to the open() function, you explicitly tell Python how to decode the file. This ensures the text is read correctly, preventing garbled characters or a UnicodeDecodeError. It's a good practice to always specify the encoding, especially when working with files from different systems or the web, as their default formats can vary. UTF-8 is a widely used standard and often the right choice.

Real-world applications

Putting these techniques into practice, you can solve common problems like finding ERROR messages in logs or parsing data with the csv module.

Searching log files for ERROR messages

A common and practical use for file iteration is to scan through a log file, checking each line for a specific keyword like ERROR to identify and count issues.

with open('application.log', 'r') as log_file:
error_count = 0
for line in log_file:
if "ERROR" in line:
error_count += 1
print(line.strip())
print(f"Total errors found: {error_count}")

This script processes application.log line by line, which is a memory-efficient approach. It combines filtering and counting into one simple loop.

• It initializes an error_count to zero.
• For each line, it uses the in operator to check for the substring "ERROR".
• If a match is found, it increments the counter and prints the line.

After the loop finishes, a final summary of the total errors is displayed. This pattern is great for quickly summarizing data from large files without using much memory.

Processing CSV data with the csv module

The csv module is perfect for parsing structured data, as its DictReader object reads each row as a dictionary, letting you access columns by name.

import csv
from collections import defaultdict

sales_by_region = defaultdict(float)
with open('sales_data.csv', 'r') as csv_file:
reader = csv.DictReader(csv_file)
for row in reader:
region = row['Region']
sales = float(row['Sales'])
sales_by_region[region] += sales

for region, total in sorted(sales_by_region.items()):
print(f"{region}: ${total:.2f}")

This script tallies sales figures from a CSV file by region. It uses a defaultdict(float) to automatically initialize the sales total for any new region to zero, which simplifies the aggregation logic.

• The csv.DictReader object iterates through the file, mapping the information in each row to a dictionary.
• It accesses data by column headers like row['Region'] and row['Sales'].
• The script adds each sale to the correct region's running total.

Finally, it sorts the results alphabetically and prints the formatted totals.

Get started with Replit

Turn your new skills into a real tool. Give Replit Agent a prompt like “a script that scans log files for errors” or “an app that processes a CSV and calculates totals.”

It writes the code, tests for errors, and deploys your application from a single prompt. Start building with Replit and bring your ideas to life.