How to profile memory usage in Python

To optimize Python applications, you must know how to profile memory. This process helps you find inefficiencies and prevent crashes, which ensures your code runs smoothly and scales effectively.

In this article, you'll explore powerful techniques and practical tips for memory analysis. You will see real-world applications and get actionable advice to debug common memory issues for more robust programs.

Basic memory usage with psutil

import psutil

def check_memory():
process = psutil.Process()
memory_info = process.memory_info()
return memory_info.rss / (1024 * 1024) # Convert to MB

print(f"Current memory usage: {check_memory():.2f} MB")--OUTPUT--Current memory usage: 42.35 MB

The psutil library offers a direct way to inspect your application's memory usage. The check_memory() function zeroes in on a key metric obtained from the process.memory_info() method.

• Resident Set Size (RSS): Accessed via the rss attribute, this tells you how much non-swapped physical memory your process is currently using. It's a crucial indicator of your app's real-world memory footprint.

The code then converts this value from bytes to megabytes for readability. This approach is great for getting a quick, on-the-spot measurement of memory consumption.

Basic memory profiling techniques

To go beyond the high-level overview that psutil provides, you can use more granular techniques to pinpoint exactly where your memory is being spent.

Using the memory_profiler module

# pip install memory_profiler
from memory_profiler import profile

@profile
def memory_heavy_function():
large_list = [i for i in range(1000000)]
return sum(large_list)

result = memory_heavy_function()--OUTPUT--Line # Mem usage Increment Occurences Line Contents
============================================================
4 42.3 MiB 42.3 MiB 1 @profile
5 def memory_heavy_function():
6 80.5 MiB 38.2 MiB 1 large_list = [i for i in range(1000000)]
7 80.5 MiB 0.0 MiB 1 return sum(large_list)

The memory_profiler module offers a line-by-line analysis. By adding the @profile decorator to a function, you get a detailed report when it runs. This output breaks down memory usage for each line, making it easy to spot inefficiencies.

• The Increment column is especially useful. It shows exactly how much memory each operation adds, pinpointing the list comprehension as the source of the 38.2 MiB increase in this example.

Tracking allocations with tracemalloc

import tracemalloc

tracemalloc.start()
large_dict = {i: i * 2 for i in range(100000)}
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

print(f"Current: {current / 10**6:.1f} MB; Peak: {peak / 10**6:.1f} MB")--OUTPUT--Current: 8.3 MB; Peak: 12.7 MB

Python's built-in tracemalloc module offers a precise way to track memory allocations. You wrap the code you want to analyze between tracemalloc.start() and tracemalloc.stop() to isolate its memory footprint. The get_traced_memory() function then provides two key figures:

• Current usage: The memory your code is using at that moment.
• Peak usage: The maximum memory consumed during the tracked period, which is vital for identifying memory spikes.

Measuring object size with sys.getsizeof()

import sys

data_types = [1, 1.0, "a", [], {}, set(), tuple()]
for item in data_types:
print(f"{type(item).__name__}: {sys.getsizeof(item)} bytes")--OUTPUT--int: 28 bytes
float: 24 bytes
str: 50 bytes
list: 56 bytes
dict: 232 bytes
set: 216 bytes
tuple: 40 bytes

The sys.getsizeof() function offers a straightforward way to measure an object's memory consumption in bytes. It's a useful tool for understanding the base memory cost of different data structures, as shown in the example. This helps you make informed decisions when choosing which data type to use.

• Keep in mind that for container types like a list or dict, this function only measures the size of the container object itself—not the memory consumed by the items it references.

Advanced memory profiling techniques

When basic tools fall short, you can visualize object graphs with objgraph, build custom profilers, or perform deep analysis with pympler for more insight.

Visualizing object references with objgraph

# pip install objgraph
import objgraph

x = []
y = [x, [x], dict(x=x)]
objgraph.show_refs([y], filename='sample-graph.png')
print(f"List references: {objgraph.count('list')}")--OUTPUT--List references: 3
Graph written to sample-graph.png

The objgraph library helps you hunt down memory leaks by creating visual maps of how objects are connected. It’s especially useful for untangling complex reference cycles where objects keep each other alive unnecessarily.

• The show_refs() function generates a diagram, like sample-graph.png, showing how the list y holds multiple references to the same list x.
• You can also use objgraph.count() to quickly tally all instances of a specific object type, which helps you confirm if objects are being created or destroyed as you expect.

Creating a custom memory profiler decorator

import psutil
import time
from functools import wraps

def memory_profiler(func):
@wraps(func)
def wrapper(*args, **kwargs):
process = psutil.Process()
memory_before = process.memory_info().rss / 1024 / 1024
start = time.time()
result = func(*args, **kwargs)
elapsed = time.time() - start
memory_after = process.memory_info().rss / 1024 / 1024
print(f"Function: {func.__name__}, Memory change: {memory_after - memory_before:.2f} MB, Time: {elapsed:.2f} sec")
return result
return wrapper--OUTPUT--# When applied to a function with @memory_profiler
Function: create_large_list, Memory change: 38.25 MB, Time: 0.15 sec

For targeted analysis, you can build your own lightweight profiler using a decorator. This approach combines psutil and time to measure both memory consumption and execution speed. When you apply the @memory_profiler decorator to a function, it automatically tracks its performance.

• The decorator records memory usage before and after the function runs, then prints the difference.
• It also times the function's execution, giving you a complete performance snapshot in one go.
• Using @wraps is a key detail; it preserves the original function's metadata, like its name.

Deep memory analysis with pympler

# pip install pympler
from pympler import asizeof, tracker

tr = tracker.SummaryTracker()
large_list = [1] * 1000000
large_dict = {i: i for i in range(10000)}
print(f"List size: {asizeof.asizeof(large_list) / 1024 / 1024:.2f} MB")
tr.print_diff()--OUTPUT--List size: 8.01 MB
types | # objects | total size
============================ | =========== | ============
int | 1010001 | 23.08 MB
dict | 1 | 0.59 MB
list | 1 | 8.01 MB
cell | 1 | 104 B

The pympler library provides a more thorough analysis than built-in tools. Its asizeof.asizeof() function recursively calculates an object's total memory footprint, including all the items it references. This gives you a far more accurate size measurement for containers like lists and dictionaries.

• The tracker.SummaryTracker() captures a snapshot of all existing objects when initialized.
• Calling tr.print_diff() then reveals a summary of all new objects created since that snapshot, grouped by type. This makes it easy to see exactly what your code is allocating.

Move faster with Replit

Replit is an AI-powered development platform that transforms natural language into working applications. With Replit Agent, you can describe what you want to build, and it creates the app for you—complete with databases, APIs, and deployment.

You can use it to turn the memory profiling techniques from this article into production-ready tools:

• Build a resource monitoring dashboard that uses psutil to display an application's real-time memory usage.
• Create a memory leak debugger that visualizes object references with objgraph to help find circular dependencies.
• Deploy a code analysis tool that provides a line-by-line memory breakdown for Python functions, using concepts from memory_profiler.

Try building your next tool with Replit Agent and see how quickly you can go from concept to a deployed application.

Common errors and challenges

Effectively managing memory means sidestepping common errors, like inefficient data handling and sneaky leaks that can degrade performance.

Forgetting to clear data in memory-intensive operations

It's a common mistake to let large data structures linger in memory after you're done with them. This often happens inside functions, where an object isn't cleared after its purpose is served, tying up resources. The code below shows this in action.

def process_large_dataset():
large_list = [i * i for i in range(1000000)]
result = sum(large_list)
# Data remains in memory until function returns
return result

process_large_dataset()
# Memory still high here even though we don't need large_list anymore

The large_list variable consumes memory for the entire duration of the process_large_dataset() function, even though it's no longer needed after the sum is calculated. The following code demonstrates a more efficient approach.

def process_large_dataset():
large_list = [i * i for i in range(1000000)]
result = sum(large_list)
del large_list # Explicitly release memory
return result

process_large_dataset()
# Memory usage drops more quickly

The improved version uses del large_list to immediately free up memory. While Python's garbage collector would eventually clear the object, being explicit is more efficient and gives you more control.

• This prevents the large list from consuming resources until the function finishes.
• It's a crucial practice in long-running processes or loops where temporary, large objects are created repeatedly. By deleting them manually, you ensure your application's memory footprint stays lean.

Optimizing memory usage with batch processing

Loading an entire dataset into memory at once is a common pitfall that can exhaust your system's resources and cause crashes. Batch processing avoids this by handling data in smaller, manageable chunks. The following code demonstrates this problem in action.

import numpy as np
import psutil

def get_memory_mb():
return psutil.Process().memory_info().rss / (1024 * 1024)

# Approach 1: Process all data at once
print(f"Before loading all data: {get_memory_mb():.2f} MB")
all_data = np.random.random(10000000) # 10 million elements
print(f"After loading all data: {get_memory_mb():.2f} MB")

Creating the all_data array in one go consumes a large chunk of memory instantly, which is risky for large datasets. The following code demonstrates a more memory-conscious strategy for handling the same operation.

import numpy as np
import psutil

def get_memory_mb():
return psutil.Process().memory_info().rss / (1024 * 1024)

# Approach 2: Process in batches
print(f"Before batch processing: {get_memory_mb():.2f} MB")
total_sum = 0
for i in range(10): # 10 batches of 1 million each
batch = np.random.random(1000000) # 1 million elements per batch
total_sum += np.sum(batch)
del batch # Release memory explicitly
print(f"After batch processing: {get_memory_mb():.2f} MB")

The second approach avoids memory spikes by processing data in smaller chunks inside a for loop. This keeps your application's memory footprint low and stable.

• Each batch is created, processed, and then immediately discarded with del batch to free up resources.
• This technique is essential when working with large files, database queries, or any data stream that's too big to fit into memory at once.

Implementing a simple memory leak detection system

Memory leaks occur when objects aren't released, causing memory usage to grow over time. A simple detection system helps you catch these issues by tracking memory usage and flagging unexpected increases, which prevents performance degradation and keeps your application stable.

The code below implements a basic MemoryLeakDetector class. It works by establishing a baseline memory reading and then periodically checking for significant growth—a classic sign of a leak.

import time
import psutil

class MemoryLeakDetector:
def __init__(self):
self.baseline = psutil.Process().memory_info().rss / (1024 * 1024)
print(f"Baseline memory: {self.baseline:.2f} MB")

def check(self, threshold_mb=1.0):
current = psutil.Process().memory_info().rss / (1024 * 1024)
increase = current - self.baseline
print(f"Current memory: {current:.2f} MB (Δ: {increase:.2f} MB)")
if increase > threshold_mb:
print("Warning: Possible memory leak detected!")

# Simulate an application with a memory leak
detector = MemoryLeakDetector()
leaky_list = []

for i in range(3):
leaky_list.extend([0] * 100000) # Leak some memory
time.sleep(1)
detector.check()

The leaky_list is never cleared, so its size increases with each loop iteration. This steady accumulation is what the detector flags as a potential leak. The following code demonstrates how a small adjustment can correct this behavior.

import time
import psutil

class MemoryLeakDetector:
def __init__(self):
self.baseline = psutil.Process().memory_info().rss / (1024 * 1024)
print(f"Baseline memory: {self.baseline:.2f} MB")

def check(self, threshold_mb=1.0):
current = psutil.Process().memory_info().rss / (1024 * 1024)
increase = current - self.baseline
print(f"Current memory: {current:.2f} MB (Δ: {increase:.2f} MB)")
if increase > threshold_mb:
print("Warning: Possible memory leak detected!")
return True
return False

# Simulate an application with leak detection and fix
detector = MemoryLeakDetector()
leaky_list = []

for i in range(3):
leaky_list.extend([0] * 100000) # Leak some memory
time.sleep(1)
if detector.check(threshold_mb=1.0) and i == 2:
leaky_list = [] # Fix the leak by clearing the list
print("Leak fixed by clearing the list")

The corrected code demonstrates how to resolve the leak. After the detector.check() method flags the issue, the code resets the list by assigning leaky_list = []. This simple reassignment releases the memory held by the old list, effectively plugging the leak.

• This is a common problem in long-running applications or loops where data is continuously appended but never cleared.
• Keep an eye out for it in background services or data processing pipelines.

Real-world applications

Putting these techniques into practice is key for real-world challenges, like processing massive datasets or preventing leaks in long-running services.

Optimizing memory usage with batch processing

When working with large numerical datasets, such as those common in data analysis, batch processing is a crucial technique for keeping memory consumption under control.

import numpy as np
import psutil

def get_memory_mb():
return psutil.Process().memory_info().rss / (1024 * 1024)

# Approach 1: Process all data at once
print(f"Before loading all data: {get_memory_mb():.2f} MB")
all_data = np.random.random(10000000) # 10 million elements
print(f"After loading all data: {get_memory_mb():.2f} MB")

# Approach 2: Process in batches
print(f"Before batch processing: {get_memory_mb():.2f} MB")
for i in range(10): # 10 batches of 1 million each
batch = np.random.random(1000000) # 1 million elements per batch
print(f"After batch processing: {get_memory_mb():.2f} MB")

This code contrasts two data processing strategies to show their impact on memory. The first approach allocates memory for all 10 million elements at once, demonstrating the high upfront cost of loading a full dataset into the all_data array.

• The second approach iterates ten times, creating a smaller one-million-element batch in each loop.
• Because the batch variable is reassigned in every iteration, the old array becomes eligible for garbage collection. This keeps peak memory usage significantly lower and shows how iterative processing effectively manages resources.

Implementing a simple memory leak detection system

You can use a simple detection system in long-running services to catch gradual memory leaks that might otherwise go unnoticed until they cause performance issues or crashes.

import time
import psutil

class MemoryLeakDetector:
def __init__(self):
self.baseline = psutil.Process().memory_info().rss / (1024 * 1024)
print(f"Baseline memory: {self.baseline:.2f} MB")

def check(self, threshold_mb=1.0):
current = psutil.Process().memory_info().rss / (1024 * 1024)
increase = current - self.baseline
print(f"Current memory: {current:.2f} MB (Δ: {increase:.2f} MB)")
if increase > threshold_mb:
print("Warning: Possible memory leak detected!")

# Simulate an application with a memory leak
detector = MemoryLeakDetector()
leaky_list = []

for i in range(3):
leaky_list.extend([0] * 100000) # Leak some memory
time.sleep(1)
detector.check()

The MemoryLeakDetector class provides a straightforward way to monitor memory. It captures a baseline memory reading when an object is created. You can then call its check() method periodically to see how much memory has increased since that initial snapshot.

• The example loop simulates a scenario where memory usage grows over time by appending to leaky_list.
• Each call to detector.check() reports the current memory and the change from the baseline, issuing a warning if it exceeds the threshold.

Get started with Replit

Turn these techniques into a real application with Replit Agent. Try prompts like, “Build a dashboard to track memory with psutil,” or “Create a tool that visualizes object references to find memory leaks.”

The agent writes the code, tests for errors, and deploys the app, turning your description into a finished product. Start building with Replit.