How to convert a list of dictionaries to a dataframe in Python

A common data manipulation task in Python is to convert a list of dictionaries into a pandas DataFrame. This operation helps structure complex data for analysis and visualization.

In this article, you'll learn several techniques for this conversion. You'll find practical tips, see real-world applications, and get advice to debug common issues, so you can choose the best method.

Using pd.DataFrame() to convert a list of dictionaries

import pandas as pd

data = [{'name': 'John', 'age': 30}, {'name': 'Alice', 'age': 25}]
df = pd.DataFrame(data)
print(df)--OUTPUT--name age
0 John 30
1 Alice 25

The pd.DataFrame() constructor is the most direct way to handle this conversion. It works because pandas is built to intuitively understand data structures. The function automatically uses the dictionary keys—like 'name' and 'age'—as column headers and then fills the rows with the corresponding values from each dictionary.

This method isn't just simple; it's also highly efficient. The constructor's underlying logic is optimized for this process, making it a fast choice even for large datasets. It's the go-to approach for its excellent balance of simplicity and performance.

Handling different dictionary structures

Although the standard pd.DataFrame() constructor is powerful, you'll often encounter dictionaries with inconsistent structures, which require more specialized techniques to handle correctly.

Working with dictionaries that have different keys

import pandas as pd

data = [{'name': 'John', 'age': 30}, {'name': 'Alice', 'age': 25, 'city': 'New York'}]
df = pd.DataFrame(data)
print(df)--OUTPUT--name age city
0 John 30 NaN
1 Alice 25 New York

When your dictionaries have different keys, the pd.DataFrame() constructor adapts gracefully. It gathers all unique keys from every dictionary in the list to form the columns of the new DataFrame.

• If a dictionary is missing a key that exists in another, pandas automatically fills the corresponding cell with NaN (Not a Number).
• This ensures your DataFrame remains rectangular and consistent, even when the input data isn't uniform.

In the example, the first dictionary lacks a 'city' key, so its value in the city column becomes NaN.

Converting nested dictionaries with json_normalize()

import pandas as pd

nested_data = [{'name': 'John', 'details': {'age': 30, 'job': 'engineer'}},
{'name': 'Alice', 'details': {'age': 25, 'job': 'doctor'}}]
df = pd.json_normalize(nested_data)
print(df)--OUTPUT--name details.age details.job
0 John 30 engineer
1 Alice 25 doctor

When your data contains nested dictionaries, the standard constructor isn't the right tool. You'll need pd.json_normalize(), which is specifically designed to flatten complex, semi-structured data into a clean, tabular format.

• It intelligently unpacks nested objects from each dictionary in your list.
• New columns are created by joining the parent key with the nested key, using a dot as a separator—for example, details.age.

Creating a dataframe from specific dictionary keys

import pandas as pd

data = [{'name': 'John', 'age': 30, 'scores': [85, 90]},
{'name': 'Alice', 'age': 25, 'scores': [95, 88]}]
df = pd.DataFrame(data, columns=['name', 'age'])
print(df)--OUTPUT--name age
0 John 30
1 Alice 25

If you only need a subset of the data from your dictionaries, you can specify which keys to include. Simply pass a list of your desired column names to the columns parameter in the pd.DataFrame() constructor. This tells pandas to create the DataFrame using only those keys.

• It's a great way to filter out unnecessary data—like the 'scores' key in the example.
• It also lets you enforce a specific column order, regardless of how the keys are arranged in the original dictionaries.

Advanced techniques

Now that you can handle various dictionary structures, you can move on to advanced techniques for optimizing performance and customizing your final DataFrame.

Creating a dataframe from dictionary of lists for better performance

import pandas as pd

# More efficient for large datasets
names = ['John', 'Alice', 'Bob']
ages = [30, 25, 45]
df = pd.DataFrame({'name': names, 'age': ages})
print(df)--OUTPUT--name age
0 John 30
1 Alice 25
2 Bob 45

Instead of a list of dictionaries, you can structure your data as a single dictionary of lists. This approach is often more performant, especially with large datasets, because it aligns better with how pandas stores data internally.

• Each key in the dictionary, like 'name' or 'age', becomes a column.
• The value for each key is a list containing all the data for that column.

When you pass this dictionary to pd.DataFrame(), pandas can create the columns directly from the lists. This is more memory-efficient than processing a list of many small dictionary objects one by one.

Customizing your dataframe with additional transformations

import pandas as pd

data = [{'name': 'John', 'age': 30}, {'name': 'Alice', 'age': 25}]
df = pd.DataFrame(data)
df['adult'] = df['age'] >= 18
df['id'] = ['ID-' + str(i) for i in range(len(df))]
print(df)--OUTPUT--name age adult id
0 John 30 True ID-0
1 Alice 25 True ID-1

After creating your DataFrame, you aren't stuck with the original structure. You can easily add new columns to enrich your data with calculated or generated values, which is a common step in data preparation.

• A new adult column is created by applying a boolean check—df['age'] >= 18—to the existing age column.
• An id column is also added, generated using a list comprehension to assign a unique identifier to each row.

These transformations show how you can customize your DataFrame on the fly, tailoring it to your specific analysis needs.

Using custom indexing with set_index()

import pandas as pd

data = [{'name': 'John', 'age': 30, 'city': 'New York'},
{'name': 'Alice', 'age': 25, 'city': 'Boston'}]
df = pd.DataFrame(data).set_index(['city', 'name'])
print(df)--OUTPUT--age
city name
New York John 30
Boston Alice 25

Instead of relying on the default integer index, you can assign a more meaningful one using set_index(). This method converts one or more columns into the DataFrame's index, which makes data lookups more intuitive.

• The example uses set_index(['city', 'name']) to create a hierarchical index, or MultiIndex, from two columns.
• This organizes the data so you can access rows using city and name labels, rather than just a numerical position. It’s a powerful way to structure data for analysis.

Move faster with Replit

Replit is an AI-powered development platform that transforms natural language into working applications. You can describe what you want to build, and Replit Agent creates it—complete with databases, APIs, and deployment.

Replit Agent can take the concepts from this article and turn them into fully functional applications. For example, you could build:

• A data ingestion tool that pulls a list of dictionaries from a JSON API, converts it to a DataFrame, and displays it in a clean table.
• A data cleaning utility that processes files with inconsistent dictionary keys, filling in missing values with NaN before exporting to a structured format.
• An analytics dashboard that flattens nested dictionary data using a method like json_normalize() to prepare it for visualization.

Describe your app idea, and Replit Agent will write the code, test it, and fix issues automatically, all within your browser.

Common errors and challenges

While the conversion process is often smooth, you might hit a few common roadblocks that require more targeted solutions.

Although pandas automatically inserts NaN for missing keys, you might want to set a different default value, like 0 or an empty string. You can handle this before creating the DataFrame by using Python’s built-in .get() method. By iterating through your list and using .get() on each dictionary, you can pre-fill missing values, ensuring your data is clean from the start.

Another frequent issue is inconsistent data types within your values—for example, an age key might have an integer in one dictionary and a string in another. When this happens, pandas will assign the column an object data type, which prevents you from performing mathematical calculations on it.

• To fix this, you can clean the data after creating the DataFrame.
• Use functions like pd.to_numeric() to convert a column to a numerical type. This function can also automatically turn any values that can't be converted into NaN, making your dataset consistent.

A common mistake is using the standard pd.DataFrame() constructor on a list of nested dictionaries. This won't flatten the data; instead, you'll get a column containing dictionary objects, which isn't very useful for analysis.

As mentioned earlier, the correct tool for this job is pd.json_normalize(). It’s designed to unpack these nested structures into a flat, usable table. Forgetting to use it is a frequent misstep that leads to a poorly formed DataFrame that you can't easily work with.

Handling missing keys with the .get() method

While the default pd.DataFrame() constructor handles missing keys by inserting NaN, you'll run into a KeyError if you try to access a non-existent key directly. This often happens when you're using a list comprehension. The following code demonstrates this exact problem.

import pandas as pd

# Buggy code - will raise KeyError
data = [{'name': 'John', 'age': 30}, {'name': 'Alice'}]
df = pd.DataFrame([{'name': d['name'], 'age': d['age']} for d in data])
print(df)

The list comprehension directly accesses the 'age' key, but it's missing from the second dictionary, causing a KeyError. The corrected code below shows how to handle this safely.

import pandas as pd

# Fixed code - use get() with a default value
data = [{'name': 'John', 'age': 30}, {'name': 'Alice'}]
df = pd.DataFrame([{'name': d['name'], 'age': d.get('age', None)} for d in data])
print(df)

The corrected code avoids a KeyError by using the dictionary’s .get() method. Instead of directly accessing d['age'], which fails if the key is missing, d.get('age', None) provides a default value. This is a robust way to handle dictionaries with inconsistent keys, especially within a list comprehension. It ensures your code runs without crashing and allows pandas to correctly insert None for the missing values.

Dealing with type inconsistencies in dictionary values

It's common for dictionary values to have inconsistent types, like an age key holding both integers and strings. When this happens, pandas defaults the column to an object data type, which prevents you from performing mathematical operations. The following code fails because of this.

import pandas as pd

# Buggy code - mixed types prevent calculations
data = [{'name': 'John', 'age': 30}, {'name': 'Alice', 'age': '25'}]
df = pd.DataFrame(data)
average_age = df['age'].mean() # TypeError: string and int
print(f"Average age: {average_age}")

The code fails because the age column contains both an integer and a string. You can't calculate an average on mixed data types, so calling .mean() triggers a TypeError. The corrected code below shows how to fix this.

import pandas as pd

# Fixed code - convert strings to appropriate types
data = [{'name': 'John', 'age': 30}, {'name': 'Alice', 'age': '25'}]
df = pd.DataFrame(data)
df['age'] = pd.to_numeric(df['age'])
average_age = df['age'].mean()
print(f"Average age: {average_age}")

The corrected code uses pd.to_numeric() to standardize the age column's data type. This function converts number-like strings into actual numbers, resolving the TypeError and allowing mathematical functions like .mean() to work correctly. You should watch for this issue when importing data from external sources, such as APIs or text files, where numeric values are often stored as strings.

Correctly extracting data from nested dictionary structures

You can't access inner values directly from a DataFrame created with nested dictionaries. The constructor makes a column of dictionary objects, so chaining key lookups like df['user']['details']['age'] will fail. The following code demonstrates this common pitfall.

import pandas as pd

# Buggy code - can't access nested data this way
nested_data = [
{'user': {'name': 'John', 'details': {'age': 30}}},
{'user': {'name': 'Alice', 'details': {'age': 25}}}
]
df = pd.DataFrame(nested_data)
print(df['user']['details']['age']) # This fails

The chained lookup df['user']['details'] fails because it's trying to access a key on an entire pandas Series, not a single dictionary. This approach isn't built for nested data. The code below shows how to do it right.

import pandas as pd

# Fixed code - properly flatten nested dictionaries
nested_data = [
{'user': {'name': 'John', 'details': {'age': 30}}},
{'user': {'name': 'Alice', 'details': {'age': 25}}}
]
df = pd.json_normalize(nested_data)
print(df['user.details.age']) # Correctly access nested fields

The corrected code uses pd.json_normalize(), which is designed to flatten nested data structures. It unpacks the inner dictionaries, creating new columns by joining the keys with a dot—like user.details.age.

This makes every piece of data directly accessible in a clean, tabular format. You'll find this function essential when working with complex data from sources like JSON APIs, where nested objects are common.

Real-world applications

Now that you've mastered these conversion methods, you can apply them to real-world challenges like processing API data and analyzing purchase records.

Processing weather API data with pd.DataFrame()

When you pull data from a weather API, it often arrives as a list of dictionaries, which you can easily convert into a DataFrame for analysis.

import pandas as pd

# Weather data from multiple cities (simulating API response)
weather_data = [
{'city': 'New York', 'temperature': 72, 'conditions': 'Sunny', 'humidity': 50},
{'city': 'Chicago', 'temperature': 65, 'conditions': 'Cloudy', 'humidity': 60},
{'city': 'Miami', 'temperature': 85, 'conditions': 'Sunny', 'humidity': 75}
]

weather_df = pd.DataFrame(weather_data)
warm_cities = weather_df[weather_df['temperature'] > 70]
print(warm_cities)

This example shows how easily you can analyze API-like data once it's in a DataFrame. The code first converts the list of dictionaries into a weather_df DataFrame, organizing the raw data into a clean table.

The most important step is the filtering. It uses boolean indexing to select specific rows:

• The expression weather_df['temperature'] > 70 evaluates each row, creating a series of True or False values.
• When you pass this series back into the DataFrame, it returns only the rows where the condition was True, effectively filtering for warm cities.

Analyzing purchase data using json_normalize() and aggregation

When you're dealing with nested purchase records, you can use pd.json_normalize() to flatten the data and then perform aggregations to calculate key metrics like total spending per customer.

import pandas as pd

# Customer purchase data with nested information
purchases = [
{'customer_id': 101, 'name': 'John', 'purchase': {'product': 'Laptop', 'price': 1200, 'quantity': 1}},
{'customer_id': 102, 'name': 'Alice', 'purchase': {'product': 'Phone', 'price': 800, 'quantity': 1}},
{'customer_id': 101, 'name': 'John', 'purchase': {'product': 'Headphones', 'price': 100, 'quantity': 2}}
]

# Normalize the nested data
purchases_df = pd.json_normalize(purchases)

# Calculate total spent per item
purchases_df['total'] = purchases_df['purchase.price'] * purchases_df['purchase.quantity']

# Group by customer and sum totals
customer_totals = purchases_df.groupby(['customer_id', 'name'])['total'].sum().reset_index()
print(customer_totals)

This example demonstrates a common data analysis workflow. First, pd.json_normalize() unpacks the nested purchase dictionary into separate columns like purchase.price. A new total column is then created by multiplying the price and quantity for each row.

• The groupby(['customer_id', 'name']) method groups all transactions belonging to the same person.
• Next, .sum() calculates the total spending for each customer group.
• Finally, .reset_index() converts the grouped output back into a clean DataFrame, making the aggregated data easy to read.

Get started with Replit

Put these concepts into practice by building a tool. Tell Replit Agent: “Build a utility that converts a JSON file of user data to a CSV” or “Create a dashboard that flattens API data and displays it.”

The agent writes the code, tests for errors, and deploys your app directly from your browser. Start building with Replit.