Remove duplicates from List of dictionaries by keys in Python

This tutorial will discuss how to remove duplicates from list of dictionaries by keys in Python.

Suppose we have a list of dictionaries, and within this list, there might be duplicate dictionaries.

# A List of Dictionaries
sampleList = [
    {"id": 44, "name": "Ritika"},
    {"id": 55, "name": "Sanjay"},
    {"id": 44, "name": "Ankit"},
    {"id": 33, "name": "Johny"}]

To determine if a dictionary is a duplicate, we’ll use a specific key for comparison. If two dictionaries have the same value for this key, they are considered duplicates. For instance, in a given list, the first and third dictionaries might be duplicates if they share the same value for the key ‘ID’ i.e.

# Duplicate dictionaries because
# key 'id' has same value in both
{"id": 44, "name": "Ritika"},
{"id": 44, "name": "Ankit"},

To remove such duplicate dictionaries from a list, we’ve created a function removeDuplicatesByKey(). This function takes in two arguments: the list of dictionaries as the first argument and the key for comparison as the second. The function then iterates through each dictionary in the list. For each dictionary, it checks if the value for the specified key has appeared in any previous dictionaries. If it has, the dictionary is deemed a duplicate and is excluded from the new list. The function ultimately returns a new list containing only unique dictionaries.

Let’s see the complete example,

def removeDuplicatesByKey(dictList, key):
    # Set to track keys we've seen
    seenKeys = set()
    # List to store dictionaries without duplicates
    uniqueList = []

    for d in dictList:
        # Check if the key value is already seen
        if d[key] not in seenKeys:
            seenKeys.add(d[key])
            uniqueList.append(d)

    return uniqueList

# A List of Dictionaries
sampleList = [
    {"id": 44, "name": "Ritika"},
    {"id": 55, "name": "Sanjay"},
    {"id": 44, "name": "Ankit"},
    {"id": 33, "name": "Johny"}
]

print("Original List:", sampleList)

keyToCheck = "id"

# Remove duplicate dictionaries from List of dictionaries
# based on provided key
sampleList = removeDuplicatesByKey(sampleList, keyToCheck)

print("List after removing duplicates based on key", keyToCheck, ":")
print(sampleList)

Output

Original List: [{'id': 44, 'name': 'Ritika'}, {'id': 55, 'name': 'Sanjay'}, {'id': 44, 'name': 'Ankit'}, {'id': 33, 'name': 'Johny'}]
List after removing duplicates based on key id :
[{'id': 44, 'name': 'Ritika'}, {'id': 55, 'name': 'Sanjay'}, {'id': 33, 'name': 'Johny'}]

Summary

Today, we learned how to remove duplicates from list of dictionaries by keys in Python.