Get List of Unique Dictionaries in Python

This tutorial will discuss about unique ways to get list of unique dictionaries in Python.

Suppose we have a list of dictionaries. Like this,

# List of Dictionaries
dictList = [{'name' : 'Shuan', 'Age' : 35, 'city' : 'London'},
            {'name' : 'Vijay', 'Age' : 49, 'city' : 'Mumbai'},
            {'name' : 'Shuan', 'Age' : 35, 'city' : 'London'},
            {'name' : 'Arpit', 'Age' : 31, 'city' : 'Delhi'},
            {'Age' : 35, 'city' : 'London', 'name' : 'Shuan'},
            {'name' : 'Vinay', 'Age' : 31, 'city' : 'Delhi'},
            {'name' : 'Shuan', 'Age' : 29, 'city' : 'Tokyo'}]

Irrespective of order of key-value pairs inside each dictionary, we want to get the unqiue dictionaries only from this kind of list. For example, from the dictionaries in above list, the unique dictionaries are,

[{'name': 'Shuan', 'Age': 35, 'city': 'London'},
 {'name': 'Vijay', 'Age': 49, 'city': 'Mumbai'},
 {'name': 'Arpit', 'Age': 31, 'city': 'Delhi'},
 {'name': 'Vinay', 'Age': 31, 'city': 'Delhi'},
 {'name': 'Shuan', 'Age': 29, 'city': 'Tokyo'}]

To do that, iterate over all dictionaries in list, and for each dictionary, sort all key value pairs in it. Then compare them to find the duplicates, and removeing them. We have created a function to do all this stuff. The function is,

GetUniqueDictionaries(listofDicts)

Using this function, you can get a list of unique dictionaries from a list of dictionaries.

Let’s see the complete example,

def GetUniqueDictionaries(listofDicts):
    """Get a List unique dictionaries
    List to contain unique dictionaries"""
    listOfUniqueDicts = []

    # A set object
    setOfValues = set()

    # iterate over all dictionaries in list
    for dictObj in listofDicts:
        list_Of_tuples = []
        # For each dictionary, iterate over all key
        # and append that to a list as tuples
        for key, value in dictObj.items():
            list_Of_tuples.append( (key, value))

        strValue = ""
        # convert list of tuples to a string
        for key, value in sorted(list_Of_tuples):
            # sort list of tuples, and iterate over them
            # append each pair to string
            strValue += str(key) + "_" + str(value) + "_"

        # Add string to set if not already exist in set
        if strValue not in setOfValues:
            # If string is not in set, then it means
            # this dictionary is unique
            setOfValues.add(strValue)
            listOfUniqueDicts.append (dictObj)
    
    return listOfUniqueDicts


# List of Dictionaries
dictList = [{'name' : 'Shuan', 'Age' : 35, 'city' : 'London'},
            {'name' : 'Vijay', 'Age' : 49, 'city' : 'Mumbai'},
            {'name' : 'Shuan', 'Age' : 35, 'city' : 'London'},
            {'name' : 'Arpit', 'Age' : 31, 'city' : 'Delhi'},
            {'Age' : 35, 'city' : 'London', 'name' : 'Shuan'},
            {'name' : 'Vinay', 'Age' : 31, 'city' : 'Delhi'},
            {'name' : 'Shuan', 'Age' : 29, 'city' : 'Tokyo'}]


# get unique dictionaries from list
listOfUniqueDicts = GetUniqueDictionaries(dictList)

print(listOfUniqueDicts)

Output

[{'name': 'Shuan', 'Age': 35, 'city': 'London'}, 
 {'name': 'Vijay', 'Age': 49, 'city': 'Mumbai'},
 {'name': 'Arpit', 'Age': 31, 'city': 'Delhi'},
 {'name': 'Vinay', 'Age': 31, 'city': 'Delhi'},
 {'name': 'Shuan', 'Age': 29, 'city': 'Tokyo'}]

This function GetUniqueDictionaries() accepts a list of dictionaries as an argument. Then it iterates over all the dictionaries in the list, and for each dictionary it performs the following tasks,

• Sort the dictionary
• Convert all the key-values of a sorted dictionary into a string
• Then check if a set contains that string or not. If not then add that to the set. If yes, then it means that dictionary is a duplicate.
• Add all unique dictionaries in a list.

In the end, it returns a list of unique dictionaries. Thanks.