Python : How to Remove Duplicates from a List

In this article we will discuss different ways to remove duplicate elements from a list in python.

List : Containing duplicate elements :

[10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]

List : After removing duplicate elements,

[10, 2, 45, 3, 5, 7, 8]

Let’s see how to do that,

Remove duplicates from a List using set

Set is an un-ordered data structure that contains only unique elements.

Now suppose we have a list that contains duplicate elements i.e.

[10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]

Let’s create a set with this list. So, only unique elements will be added in the set. Then convert this set again to a list i.e.

# Convert list to set and then back to list
listOfNums = list(set(listOfNums))

So, now list will contains only unique elements i.e.

[2, 3, 5, 7, 8, 10, 45]

But, order of elements is disturbed.

This technique removes the duplicates but it does not keep the elements in same order as original.
If we want to keep the unique elements of list in same order as original, then we need to use other technique.

Removing duplicates from a List by keeping the order

Suppose we have a list that contains duplicate elements i.e.

[10, 2, 45, 3, 5, 7, 2 , 10, 45,  8, 10]

Now we want to remove duplicates from it and also want to keep the order of unique elements as it was in original list i.e.

Unique elements : [10, 2, 45, 3, 5, 7, 8]

To do that we need to create a new list for unique elements. Then iterate over the original list and for each element in the list, add it to new uniqueList only if its not already in the list.

'''
    Remove duplicate elements from list
'''
def removeDuplicates(listofElements):
    
    # Create an empty list to store unique elements
    uniqueList = []
    
    # Iterate over the original list and for each element
    # add it to uniqueList, if its not already there.
    for elem in listofElements:
        if elem not in uniqueList:
            uniqueList.append(elem)
    
    # Return the list of unique elements        
    return uniqueList

Let’s use the above function to remove duplicates from list i.e.

# List of Numbers with duplicates
listOfNums = [10, 2, 45, 3, 5, 7, 2 , 10, 45,  8, 10]

# Remove duplicates from list by keeping the order as original
listOfNums = removeDuplicates(listOfNums)

Contents of the list will be,

[10, 2, 45, 3, 5, 7, 8]

Complete example is as follows,

'''
    Remove duplicate elements from list
'''
def removeDuplicates(listofElements):
    
    # Create an empty list to store unique elements
    uniqueList = []
    
    # Iterate over the original list and for each element
    # add it to uniqueList, if its not already there.
    for elem in listofElements:
        if elem not in uniqueList:
            uniqueList.append(elem)
    
    # Return the list of unique elements        
    return uniqueList


def main():

    # List of Numbers with duplicates   
    listOfNums = [10, 2, 45, 3, 5, 7, 2 , 10, 45,  8, 10]
    
    # Print the List
    print("Original List : " , listOfNums)
    
    '''
        Removing duplicates from a List using set
    '''
    
    # Convert list to set and then back to list
    listOfNums = list(set(listOfNums))
    
    # Now list contains unique elements only
    print("List with unique elements : ", listOfNums)
    
    
    '''
        Removing duplicates from a List by keeping the Order
    '''
    
    # List of Numbers with duplicates
    listOfNums = [10, 2, 45, 3, 5, 7, 2 , 10, 45,  8, 10]
    
    # Print the List
    print("Original List : " , listOfNums)
            
    listOfNums = removeDuplicates(listOfNums)
    
    # Now list contains unique elements only
    print("List with unique elements : ", listOfNums)
    
    
        
if __name__ == '__main__':
    main()

 

Output:

Original List :  [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
List with unique elements :  [2, 3, 5, 7, 8, 10, 45]
Original List :  [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
List with unique elements :  [10, 2, 45, 3, 5, 7, 8]