Python : How to Remove Duplicates from a List
In this article we will discuss different ways to remove duplicate elements from a list in python.
List : Containing duplicate elements :
|
1 |
[10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10] |
List : After removing duplicate elements,
|
1 |
[10, 2, 45, 3, 5, 7, 8] |
Let’s see how to do that,
Remove duplicates from a List using set
Set is an un-ordered data structure that contains only unique elements.
Now suppose we have a list that contains duplicate elements i.e.
|
1 |
[10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10] |
Let’s create a set with this list. So, only unique elements will be added in the set. Then convert this set again to a list i.e.
|
1 2 |
# Convert list to set and then back to list listOfNums = list(set(listOfNums)) |
So, now list will contains only unique elements i.e.
|
1 |
[2, 3, 5, 7, 8, 10, 45] |
But, order of elements is disturbed.
This technique removes the duplicates but it does not keep the elements in same order as original.
If we want to keep the unique elements of list in same order as original, then we need to use other technique.
Removing duplicates from a List by keeping the order
Suppose we have a list that contains duplicate elements i.e.
|
1 |
[10, 2, 45, 3, 5, 7, 2 , 10, 45, 8, 10] |
Now we want to remove duplicates from it and also want to keep the order of unique elements as it was in original list i.e.
Unique elements : [10, 2, 45, 3, 5, 7, 8]
To do that we need to create a new list for unique elements. Then iterate over the original list and for each element in the list, add it to new uniqueList only if its not already in the list.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
''' Remove duplicate elements from list ''' def removeDuplicates(listofElements): # Create an empty list to store unique elements uniqueList = [] # Iterate over the original list and for each element # add it to uniqueList, if its not already there. for elem in listofElements: if elem not in uniqueList: uniqueList.append(elem) # Return the list of unique elements return uniqueList |
Let’s use the above function to remove duplicates from list i.e.
|
1 2 3 4 5 |
# List of Numbers with duplicates listOfNums = [10, 2, 45, 3, 5, 7, 2 , 10, 45, 8, 10] # Remove duplicates from list by keeping the order as original listOfNums = removeDuplicates(listOfNums) |
Contents of the list will be,
|
1 |
[10, 2, 45, 3, 5, 7, 8] |
Complete example is as follows,
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
''' Remove duplicate elements from list ''' def removeDuplicates(listofElements): # Create an empty list to store unique elements uniqueList = [] # Iterate over the original list and for each element # add it to uniqueList, if its not already there. for elem in listofElements: if elem not in uniqueList: uniqueList.append(elem) # Return the list of unique elements return uniqueList def main(): # List of Numbers with duplicates listOfNums = [10, 2, 45, 3, 5, 7, 2 , 10, 45, 8, 10] # Print the List print("Original List : " , listOfNums) ''' Removing duplicates from a List using set ''' # Convert list to set and then back to list listOfNums = list(set(listOfNums)) # Now list contains unique elements only print("List with unique elements : ", listOfNums) ''' Removing duplicates from a List by keeping the Order ''' # List of Numbers with duplicates listOfNums = [10, 2, 45, 3, 5, 7, 2 , 10, 45, 8, 10] # Print the List print("Original List : " , listOfNums) listOfNums = removeDuplicates(listOfNums) # Now list contains unique elements only print("List with unique elements : ", listOfNums) if __name__ == '__main__': main() |
Output:
|
1 2 3 4 |
Original List : [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10] List with unique elements : [2, 3, 5, 7, 8, 10, 45] Original List : [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10] List with unique elements : [10, 2, 45, 3, 5, 7, 8] |
If you didn't find what you were looking, then do suggest us in the comments below. We will be more than happy to add that.
Leave a Reply