In this article we will discuss different ways to remove duplicate elements from a list in python.
List : Containing duplicate elements :
[10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
List : After removing duplicate elements,
[10, 2, 45, 3, 5, 7, 8]
Let’s see how to do that,
Remove duplicates from a List using set
Set is an un-ordered data structure that contains only unique elements.
Now suppose we have a list that contains duplicate elements i.e.
[10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10]
Let’s create a set with this list. So, only unique elements will be added in the set. Then convert this set again to a list i.e.
# Convert list to set and then back to list listOfNums = list(set(listOfNums))
So, now list will contains only unique elements i.e.
[2, 3, 5, 7, 8, 10, 45]
But, order of elements is disturbed.
This technique removes the duplicates but it does not keep the elements in same order as original.
If we want to keep the unique elements of list in same order as original, then we need to use other technique.
Removing duplicates from a List by keeping the order
Suppose we have a list that contains duplicate elements i.e.
[10, 2, 45, 3, 5, 7, 2 , 10, 45, 8, 10]
Now we want to remove duplicates from it and also want to keep the order of unique elements as it was in original list i.e.
Unique elements : [10, 2, 45, 3, 5, 7, 8]
To do that we need to create a new list for unique elements. Then iterate over the original list and for each element in the list, add it to new uniqueList only if its not already in the list.
''' Remove duplicate elements from list ''' def removeDuplicates(listofElements): # Create an empty list to store unique elements uniqueList = [] # Iterate over the original list and for each element # add it to uniqueList, if its not already there. for elem in listofElements: if elem not in uniqueList: uniqueList.append(elem) # Return the list of unique elements return uniqueList
Let’s use the above function to remove duplicates from list i.e.
# List of Numbers with duplicates listOfNums = [10, 2, 45, 3, 5, 7, 2 , 10, 45, 8, 10] # Remove duplicates from list by keeping the order as original listOfNums = removeDuplicates(listOfNums)
Contents of the list will be,
[10, 2, 45, 3, 5, 7, 8]
Complete example is as follows,
''' Remove duplicate elements from list ''' def removeDuplicates(listofElements): # Create an empty list to store unique elements uniqueList = [] # Iterate over the original list and for each element # add it to uniqueList, if its not already there. for elem in listofElements: if elem not in uniqueList: uniqueList.append(elem) # Return the list of unique elements return uniqueList def main(): # List of Numbers with duplicates listOfNums = [10, 2, 45, 3, 5, 7, 2 , 10, 45, 8, 10] # Print the List print("Original List : " , listOfNums) ''' Removing duplicates from a List using set ''' # Convert list to set and then back to list listOfNums = list(set(listOfNums)) # Now list contains unique elements only print("List with unique elements : ", listOfNums) ''' Removing duplicates from a List by keeping the Order ''' # List of Numbers with duplicates listOfNums = [10, 2, 45, 3, 5, 7, 2 , 10, 45, 8, 10] # Print the List print("Original List : " , listOfNums) listOfNums = removeDuplicates(listOfNums) # Now list contains unique elements only print("List with unique elements : ", listOfNums) if __name__ == '__main__': main()
Output:
Original List : [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10] List with unique elements : [2, 3, 5, 7, 8, 10, 45] Original List : [10, 2, 45, 3, 5, 7, 2, 10, 45, 8, 10] List with unique elements : [10, 2, 45, 3, 5, 7, 8]
Pandas Tutorials -Learn Data Analysis with Python
-
Pandas Tutorial Part #1 - Introduction to Data Analysis with Python
-
Pandas Tutorial Part #2 - Basics of Pandas Series
-
Pandas Tutorial Part #3 - Get & Set Series values
-
Pandas Tutorial Part #4 - Attributes & methods of Pandas Series
-
Pandas Tutorial Part #5 - Add or Remove Pandas Series elements
-
Pandas Tutorial Part #6 - Introduction to DataFrame
-
Pandas Tutorial Part #7 - DataFrame.loc[] - Select Rows / Columns by Indexing
-
Pandas Tutorial Part #8 - DataFrame.iloc[] - Select Rows / Columns by Label Names
-
Pandas Tutorial Part #9 - Filter DataFrame Rows
-
Pandas Tutorial Part #10 - Add/Remove DataFrame Rows & Columns
-
Pandas Tutorial Part #11 - DataFrame attributes & methods
-
Pandas Tutorial Part #12 - Handling Missing Data or NaN values
-
Pandas Tutorial Part #13 - Iterate over Rows & Columns of DataFrame
-
Pandas Tutorial Part #14 - Sorting DataFrame by Rows or Columns
-
Pandas Tutorial Part #15 - Merging or Concatenating DataFrames
-
Pandas Tutorial Part #16 - DataFrame GroupBy explained with examples
Are you looking to make a career in Data Science with Python?
Data Science is the future, and the future is here now. Data Scientists are now the most sought-after professionals today. To become a good Data Scientist or to make a career switch in Data Science one must possess the right skill set. We have curated a list of Best Professional Certificate in Data Science with Python. These courses will teach you the programming tools for Data Science like Pandas, NumPy, Matplotlib, Seaborn and how to use these libraries to implement Machine learning models.
Checkout the Detailed Review of Best Professional Certificate in Data Science with Python.
Remember, Data Science requires a lot of patience, persistence, and practice. So, start learning today.