Drop duplicate list python

How to Remove Duplicates from a List in Python

You can remove duplicates from a Python using the dict.fromkeys(), which generates a dictionary that removes any duplicate values. You can also convert a list to a set. You must convert the dictionary or set back into a list to see a list whose duplicates have been removed.

When you’re working with a list in Python, there may be a scenario where you want to remove duplicates from the list. For example, say you are a chef who is merging the old and new menus for your restaurant. You may want to remove any duplicates that occur from combining the two menus.

This tutorial will cover using the dict.fromkeys() method converting a list to a set. We’ll discuss how to convert the output from these two methods back into a list so you can interpret your data as a list.

Python Remove Duplicates From a List

You can remove duplicates using a Python set or the dict.fromkeys() method.

The dict.fromkeys() method converts a list into a dictionary. Dictionaries cannot contain duplicate values so a dictionary with only unique values is returned by dict.fromkeys().

Sets, like dictionaries, cannot contain duplicate values. If we convert a list to a set, all the duplicates are removed.

Python Remove Duplicates from List: dict.fromkeys()

Python dictionaries store data in a key/value structure. Dictionaries map keys to values and create a pair that stores data in Python.

Dictionaries in Python cannot include duplicate keys. If we convert our list to a dictionary, it will remove any duplicate values. That’s where the fromkeys() method comes in.

dict.fromkeys() Example

fromkeys() is a built-in function that generates a dictionary from the keys you have specified. Because dictionaries cannot include duplicate keys, the function will remove any duplicate values from our list. Then, we can convert our dictionary back to a list.

The syntax for the fromkeys() method is as follows:

dictionary_name.fromkeys(keys, value)

fromkeys takes two parameters: keys and value. The keys parameter is the iterable, which specifies the keys of the new dictionary you want to create. Our value parameter is optional and holds the value for all keys. We do not need this parameter for this example, so we will ignore it.

Let’s say that we are a chef who is merging the old and new pizza menus into one big menu. The new menu includes a number of new pizzas that we have been working on for months. But, we want to offer the pizzas on the old menu. We do not want duplicates to appear on the menu, because that would confuse customers.

To remove any duplicates that may exist in the new menu, we could use the following code:

new_menu = ['Hawaiian', 'Margherita', 'Mushroom', 'Prosciutto', 'Meat Feast', 'Hawaiian', 'Bacon', 'Black Olive Special', 'Sausage', 'Sausage'] final_new_menu = list(dict.fromkeys(new_menu)) print(final_new_menu)

Our code removes our duplicated elements and returns:

['Hawaiian', 'Margherita', 'Mushroom', 'Prosciutto', 'Meat Feast', 'Bacon', 'Black Olive Special', 'Sausage']

On the first line, we declare a Python variable called new_menu which stores all the pizzas on the new menu. Note that this list contains duplicate values — both Sausage and Hawaiian appear twice in the Python lists.

We use the dict.fromkeys() method to create a dictionary from our new_menu variable. Next, we use list() to convert our data from a dictionary to a Python array, or list. On the final line of code, we print out our revised list.

As you can see, our program has removed the duplicate values from our data, and Sausage and Hawaiian appear once in our new list.

Remove Duplicates from List Python: Sets

The Python set object is a data structure that is both unordered and unindexed. Sets store collections of unique items in Python. Unlike lists, sets cannot store duplicate values.

So, because sets cannot store duplicate items, we can convert our list to a set to remove them. In Python, sets are a list of comma-separated items enclosed within curly brackets. Here’s an example of a set in Python that stores a list of pizzas:

pizzas = ('Margherita', 'Hawaiian', 'Mushroom', 'Meat Feast')

Remove Duplicates Using a Set Example

Let’s take our example from above to illustrate how sets can be used to remove duplicate values from a list.

Say that we are a pizza chef who wants to merge both our old and new menus. But there may be duplicates between these menus, which we will want to remove before creating our final menu. To remove these duplicates using the set approach, we could use the following code:

new_menu = ['Hawaiian', 'Margherita', 'Mushroom', 'Prosciutto', 'Meat Feast', 'Hawaiian', 'Bacon', 'Black Olive Special', 'Sausage', 'Sausage'] final_new_menu = list(set(new_menu)) print(final_new_menu)

Here’s our list after removing duplicate values:

['Mushroom', 'Black Olive Special', 'Meat Feast', 'Bacon', 'Margherita', 'Sausage', 'Prosciutto', 'Hawaiian']

At the start of our program, we declare a list called new_menu which stores all the pizzas on the new menu.

Then, we declare a variable called final_new_menu which stores our revised menu. This variable converts the new_menu variable to a set using set(), then converts the set back to a list using list(). This process removes all the duplicate values in our new_menu variable.

Finally, our program prints the new menu to the console. All the unique elements from the original list object appear. We now have a unique list.

«Career Karma entered my life when I needed it most and quickly helped me match with a bootcamp. Two months after graduating, I found my dream job that aligned with my values and goals in life!»

Venus, Software Engineer at Rockbot

To learn more about Python programming, read our How to Learn Python guide.

About us: Career Karma is a platform designed to help job seekers find, research, and connect with job training programs to advance their careers. Learn about the CK publication.

Источник

How to Remove Duplicates From a Python List

Learn how to remove duplicates from a List in Python.

Example

Remove any duplicates from a List:

Example Explained

First we have a List that contains duplicates:

A List with Duplicates

Create a dictionary, using the List items as keys. This will automatically remove any duplicates because dictionaries cannot have duplicate keys.

Create a Dictionary

Then, convert the dictionary back into a list:

Convert Into a List

Now we have a List without any duplicates, and it has the same order as the original List.

Print the List to demonstrate the result

Print the List

Create a Function

If you like to have a function where you can send your lists, and get them back without duplicates, you can create a function and insert the code from the example above.

Example

def my_function(x):
return list(dict.fromkeys(x))

mylist = my_function([«a», «b», «a», «c», «c»])

Example Explained

Create a function that takes a List as an argument.

Create a Function

def my_function(x):
return list(dict.fromkeys(x))

mylist = my_function([«a», «b», «a», «c», «c»])

Create a dictionary, using this List items as keys.

Create a Dictionary

def my_function(x):
return list( dict.fromkeys(x) )

mylist = my_function([«a», «b», «a», «c», «c»])

Convert the dictionary into a list.

Convert Into a List

def my_function(x):
return list( dict.fromkeys(x) )

mylist = my_function([«a», «b», «a», «c», «c»])

Return List

def my_function(x):
return list(dict.fromkeys(x))

mylist = my_function([«a», «b», «a», «c», «c»])

Call the function, with a list as a parameter:

Call the Function

def my_function(x):
return list(dict.fromkeys(x))

mylist = my_function([«a», «b», «a», «c», «c»])

Print the Result

def my_function(x):
return list(dict.fromkeys(x))

mylist = my_function([«a», «b», «a», «c», «c»])

print(mylist)

COLOR PICKER

Report Error

If you want to report an error, or if you want to make a suggestion, do not hesitate to send us an e-mail:

Thank You For Helping Us!

Your message has been sent to W3Schools.

Top Tutorials

Top References

Top Examples

Get Certified

W3Schools is optimized for learning and training. Examples might be simplified to improve reading and learning. Tutorials, references, and examples are constantly reviewed to avoid errors, but we cannot warrant full correctness of all content. While using W3Schools, you agree to have read and accepted our terms of use, cookie and privacy policy.

Источник