In this article, we will discuss different ways to replace blank values / whitespaces with NaN values in a Pandas DataFrame.
A DataFrame is a Python Data Structure that stores the data in a tabular format i.e. in rows and columns. We can create a DataFrame using pandas.DataFrame() method. Let’s create a dataframe with four rows and two columns and with some empty string values,
import pandas as pd # Create dataframe with two columns and four rows df = pd.DataFrame({ "Name" : [" ", "sravan", "ramya", " "], "Subjects" : [" ", "python", " ", " "]}) # Display the Dataframe print(df)
Output:
Name Subjects 0 1 sravan python 2 ramya 3
In the above DataFrame, there are different types of empty strings i.e. with single whitespace and more than single space empty strings. Let’s see how to replace all the empty strings with with NaN.
Replace empty strings in Dataframe using replace() and regex
In Pandas, both the Dataframe and Series class provides a function replace() to change the contents. Let’s look into their syntax,
Frequently Asked:
DataFrame.replace()
DataFrame.replace(to_replace, replacement, regex=True)
It accepts three values as arguments i.e
- to_replace: Direct value or a regex pattern. If regex pattern, then based on this, it will decide which values needs to be replaced.
- replacement: The replacement value
- regex: If True, then first parameter “to_replace” is used as regex pattern.
In entire DataFrame, it will look for values that matches the regex pattern and replace those values with the given replacement string.
Series.replace()
Latest Python - Video Tutorial
Series.replace(to_replace, replacement, regex=True)
It accepts three values as arguments i.e
- to_replace: Direct value or a regex pattern. If regex pattern, then based on this, it will decide which values needs to be replaced.
- replacement: The replacement value
- regex: If True, then first parameter “to_replace” is used as regex pattern.
In entire Series, it will look for values that matches the regex pattern and replace those values with the given replacement string.
Let’s use these functions to replace empty strings with NaN, either in entire Dataframe or in a column only.
Replace empty strings with NaN in a DataFrame Column
Select a DataFrame column as a Series object and call the replace() function on it with following parameters,
- As a first parameter pass a regex pattern that will match one or more whitespaces i.e. “^\s*$” .
- As second parameter pass a replacement value i.e. np.NaN
- As third parameter pass regex=True
It will replace all the empty strings with NaN values in the column. For example,
import pandas as pd import numpy as np # Create dataframe with two columns and four rows df = pd.DataFrame({ "Name" : [" ", "sravan", "ramya", " "], "Subjects" : [" ", "python", " ", " "]}) # Display the Dataframe print(df) # Replace empty strings with NaN in column 'Name' df['Name'] = df['Name'].replace(["^\s*$"], np.NaN, regex=True) # Display the Dataframe print(df)
Output:
Name Subjects 0 1 sravan python 2 ramya 3 Name Subjects 0 NaN 1 sravan python 2 ramya 3 NaN
It replaced all the empty strings in column ‘Name’ with NaN values.
Replace empty strings with NaN values in entire dataframe
Call the replace() function on the DataFrame object with following parameters,
- As a first parameter pass a regex pattern that will match one or more whitespaces i.e. “^\s*$” .
- As second parameter pass a replacement value i.e. np.NaN
- As third parameter pass regex=True
It will replace all the empty strings with NaN values in the entire. For example,
import pandas as pd import numpy as np # Create dataframe with two columns and four rows df = pd.DataFrame({ "Name" : [" ", "sravan", "ramya", " "], "Subjects" : [" ", "python", " ", " "]}) # Display the Dataframe print(df) # Replace empty strings with NaN in entire DataFrame df = df.replace(["^\s*$"], np.NaN, regex=True) # Display the Dataframe print(df)
Output:
Name Subjects 0 1 sravan python 2 ramya 3 Name Subjects 0 NaN NaN 1 sravan python 2 ramya NaN 3 NaN NaN
It replaced all the empty strings with NaN values in the entire DataFrame.
Summary
In this article we learned how to replace empty strings with NaN values in a Dataframe using regex and replace() function.
Latest Video Tutorials