Replace NaN with preceding/previous values in Pandas

This tutorial will discuss about different ways to replace NaN with preceding / previous values in pandas.

Table Of Contents

Introduction

Suppose we have a DataFrame with some NaN values i.e.

   First  Second
0   10.0    51.0
1    NaN    52.0
2   11.0     NaN
3    NaN    53.0
4   44.0    54.0
5   55.0    55.0

Now we want to replace all the NaN values in the DataFrame with the previous valid value from the same column. Like this,

   First  Second
0   10.0    51.0
1   10.0    52.0
2   11.0    52.0
3   11.0    53.0
4   44.0    54.0
5   55.0    55.0

The NaN value in column “Second” got replaced with the previous value in the same column i.e. 52. The NaN value in column “First” got replaced with the previous value in the same column i.e. 10 and 11.

Let’s see how to do that.

Preparing DataSet

First we will create a DataFrame with some NaN values.

import pandas as pd
import numpy as np

# Create a DataFrame
df = pd.DataFrame(
    {'First':  [10, np.NaN, 11, np.NaN, 44, 55],
     'Second': [51, 52, np.NaN, 53, 54, 55]})

print(df)

Output

   First  Second
0   10.0    51.0
1    NaN    52.0
2   11.0     NaN
3    NaN    53.0
4   44.0    54.0
5   55.0    55.0

Replace NaN values with preceding values in DataFrame

Call the fillna() function on DataFrame, and pass the parameter method="ffill". It should fill each NaN value in DataFrame with the last valid value from the same column.

# Replace each NaN values with the previous valid value
# in that column
df.fillna(method="ffill", inplace=True)

print(df)

Output

   First  Second
0   10.0    51.0
1   10.0    52.0
2   11.0    52.0
3   11.0    53.0
4   44.0    54.0
5   55.0    55.0

Replace NaN values in a column with preceding value

Select the column as Pandas Series object, and call fillna() function on that column/series with parameter method="ffill". It should fill all the NaNs in that column, with the previous value from the same column.

# Replace NaN values in column 'Second' with the previous valid value
# in that column
df['Second'].fillna(method="ffill", inplace=True)

print(df)

Output

   First  Second
0   10.0    51.0
1    NaN    52.0
2   11.0    52.0
3    NaN    53.0
4   44.0    54.0
5   55.0    55.0

Summary

We learned how to replace NaN values with previous values in Pandas.