How do you detect, handle, and fill missing values in a Pandas DataFrame?

Missing values are represented in Pandas as NaN (float Not-a-Number from NumPy), NaT (Not-a-Time for datetime columns), or pd.NA (the newer nullable integer/string missing marker). Handling them correctly is the most time-consuming step of real-world data cleaning.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'name':   ['Alice', 'Bob', None, 'Dave'],
    'age':    [30, np.nan, 35, 28],
    'score':  [88, 72, np.nan, np.nan],
})

# --- Detection ---
df.isnull()               # boolean DataFrame — True where NaN
df.isnull().sum()         # count NaNs per column
df.isnull().sum() / len(df) * 100  # % missing per column
df.notnull()              # inverse of isnull

# --- Dropping ---
df.dropna()               # drop rows with ANY NaN
df.dropna(how='all')      # drop rows where ALL values are NaN
df.dropna(subset=['age']) # drop rows with NaN only in 'age'
df.dropna(axis=1, thresh=3)  # drop columns with fewer than 3 non-NaN values

# --- Filling ---
df['score'].fillna(df['score'].mean())   # fill with column mean
df['score'].fillna(method='ffill')       # forward fill (propagate last valid)
df['score'].fillna(method='bfill')       # backward fill
df.fillna({'age': 0, 'name': 'Unknown'})  # column-specific fills

# --- Interpolation ---
df['score'].interpolate(method='linear')  # linear interpolation between values

# --- Replace specific sentinel values ---
df.replace(-999, np.nan)  # treat -999 as missing

Choosing between dropping and filling requires domain knowledge. Dropping rows is acceptable when missing data is rare (below 5%) and appears randomly. Filling with mean/median is common for numerical features; filling with mode or a sentinel ('Unknown') for categoricals. For time-series, forward fill preserves temporal order.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.