Python / Data Science Essentials Interview Questions
How do you detect, handle, and fill missing values in a Pandas DataFrame?
Missing values are represented in Pandas as NaN (float Not-a-Number from NumPy), NaT (Not-a-Time for datetime columns), or pd.NA (the newer nullable integer/string missing marker). Handling them correctly is the most time-consuming step of real-world data cleaning.
import pandas as pd import numpy as np df = pd.DataFrame({ 'name': ['Alice', 'Bob', None, 'Dave'], 'age': [30, np.nan, 35, 28], 'score': [88, 72, np.nan, np.nan], }) # --- Detection --- df.isnull() # boolean DataFrame â True where NaN df.isnull().sum() # count NaNs per column df.isnull().sum() / len(df) * 100 # % missing per column df.notnull() # inverse of isnull # --- Dropping --- df.dropna() # drop rows with ANY NaN df.dropna(how='all') # drop rows where ALL values are NaN df.dropna(subset=['age']) # drop rows with NaN only in 'age' df.dropna(axis=1, thresh=3) # drop columns with fewer than 3 non-NaN values # --- Filling --- df['score'].fillna(df['score'].mean()) # fill with column mean df['score'].fillna(method='ffill') # forward fill (propagate last valid) df['score'].fillna(method='bfill') # backward fill df.fillna({'age': 0, 'name': 'Unknown'}) # column-specific fills # --- Interpolation --- df['score'].interpolate(method='linear') # linear interpolation between values # --- Replace specific sentinel values --- df.replace(-999, np.nan) # treat -999 as missing
Choosing between dropping and filling requires domain knowledge. Dropping rows is acceptable when missing data is rare (below 5%) and appears randomly. Filling with mean/median is common for numerical features; filling with mode or a sentinel ('Unknown') for categoricals. For time-series, forward fill preserves temporal order.
More Related questions...