Python / Data Science Essentials Interview Questions
How do you reduce a Pandas DataFrame's memory usage through dtype optimisation?
DataFrames loaded from CSV often use unnecessarily large dtypes — 64-bit integers for values that fit in 8 bits, generic object dtype for repeated string categories. Downcasting dtypes can reduce memory by 4–8× without any data loss, enabling analysis of larger datasets within available RAM.
import pandas as pd import numpy as np df = pd.read_csv('large.csv') print(f'Memory before: {df.memory_usage(deep=True).sum() / 1e6:.1f} MB') # --- Integer downcasting --- for col in df.select_dtypes('int64').columns: df[col] = pd.to_numeric(df[col], downcast='integer') # downcast tries int8 -> int16 -> int32 depending on value range # --- Float downcasting --- for col in df.select_dtypes('float64').columns: df[col] = pd.to_numeric(df[col], downcast='float') # float32 # --- Categorical: object columns with low cardinality --- # If a column has < 5% unique values, Categorical saves memory for col in df.select_dtypes('object').columns: n_unique = df[col].nunique() if n_unique / len(df) < 0.05: # less than 5% cardinality df[col] = df[col].astype('category') print(f'Memory after : {df.memory_usage(deep=True).sum() / 1e6:.1f} MB') # Categorical also speeds up groupby on low-cardinality columns # because grouping enumerates integers rather than comparing strings
The Categorical dtype stores repeated strings as integer codes internally — a column with 5 unique city names in a million-row dataset stores one integer per row rather than one full string per row. This speeds up groupby, sort_values, and value_counts in addition to saving memory.
More Related questions...