How do you reduce a Pandas DataFrame's memory usage through dtype optimisation?

DataFrames loaded from CSV often use unnecessarily large dtypes — 64-bit integers for values that fit in 8 bits, generic object dtype for repeated string categories. Downcasting dtypes can reduce memory by 4–8× without any data loss, enabling analysis of larger datasets within available RAM.

import pandas as pd
import numpy as np

df = pd.read_csv('large.csv')
print(f'Memory before: {df.memory_usage(deep=True).sum() / 1e6:.1f} MB')

# --- Integer downcasting ---
for col in df.select_dtypes('int64').columns:
    df[col] = pd.to_numeric(df[col], downcast='integer')
    # downcast tries int8 -> int16 -> int32 depending on value range

# --- Float downcasting ---
for col in df.select_dtypes('float64').columns:
    df[col] = pd.to_numeric(df[col], downcast='float')  # float32

# --- Categorical: object columns with low cardinality ---
# If a column has < 5% unique values, Categorical saves memory
for col in df.select_dtypes('object').columns:
    n_unique = df[col].nunique()
    if n_unique / len(df) < 0.05:    # less than 5% cardinality
        df[col] = df[col].astype('category')

print(f'Memory after : {df.memory_usage(deep=True).sum() / 1e6:.1f} MB')

# Categorical also speeds up groupby on low-cardinality columns
# because grouping enumerates integers rather than comparing strings

The Categorical dtype stores repeated strings as integer codes internally — a column with 5 unique city names in a million-row dataset stores one integer per row rather than one full string per row. This speeds up groupby, sort_values, and value_counts in addition to saving memory.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.