What are the key performance tips when using NumPy for large-scale data processing?

NumPy is fast by default, but a few common mistakes can undermine that speed. Knowing these patterns makes the difference between code that runs in seconds and code that runs in minutes.

import numpy as np

n = 10_000_000
arr = rng.random(n)

# 1. AVOID Python loops — always prefer ufuncs
# Slow:
result = [x**2 for x in arr]        # Python loop, ~3s
# Fast:
result = arr ** 2                    # NumPy ufunc, ~0.03s

# 2. Pre-allocate output arrays instead of growing them
# Slow:
out = []
for chunk in chunks:
    out.append(chunk.sum())          # repeated list growth
# Fast:
out = np.empty(len(chunks))
for i, chunk in enumerate(chunks):
    out[i] = chunk.sum()

# 3. Use views instead of copies when slicing
sub = arr[1000:2000]   # view — no memory allocation
sub2 = arr[1000:2000].copy()  # explicit copy — only when mutation safety needed

# 4. Choose the right dtype — float32 vs float64
a64 = np.ones(n, dtype=np.float64)  # 80 MB
a32 = np.ones(n, dtype=np.float32)  # 40 MB — also faster on many ops

# 5. Use out= argument to avoid temporary arrays
np.add(a32, a32, out=a32)   # in-place: no temporary intermediate created

# 6. np.einsum for complex multi-dimensional contractions
A = rng.random((100, 200))
B = rng.random((200, 300))
C = np.einsum('ij,jk->ik', A, B)  # equivalent to A @ B but explicit

The most impactful optimisation in almost every case is the first: eliminating Python loops. After that, reducing the number of temporary arrays (using out= or in-place operators like +=) and choosing smaller dtypes are the next biggest wins.

Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!

Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.

Invest now!!! Get Free equity stock (US, UK only)!

Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.

The Robinhood app makes it easy to trade stocks, crypto and more.

Webull! Receive free stock by signing up using the link: Webull signup.