Python / Data Science Essentials Interview Questions
What are the key performance tips when using NumPy for large-scale data processing?
NumPy is fast by default, but a few common mistakes can undermine that speed. Knowing these patterns makes the difference between code that runs in seconds and code that runs in minutes.
import numpy as np n = 10_000_000 arr = rng.random(n) # 1. AVOID Python loops â always prefer ufuncs # Slow: result = [x**2 for x in arr] # Python loop, ~3s # Fast: result = arr ** 2 # NumPy ufunc, ~0.03s # 2. Pre-allocate output arrays instead of growing them # Slow: out = [] for chunk in chunks: out.append(chunk.sum()) # repeated list growth # Fast: out = np.empty(len(chunks)) for i, chunk in enumerate(chunks): out[i] = chunk.sum() # 3. Use views instead of copies when slicing sub = arr[1000:2000] # view â no memory allocation sub2 = arr[1000:2000].copy() # explicit copy â only when mutation safety needed # 4. Choose the right dtype â float32 vs float64 a64 = np.ones(n, dtype=np.float64) # 80 MB a32 = np.ones(n, dtype=np.float32) # 40 MB â also faster on many ops # 5. Use out= argument to avoid temporary arrays np.add(a32, a32, out=a32) # in-place: no temporary intermediate created # 6. np.einsum for complex multi-dimensional contractions A = rng.random((100, 200)) B = rng.random((200, 300)) C = np.einsum('ij,jk->ik', A, B) # equivalent to A @ B but explicit
The most impactful optimisation in almost every case is the first: eliminating Python loops. After that, reducing the number of temporary arrays (using out= or in-place operators like +=) and choosing smaller dtypes are the next biggest wins.
More Related questions...