Python / Python Deep Learning and Neural Networks Interview Questions
1. What is a neural network and how does forward propagation work mathematically?
A neural network is a parameterised function composed of stacked layers. Each layer applies a linear transformation followed by a non-linear activation: h = σ(Wx + b) , where W is a weight matrix, b is a bias vector, and σ is an activation function. Stacking L such layers gives a universal functi...
2. Explain backpropagation mathematically. How does the chain rule enable computing gradients through many layers?
Backpropagation is the algorithm for computing the gradient of a scalar loss L with respect to every parameter in the network. It exploits the chain rule of calculus: if the loss depends on parameter W through intermediate quantities h₁, h₂, ..., hₙ , then ∂L/∂W = (∂L/∂hₙ)(∂hₙ/∂hₙ₋₁)···(∂h₁/∂W) ....
3. What are the most common activation functions and why did ReLU replace sigmoid/tanh as the default?
Activation functions introduce non-linearity without them, stacking linear layers would collapse into a single linear transformation. Several families exist, each with different mathematical properties that affect training dynamics. Common Activation Functions Function Formula Range Key propert...
4. What are vanishing and exploding gradients, and what techniques are used to address them?
Vanishing gradients occur when gradients shrink exponentially as they are backpropagated through many layers — the product of many small numbers (e.g. sigmoid derivatives ≤ 0.25) approaches zero, making early layer weights unable to update meaningfully. Exploding gradients are the opposite: the p...
5. Why does weight initialization matter in neural networks, and what is the difference between Xavier and He initialization?
If weights are initialized too small, activations and gradients shrink layer by layer — a form of vanishing gradient from the start. If too large, they explode. The goal of principled initialisation is to keep the variance of activations and gradients roughly constant across all layers at the sta...
6. How does Batch Normalization work mathematically and why does it stabilize training?
Batch Normalisation (BN) normalises the pre-activation values within a mini-batch to have zero mean and unit variance, then rescales them with learnable parameters γ (scale) and β (shift): BN(x) = γ · (x - μ_B) / √(σ²_B + ε) + β , where μ_B and σ²_B are the batch mean and variance, and ε is a sma...
7. Compare SGD, SGD with momentum, RMSProp, and Adam optimizers. When do you choose each?
All these optimizers share the same goal — updating parameters to reduce loss — but differ in how they use gradient history to adapt the update step. Understanding the mechanics helps diagnose slow training and poor generalisation. Optimizer Comparison Optimizer Update rule (simplified) Key advan...
8. How does Dropout work mathematically, and why does it act as regularization?
During training, Dropout randomly sets each neuron's output to zero with probability p (the drop probability) and scales the remaining activations by 1/(1-p) to preserve the expected sum. This means each forward pass trains a different thinned sub-network — with n neurons, there are 2ⁿ possible s...
9. Explain how convolutional layers work and why they are well-suited to image data.
A convolutional layer applies a set of learnable filters (kernels) by sliding each filter over the spatial dimensions of the input and computing a dot product at each position. For a 2D image, a kernel of size k×k with C_in input channels and C_out output channels has k×k×C_in×C_out parameters to...
10. How do RNNs work and why did LSTMs solve the long-range dependency problem?
A vanilla RNN processes a sequence step-by-step, maintaining a hidden state hₜ = tanh(Wₓxₜ + Wₕhₜ₋₁ + b) that acts as a compressed memory of everything seen so far. The problem is that this hidden state must be updated at every step — and during backpropagation through time (BPTT), gradients are ...
11. What is the self-attention mechanism in Transformers and why did it replace RNNs for sequence modeling?
Self-attention computes a weighted sum of all input vectors, where the weight between positions i and j reflects how much position i should 'attend to' position j. Concretely, input vectors are linearly projected into queries (Q), keys (K), and values (V), and the attention output is: Attention(Q...
12. What loss functions does PyTorch provide for classification and regression, and which to use when?
The choice of loss function should match the output type and the probabilistic assumption about the data-generating process — it is the mathematical link between model predictions and the training signal. Common PyTorch Loss Functions Task Loss PyTorch class Notes Binary classification Binary cro...
13. What is transfer learning and how do you fine-tune a pretrained model in PyTorch?
Transfer learning reuses a model trained on a large dataset (typically ImageNet for vision, or a large text corpus for NLP) as a starting point for a related task with less data. The pretrained model has already learned general features (edges, textures, shapes for images; grammar, semantics for ...
14. How does PyTorch's Dataset and DataLoader pipeline work, and what are the key performance considerations?
PyTorch's data loading follows a clean two-class design: Dataset encapsulates how to access a single sample (index → (X, y)), and DataLoader wraps a Dataset to handle batching, shuffling, and parallel data loading. Separating these responsibilities makes it easy to write dataset-specific logic on...
15. Why is learning rate scheduling important and what are the most common strategies?
A fixed learning rate is a poor choice for most training runs: too high early on causes instability; too high late in training prevents fine convergence to a sharp minimum. Learning rate schedulers systematically vary the lr during training to get the best of both worlds — fast progress early, pr...
16. What are the most effective regularization strategies for deep learning and how do they differ from classical ML regularization?
Deep neural networks have millions of parameters and can trivially memorise training data. Classical regularisation (L1/L2 on weights) still applies, but modern deep learning has developed additional techniques that often work better or are used in combination. DL Regularization Techniques Techni...
17. What are embedding layers in deep learning and how are they different from one-hot encoding?
An embedding layer is a learnable lookup table that maps discrete tokens (words, categories, user IDs) to dense, low-dimensional real-valued vectors. It is mathematically a matrix E ∈ ℝ^{V×d} (vocabulary size × embedding dimension), and looking up token i simply retrieves row i — equivalent to mu...
18. How do you save and load PyTorch models correctly, and what is included in a proper checkpoint?
PyTorch provides two main ways to persist a model: saving the full model object (convenient but fragile to class definition changes) or saving only the state dictionary (recommended for production and reproducibility). The state dict is a Python OrderedDict mapping layer names to their parameter ...
19. What is mixed precision training and how does it speed up deep learning with torch.cuda.amp?
Modern GPUs (Volta and later) have dedicated hardware for 16-bit floating-point operations (FP16 / BFloat16) that can be 2–8× faster than FP32 for matrix multiplications. Mixed precision training runs the forward pass and gradient computations in FP16 (or BF16) for speed, while maintaining a mast...
20. What is the difference between model.eval(), torch.no_grad(), and torch.inference_mode()? When do you use each?
These three mechanisms serve different but complementary purposes that are often confused. Understanding the distinction prevents subtle bugs in training, validation, and inference code. eval vs no_grad vs inference_mode Mechanism What it controls Effect model.eval() Layer behaviour (Dropout, Bat...
21. How do you use GPUs in PyTorch and what are the key patterns for writing device-agnostic code?
PyTorch's device abstraction allows the same code to run on CPU, single GPU, or multiple GPUs with minimal changes. The fundamental operations are moving tensors to a device with .to(device) or .cuda() , and ensuring model and data tensors always reside on the same device before any computation. ...
22. What are the differences between Batch Norm, Layer Norm, Group Norm, and Instance Norm?
All normalisation variants compute mean and variance and apply the same transformation (x-μ)/√(σ²+ε) — they differ only in which dimensions the mean and variance are computed over. This seemingly small difference has large practical consequences depending on the architecture and batch size. Norma...
23. What is an autoencoder and what can a well-trained latent space be used for?
An autoencoder is a neural network trained to reconstruct its input through a bottleneck. The encoder f: X → Z maps inputs to a lower-dimensional latent space Z, and the decoder g: Z → X̂ reconstructs the input. Training minimises the reconstruction loss (e.g. MSE for continuous inputs, binary cr...
24. How do you diagnose a neural network that is not training correctly from its loss curves?
Reading loss curves is one of the most important practical skills in deep learning. The shape of the training and validation loss over time reveals the failure mode and guides the fix. Common Training Failure Modes Loss curve shape Diagnosis Likely fix Loss is NaN from the start Exploding gradien...
25. What is the mathematical setup of a Generative Adversarial Network (GAN) and what training challenges do they have?
A GAN consists of two competing networks: a generator G that maps random noise z ~ p(z) to fake data samples, and a discriminator D that classifies inputs as real or fake. They play a minimax game with objective: min_G max_D E[log D(x)] + E[log(1 - D(G(z)))] . At the Nash equilibrium, G produces ...
26. What is torch.compile and how does it speed up PyTorch model execution?
Introduced in PyTorch 2.0, torch.compile applies ahead-of-time compilation to a PyTorch model or function. Rather than executing each operation eagerly (PyTorch's default), it captures the computation as a graph, optimises it (fusing operations, eliminating redundant memory reads/writes), and com...
27. Why do Transformers need positional encodings and how does sinusoidal encoding work?
Self-attention is permutation equivariant — swapping two positions in the input produces the same output with those two positions swapped, because attention treats all positions symmetrically. Without positional information, a transformer cannot distinguish 'The dog bit the man' from 'The man bit...
28. What are the most impactful hyperparameters to tune in deep learning and what is the recommended search order?
Deep learning has many hyperparameters, but they are not equally important. Empirical research and practitioner experience has established a rough hierarchy of impact. Tuning in the wrong order wastes compute — finding the optimal dropout rate is pointless if the learning rate is still wildly off...
29. What is an encoder-decoder architecture and how is it used for sequence-to-sequence tasks?
Encoder-decoder (seq2seq) architectures handle tasks where the input and output are sequences of potentially different lengths — machine translation, summarisation, speech recognition, image captioning. The encoder processes the full input sequence and produces a context representation; the decod...
30. What is model quantization in deep learning and how does PyTorch support it?
Quantization reduces model size and inference latency by representing weights and activations in lower-precision integer formats (INT8, INT4, INT2) rather than FP32 or FP16. A 32-bit float weight is replaced by an 8-bit integer plus a scale factor and zero-point: x_float = scale × (x_int - zero_p...
31. What does a production-quality PyTorch training loop look like, incorporating all best practices?
A well-structured training loop separates concerns cleanly: data loading, forward pass, loss computation, backpropagation, gradient management, metric tracking, and model persistence. Each step has specific pitfalls that silently degrade results. import torch import torch.nn as nn from torch.cuda...
32. How does batch size affect deep learning training mathematically and practically?
Batch size controls the trade-off between gradient estimate quality and training speed. With batch size B, the gradient is estimated as the average loss gradient over B samples — the variance of this estimate is proportional to σ²/B , where σ² is the per-sample gradient variance. Larger batches g...
33. How do you choose the right layer type (Linear, Conv, Attention) for a given input modality?
Each layer type encodes different structural assumptions (inductive biases) about the data. Using a layer whose assumptions match the data's structure allows the model to learn faster and with less data than a generic alternative. Layer Selection by Modality and Structure Data type Structure Reco...
34. What evaluation metrics are most commonly used in deep learning tasks and how do you implement them in PyTorch?
The choice of evaluation metric should match the task's real-world objective, not just be the easiest to compute. The training loss and the evaluation metric are often different — models are trained with cross-entropy but evaluated with accuracy, F1, mAP, or BLEU depending on the application. Met...
35. How do you export a PyTorch model for production deployment using TorchScript or ONNX?
Research-time PyTorch models depend on Python's interpreter and PyTorch's eager execution mode — both are too slow and have too many dependencies for production deployment. Two standard serialisation formats allow deploying PyTorch models without Python: TorchScript (PyTorch-native, supports dyna...
36. What is knowledge distillation and how does it compress large neural networks into smaller ones?
Knowledge distillation (Hinton et al., 2015) trains a small student network to mimic the output distribution of a large, accurate teacher network. Instead of training only on hard labels (the correct class as a one-hot vector), the student is also trained to match the teacher's soft probabilities...
37. What is self-supervised learning and how do contrastive methods like SimCLR learn representations?
Self-supervised learning (SSL) is a form of unsupervised learning where the model is trained on a pretext task defined entirely from the data itself — no human-provided labels. The learned representations can then be transferred to downstream tasks with few or no labels (linear probe, fine-tuning...
38. How would you implement and train a simple feedforward neural network in PyTorch from scratch, without using nn.Sequential?
This question tests whether you understand the full PyTorch workflow: defining a custom nn.Module , implementing forward , and running the standard train loop. It is a common practical screen in ML engineering interviews. import torch import torch.nn as nn import torch.optim as optim from torch.u...
