Python / PyTorch Fundamentals Interview Questions
1. What is PyTorch and what are its key advantages over other deep learning frameworks?
PyTorch is an open-source deep learning framework developed by Meta AI (Facebook), released in 2016. It is built around two core ideas: tensor computation with GPU acceleration (similar to NumPy but on the GPU) and automatic differentiation via a dynamic computation graph (called define-by-run or...
2. What is a PyTorch tensor and how does it differ from a NumPy array?
A tensor is PyTorch's core data structure — an n-dimensional array similar to NumPy's ndarray , but with two critical extra capabilities: it can live on a GPU for accelerated computation, and it supports automatic differentiation (autograd) for computing gradients during backpropagation. import t...
3. What are the most important tensor operations in PyTorch?
PyTorch provides a rich set of tensor operations covering arithmetic, shape manipulation, reduction, and linear algebra. Most have both a functional form ( torch.add ) and a method form ( tensor.add ), plus in-place variants with a trailing underscore ( tensor.add_ ). import torch a = torch . ten...
4. What are tensor data types (dtypes) in PyTorch and why do they matter?
Every tensor has a dtype that determines the numeric type and precision of its elements. Choosing the right dtype affects memory usage, computation speed, and numeric precision — a critical consideration when training on GPUs. Common PyTorch dtypes dtype Alias Bits Use case torch.float32 torch.fl...
5. How does broadcasting work in PyTorch and what are the rules?
Broadcasting allows PyTorch to perform arithmetic between tensors of different shapes without explicit copying. PyTorch follows the same broadcasting rules as NumPy. Understanding broadcasting is essential to avoid subtle shape bugs. import torch # Rule: align shapes from the RIGHT, expand dims o...
6. What is autograd in PyTorch and how does it compute gradients?
PyTorch's autograd engine implements automatic differentiation. When you perform operations on tensors with requires_grad=True , PyTorch records every operation in a dynamic computation graph . Calling .backward() on a scalar loss traverses this graph in reverse using the chain rule, accumulating...
7. What is the computation graph in PyTorch and how does the dynamic graph differ from a static graph?
PyTorch builds a dynamic computation graph (also called eager execution or define-by-run). Every time you run the forward pass, a new graph is constructed on-the-fly based on the actual Python code paths executed. This is in contrast to TensorFlow 1.x's static graph, which is compiled once and th...
8. How do torch.no_grad() and tensor.detach() differ, and when do you use each?
Both torch.no_grad() and .detach() stop gradient tracking, but they work at different levels and serve different purposes. import torch model_param = torch . tensor( 2.0 , requires_grad = True ) # ââ torch.no_grad(): context manager â disables ALL grad tracking # Use for inference and valid...
9. What is nn.Module and how do you build a custom neural network in PyTorch?
nn.Module is the base class for all neural network components in PyTorch. Subclassing it gives you parameter management, device placement, train/eval mode toggling, state dict serialisation, and hooks — all for free. import torch import torch.nn as nn class MLP (nn . Module): def __init__ (self, ...
10. What are nn.Sequential and other container modules in PyTorch?
PyTorch provides several container modules that compose layers without requiring a custom nn.Module subclass. They are convenient for simple feedforward architectures but less flexible than full subclassing. import torch import torch.nn as nn # ââ nn.Sequential: layers applied in order model ...
11. What built-in layers does PyTorch's nn module provide and how do you use the most common ones?
PyTorch's torch.nn module contains all the standard neural network building blocks. Understanding what each layer does mathematically helps you choose the right component and configure it correctly. Most common nn layers Layer Formula / purpose Key parameters nn.Linear y = xW^T + b — fully connec...
12. What are activation functions in PyTorch and how do you apply them?
Activation functions introduce non-linearity into neural networks, enabling them to learn complex mappings. PyTorch provides them both as nn.Module classes (usable as layers) and as functional forms in torch.nn.functional . Common activation functions Function Formula Typical use ReLU max(0, x) D...
13. What are the most important loss functions in PyTorch and when do you use each?
Choosing the right loss function is critical — it defines what the model is optimising for. PyTorch provides loss functions in torch.nn as modules and in torch.nn.functional as functions. Common PyTorch loss functions Loss Use case Input / Target nn.MSELoss Regression — minimise squared error pre...
14. What optimizers does PyTorch provide and how do you configure them?
Optimizers update model parameters based on computed gradients. PyTorch provides all major optimizers in torch.optim . Choosing and configuring the right optimizer significantly affects training speed and final performance. PyTorch optimizers Optimizer Key feature Typical use SGD Simple, supports...
15. What are learning rate schedulers in PyTorch and how do you use them?
A learning rate scheduler adjusts the learning rate during training — typically starting high for fast initial progress and decaying for fine-grained convergence. Schedulers wrap an optimizer and must be stepped after each epoch (or each batch for some schedulers). Common LR schedulers Scheduler ...
16. What are the most common built-in layers in torch.nn and what do they do?
PyTorch's torch.nn module provides all the standard building blocks for neural networks. Understanding what each layer does mathematically and when to use it is fundamental to building effective models. Common nn layers Layer Formula / behaviour Typical use nn.Linear(in, out) y = xW^T + b Fully c...
17. How do you initialise weights in a PyTorch model?
PyTorch uses sensible default initialisations (Kaiming uniform for Linear and Conv layers), but custom initialisation is often needed to match a paper or improve convergence. The torch.nn.init module provides all standard schemes. import torch , torch.nn as nn # Default initialisation: # nn.Linea...
18. What loss functions does PyTorch provide and when do you use each?
Loss functions (criteria) measure the difference between predictions and targets. PyTorch provides them in torch.nn . Choosing the right one for your task is critical — using the wrong loss gives poor training signal even if the architecture is correct. Common PyTorch loss functions Loss Class Ta...
19. What optimizers does PyTorch provide and how do you choose between them?
An optimizer updates model parameters based on computed gradients. PyTorch provides all major optimizers in torch.optim . Choosing the right optimizer and tuning its hyperparameters has a large impact on training speed and final performance. Common PyTorch optimizers Optimizer Class Key parameter...
20. What are learning rate schedulers in PyTorch and how do you use them?
A learning rate (LR) scheduler adjusts the learning rate during training. Starting with a high LR enables fast early progress; decaying it later allows finer convergence. PyTorch provides many schedulers in torch.optim.lr_scheduler . Common LR schedulers Scheduler Behaviour Use case StepLR Multip...
21. What activation functions are commonly used in PyTorch and how do you choose between them?
Activation functions introduce non-linearity, allowing networks to model complex functions. PyTorch provides them as both nn.Module classes (for use in nn.Sequential ) and functional calls in torch.nn.functional . Common PyTorch activations Activation nn class Range Typical use ReLU nn.ReLU() [0,...
22. What loss functions does PyTorch provide and how do you choose the right one?
The loss function defines the training objective. PyTorch's torch.nn module provides loss classes for classification, regression, and more specialised tasks. Choosing the wrong loss for your task is one of the most common beginner mistakes. Common PyTorch loss functions Loss Class Input shape Use...
23. What optimizers does PyTorch provide and what is the difference between SGD, Adam, and AdamW?
Optimizers update model parameters based on computed gradients. PyTorch's torch.optim module provides many algorithms; understanding their differences helps you choose the right one and tune hyperparameters effectively. Common PyTorch optimizers Optimizer Key idea Typical lr Best for SGD Plain gr...
24. What is the standard PyTorch training loop and what does each step do?
The PyTorch training loop follows a fixed five-step pattern repeated for every batch. Understanding exactly what each line does and what happens if you skip or reorder a step is essential for debugging training issues. import torch import torch.nn as nn import torch.optim as optim model = nn ...
25. What are Dataset and DataLoader in PyTorch and how do they work together?
PyTorch's data pipeline follows a clean two-class design: Dataset defines how to access a single sample (index → data), and DataLoader wraps a Dataset to handle batching, shuffling, and parallel loading. import torch from torch.utils.data import Dataset, DataLoader import numpy as np class Tabula...
26. How do you move tensors and models between CPU and GPU in PyTorch?
PyTorch's device abstraction allows the same code to run on CPU or GPU with minimal changes. The fundamental rule: a model and its input tensors must reside on the same device before any computation, or PyTorch raises a RuntimeError. import torch import torch.nn as nn # Device-agnostic pattern â...
27. What is the difference between model.parameters() and model.state_dict() in PyTorch?
Both expose a model's learnable values, but they serve different purposes. parameters() returns an iterator of nn.Parameter tensor objects (used by the optimizer); state_dict() returns an OrderedDict mapping layer names to tensors (used for saving/loading and inspection). import torch import torc...
28. How do you save and load PyTorch models correctly, including full training checkpoints?
PyTorch supports saving either the full model object or just its weights (state_dict). Saving only the state_dict is the recommended approach because it decouples weights from the Python class definition. A full training checkpoint includes the optimizer state too, so training can resume exactly ...
29. What is overfitting and what regularization techniques does PyTorch support to address it?
Overfitting occurs when a model memorises the training data instead of learning generalisable patterns — visible as low training loss but high validation loss. PyTorch provides several built-in tools to combat overfitting. PyTorch regularization techniques Technique How to apply Effect Dropout nn...
30. What is the vanishing/exploding gradient problem and how do you detect and fix it in PyTorch?
During backpropagation, gradients are computed via repeated multiplication through the chain rule. In deep networks, this can cause gradients to shrink toward zero (vanishing) or grow toward infinity (exploding) as they propagate backward through many layers, preventing effective training. import...
31. What is weight initialization in PyTorch and why does it matter?
How a network's weights are initialised at the start of training significantly affects whether training converges quickly, slowly, or not at all. PyTorch's default initialisation (Kaiming uniform for Linear/Conv layers) works well in most cases, but understanding the principles helps when debuggi...
32. What is the difference between nn.Parameter and a regular tensor attribute in nn.Module?
nn.Parameter is a special tensor subclass that, when assigned as an attribute of an nn.Module , is automatically registered in the module's parameter list — meaning it appears in model.parameters() , gets moved by .to(device) , and is saved in state_dict() . A plain tensor attribute does none of ...
33. How do you implement and use learning rate schedulers in PyTorch?
A fixed learning rate throughout training is rarely optimal — too high late in training prevents fine convergence, while too low early on wastes time. PyTorch's torch.optim.lr_scheduler module adjusts the learning rate systematically as training progresses. import torch import torch.nn as nn impo...
34. How do you debug a PyTorch training loop where the loss is not decreasing or is NaN?
Diagnosing a stuck or diverging training loop is one of the most valuable practical PyTorch skills. The shape of the loss curve and a few targeted checks usually reveal the root cause. Common training failure modes Symptom Likely cause Fix Loss is NaN from step 1 Exploding gradients, bad data (in...
35. What is the difference between torch.tensor() and torch.Tensor() (capital T) for creating tensors?
This is a subtle but important PyTorch gotcha. torch.tensor() (lowercase, a function) infers dtype from the input data and copies it — the recommended way to create tensors from data. torch.Tensor() (uppercase, a class constructor) is an alias for torch.FloatTensor and behaves inconsistently depe...
36. How does gradient accumulation work in PyTorch and when would you use it?
Gradient accumulation simulates a larger effective batch size than fits in GPU memory by summing gradients over several smaller forward/backward passes before calling optimizer.step() . This is useful when training large models on limited GPU memory. import torch import torch.nn as nn model = nn ...
37. What is mixed precision training in PyTorch and how do you implement it with torch.cuda.amp?
Mixed precision training runs most operations in FP16 (or BF16) for speed while keeping a master copy of weights in FP32 for numerical stability. Modern GPUs (Volta and later) have dedicated hardware (Tensor Cores) that make FP16 matrix multiplication significantly faster than FP32. import torch ...
38. What is torch.compile() and how does it speed up PyTorch model execution?
Introduced in PyTorch 2.0, torch.compile() performs just-in-time compilation of a model. Instead of executing each tensor operation eagerly (PyTorch's default), it captures the computation graph, fuses operations, and generates optimised kernels — primarily reducing GPU memory round-trips. import...
39. What is the difference between batch size, epoch, and iteration in PyTorch training?
These three terms are fundamental to understanding any training loop, and confusing them is a common source of bugs when computing metrics or setting up learning rate schedules. Training terminology Term Definition Example Batch size Number of samples processed together in one forward/backward pa...
40. How do you compute and track evaluation metrics like accuracy during PyTorch training?
Tracking metrics correctly requires accumulating values across all batches (not just averaging per-batch metrics naively, which can be biased if the last batch has a different size) and ensuring computations happen without gradient tracking. import torch import torch.nn as nn @torch . no_grad() #...
41. What is the purpose of torch.manual_seed() and how do you ensure reproducibility in PyTorch?
PyTorch uses pseudo-random number generators for weight initialisation, dropout masks, data shuffling, and more. Setting seeds explicitly ensures experiments are reproducible — critical for debugging, comparing model variants fairly, and scientific rigor. import torch import numpy as np import ra...
42. How does PyTorch handle multi-dimensional indexing and slicing of tensors?
PyTorch tensor indexing follows NumPy-style conventions, including basic slicing, advanced (fancy) indexing with integer/boolean tensors, and the powerful ... (ellipsis) operator for indexing high-dimensional tensors concisely. import torch x = torch . arange( 24 ) . reshape( 2 , 3 , 4 ) # shape ...
43. What is the difference between.view(),.reshape(), and.contiguous() in PyTorch, and why does it matter?
These three methods deal with how a tensor's underlying memory is interpreted as a different shape. Understanding the difference prevents a class of confusing runtime errors related to tensor memory layout. import torch x = torch . arange( 12 ) . reshape( 3 , 4 ) # shape (3, 4), contiguous memory...
44. How do you freeze layers and perform transfer learning / fine-tuning in PyTorch?
Transfer learning reuses a model pretrained on a large dataset and adapts it to a new task. Freezing layers (setting requires_grad=False ) prevents their weights from updating during backpropagation — useful when you want to keep pretrained features fixed and only train a new task-specific head. ...
45. What is the purpose of torch.utils.data.random_split() and how do you create train/validation/test splits in PyTorch?
Splitting a dataset into training, validation, and test subsets is a fundamental step before training. PyTorch's random_split() creates non-overlapping random subsets from a single Dataset, while preserving the lazy-loading behaviour of the original Dataset. import torch from torch.utils.data imp...
46. What is Batch Normalization in PyTorch and how does it differ from Layer Normalization?
Normalization layers stabilise training by re-centring and re-scaling activations. PyTorch provides several variants; Batch Normalization (BatchNorm) and Layer Normalization (LayerNorm) are the two most widely used, but they normalise over different dimensions and suit different architectures. Ba...
47. How do you implement and use a custom loss function in PyTorch?
When built-in loss functions do not fit your task, you can write a custom loss as either a plain function or an nn.Module subclass. As long as the loss is computed from PyTorch tensor operations with requires_grad=True parameters, autograd handles differentiation automatically. import torch impor...
48. What is torch.compile() vs TorchScript and how do you export a PyTorch model for production deployment?
PyTorch offers two main paths for production deployment beyond running the Python interpreter: TorchScript (serialises the model as a language-independent IR) and torch.compile() (JIT compiles for speed within Python). For cross-language/cross-framework deployment, ONNX export is also widely used...
