Python / PyTorch Fundamentals Interview Questions
What activation functions are commonly used in PyTorch and how do you choose between them?
Activation functions introduce non-linearity, allowing networks to model complex functions. PyTorch provides them as both nn.Module classes (for use in nn.Sequential) and functional calls in torch.nn.functional.
| Activation | nn class | Range | Typical use |
|---|---|---|---|
| ReLU | nn.ReLU() | [0, ∞) | Default for hidden layers — fast, avoids vanishing gradient for x>0 |
| LeakyReLU | nn.LeakyReLU(0.01) | (-∞, ∞) | Fixes ReLU's dying neuron problem |
| Sigmoid | nn.Sigmoid() | (0, 1) | Binary classification output layer |
| Tanh | nn.Tanh() | (-1, 1) | RNN hidden states (zero-centred) |
| Softmax | nn.Softmax(dim=-1) | (0,1), sums to 1 | Multi-class output (use with NLLLoss, not CrossEntropyLoss) |
| GELU | nn.GELU() | (-∞, ∞) | Transformers (BERT, GPT) |
import torch
import torch.nn as nn
import torch.nn.functional as F
x = torch.tensor([-2.0, -0.5, 0.0, 0.5, 2.0])
# Module form — for use inside nn.Sequential / __init__
relu = nn.ReLU()
print(relu(x)) # tensor([0.0, 0.0, 0.0, 0.5, 2.0])
# Functional form — for use directly inside forward()
print(F.relu(x))
print(F.leaky_relu(x, negative_slope=0.01))
print(F.gelu(x))
# Using inside a model
class Net(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(10, 20)
self.fc2 = nn.Linear(20, 1)
def forward(self, x):
x = F.relu(self.fc1(x)) # functional — common in forward()
return torch.sigmoid(self.fc2(x)) # binary output
# IMPORTANT: never apply softmax before CrossEntropyLoss
# CrossEntropyLoss = LogSoftmax + NLLLoss internally
logits = torch.randn(4, 10) # raw scores, NOT softmaxed
loss_fn = nn.CrossEntropyLoss()
targets = torch.randint(0, 10, (4,))
loss = loss_fn(logits, targets) # correct — pass raw logits!Common mistake: applying Softmax before CrossEntropyLoss — the loss function already applies LogSoftmax internally, so double-softmaxing produces incorrect gradients and degraded training.
Invest now in Acorns!!! 🚀
Join Acorns and get your $5 bonus!
Acorns is a micro-investing app that automatically invests your "spare change" from daily purchases into diversified, expert-built portfolios of ETFs. It is designed for beginners, allowing you to start investing with as little as $5. The service automates saving and investing. Disclosure: I may receive a referral bonus.
Invest now!!! Get Free equity stock (US, UK only)!
Use Robinhood app to invest in stocks. It is safe and secure. Use the Referral link to claim your free stock when you sign up!.
The Robinhood app makes it easy to trade stocks, crypto and more.
Webull! Receive free stock by signing up using the link: Webull signup.
More Related questions...
