Python / Python Deep Learning and Neural Networks Interview Questions
What evaluation metrics are most commonly used in deep learning tasks and how do you implement them in PyTorch?
The choice of evaluation metric should match the task's real-world objective, not just be the easiest to compute. The training loss and the evaluation metric are often different — models are trained with cross-entropy but evaluated with accuracy, F1, mAP, or BLEU depending on the application.
| Task | Primary metric | When it falls short |
|---|---|---|
| Classification (balanced) | Accuracy | Misleading on imbalanced classes |
| Classification (imbalanced) | F1 / AUC-ROC / PR-AUC | PR-AUC better than ROC-AUC for severe imbalance |
| Object detection | mAP (mean Average Precision) | Doesn't account for localisation precision at all scales |
| Regression | MAE / RMSE / R² | RMSE sensitive to outliers; R² can be negative |
| Machine translation | BLEU score | Doesn't capture semantic similarity |
| Language generation | Perplexity / ROUGE / BERTScore | Perplexity doesn't measure fluency |
| Segmentation | Intersection over Union (IoU / mIoU) | Sensitive to class imbalance |
import torch from torchmetrics import Accuracy, F1Score, AUROC, MeanSquaredError # torchmetrics: handles accumulation across batches correctly n_classes = 5 acc = Accuracy(task='multiclass', num_classes=n_classes) f1 = F1Score(task='multiclass', num_classes=n_classes, average='macro') auroc = AUROC(task='multiclass', num_classes=n_classes) model.eval() with torch.no_grad(): for X, y in val_loader: logits = model(X) preds = logits.argmax(dim=1) probs = torch.softmax(logits, dim=1) acc.update(preds, y) f1.update(preds, y) auroc.update(probs, y) print(f'Val Acc: {acc.compute():.4f}') print(f'Val F1: {f1.compute():.4f}') print(f'Val AUROC:{auroc.compute():.4f}') # Manual accuracy (without torchmetrics) all_preds, all_labels = [], [] with torch.no_grad(): for X, y in val_loader: preds = model(X).argmax(1) all_preds.append(preds.cpu()) all_labels.append(y.cpu()) preds = torch.cat(all_preds) labels = torch.cat(all_labels) accuracy = (preds == labels).float().mean() print(f'Accuracy: {accuracy:.4f}')
Why is accuracy a misleading metric for highly imbalanced classification datasets?
What does the torchmetrics library's .update() and .compute() pattern accomplish?
More Related questions...
