AI / LLM Basics Interview Questions
What is Cross-Entropy Loss?
Cross-Entropy Loss is the standard measurement used to train an LLM, quantifying how far off the model's predicted probability distribution was from the actual next token in the training data.
- Penalizes the model more heavily when it assigns low probability to the token that actually came next
- Averaged across every token prediction in a training batch to produce one overall loss value
- That loss value is what training actually tries to minimize, through repeated adjustments to the model's parameters
Lower cross-entropy loss on held-out data generally indicates a model that's genuinely learning useful patterns, rather than just memorizing its specific training examples.
What does Cross-Entropy Loss quantify?
When is the model penalized more heavily under this loss?
More Related questions...
