AI / LLM Basics Interview Questions
What is a Mixture of Experts (MoE) architecture?
Mixture of Experts is a model design where, instead of every input passing through the entire network, a routing mechanism sends each input to only a subset of specialized sub-networks, called experts.
- Only a fraction of the model's total parameters are actually used for any given input
- Lets a model have a very large total parameter count while keeping the actual compute cost per request much lower than using all parameters every time
- The routing mechanism itself is learned during training, deciding which experts are best suited for a given input
This approach is a practical way to scale a model's total capacity without proportionally scaling the cost of running it on every single request.
What does a routing mechanism do in a Mixture of Experts model?
What does MoE let a model scale without proportionally scaling?
More Related questions...
