AI / LLM Basics Interview Questions
What is Quantization?
Quantization is a technique that reduces the numerical precision used to store a model's parameters, shrinking its memory footprint and speeding up inference.
- Converts weights from a high-precision format, like 32-bit floating point, down to a lower-precision one, like 8-bit or even 4-bit
- Reduces both the memory needed to store the model and the compute needed to run it
- Typically causes some small loss in output quality, though well-implemented quantization can often keep that loss minimal
Quantization is a major reason large models can run on comparatively modest hardware, like a laptop or a single consumer GPU, instead of requiring a full data center.
What does quantization reduce?
What does quantization commonly enable?
More Related questions...
